Fraunhofer Institute for Integrated Circuits IIS
Fraunhofer Institute for Integrated Circuits IIS
Abstract operating room environment representing the HaptiVisT research project on haptic-visual surgical training, digital twins, and immersive medical simulation.
© Parilov – stock.adobe.com

HaptiVisT

Haptic feedback, virtual reality, and patient-specific imaging data for surgical training

© Parilov – stock.adobe.com

Training Surgical Procedures on a Digital Twin

Seeing, feeling, learning: from the digital twin to surgical training

How can complex surgical interventions be practiced before they are carried out on patients? The HaptiVisT research project addressed this question. The goal was to combine medical image data, virtual reality, and haptic feedback into a realistic training system for surgical education and further training.

Experience and routine are crucial, especially for surgical interventions in anatomically challenging areas. At the same time, training opportunities are limited, and access to specialized training centers is often restricted. Digital training systems open up new possibilities here, allowing surgical processes to be practiced safely, reproducibly, and independently of time and location in so-called “serious games.”

The project resulted in the development of the virtual demonstrator “HandsOn.surgery.” Using patient-specific CT and DVT data, surgeons can practice individual procedures on a digital twin of the patient without risk, at any time, and as often as needed. Haptic feedback allows users not only to see but also to feel virtual tissue structures.

In addition to the training system, new approaches were developed for modeling surgical instruments, for haptic rendering of biological structures, and for designing intuitive human-machine interactions. The insights gained are still making important contributions to research fields such as medical robotics, digital twins, and immersive training systems.

The “HandsOn.surgery” demonstrator combines real CT/DVT data, stereoscopic visualization, and haptic feedback to create an immersive training environment for surgical interventions.

From clinical images to virtual training

The HaptiVisT project explored and combined key technologies for haptic-visual surgical training, demonstrating them in a virtual surgical training platform.

Digital twins

Patient-specific CT and DVT data serve as the basis for individualized training situations. Fraunhofer IIS developed methods for segmenting complex anatomical structures and creating digital twins for surgical training.

Haptic feedback

Realistic force and touch feedback enable the training of surgical interventions in a virtual environment. Fraunhofer IIS explored haptic mapping of biological tissues and supported the integration and calibration of the haptic system.

Evaluation of human-machine interaction

Fraunhofer IIS evaluated the interaction between users and the system in terms of realism, user experience, and training effects.

Key facts at a glance

Expand all Close all

  • As part of the HaptiVisT project, a haptic-visual training system was developed that combines virtual reality, patient-specific image data, and haptic feedback to create an immersive learning environment. The goal was to enable users to experience surgical interventions in a realistic manner and to explore new approaches to medical education and further education.

  • Mock-up HaptiVisT – Haptic-visual Training for Cochlea Implants
    © Fraunhofer IIS
    Mock-up HaptiVisT – Haptic-visual Training for Cochlea Implants

    Patient-specific computed tomography (CT) and digital volume tomography (DVT) data are processed to create a digital twin. Based on this three-dimensional image data, surgical interventions can be visualized in a virtual environment. Using a haptic arm, users can interact with virtual instruments and, by means of force feedback, immediately perceive resistance and tissue properties haptically during the surgical intervention.

  • Varying osseous transparency and visibility of risk structures in the middle ear
    © Fraunhofer IIS
    Varying osseous transparency and visibility of risk structures in the middle ear. The nerves (chorda tympani, nervus facialis) are shown in yellow, while the cochlea and semicircular canals (critical to sense of balance) are shown in gray-blue. From left to right, the transparency decreases and the difficulty increases.

    The focus was on procedures involving the temporal bone – such as in cochlear implantations – as well as applications in hand surgery. Both fields are characterized by complex anatomical structures and high demands on precision and surgical experience, and therefore served as exemplary scenarios for evaluating the training system.

  • The training system was scientifically evaluated throughout the entire project and assessed in collaboration with medical experts. The focus was on realism, user-friendliness, perceived training benefits, and the quality of haptic and visual interaction.

    The goal was to systematically examine the system’s technical implementation, usability, and perceived training benefits, and to continuously integrate the results into its further development. For this purpose, Fraunhofer IIS developed an evaluation concept with appropriate standards, methods, and procedures for assessing haptic-visual training systems.

  • HaptiVisT combines key areas of modern human-machine interaction, including digital twins, haptic interaction, virtual training environments, and the scientific evaluation of complex assistance systems.

    The approaches developed in the project for mapping biological structures, providing haptic feedback, and designing intuitive interactions provide important insights for current research in the fields of robotics, medical simulation, and digital training.