Holistic Intervention Glove System – Project Whole´O´Hand

The small incisions used for optical and haptical access within so-called minimal-invasive surgery (MIS) are known to reduce the trauma as well as the collateral damage for the patient. Nevertheless, such small keyhole incisions simultaneously reduce also the vision, the orientation, as well as the possibilities of maneuverability at the site for the surgeon. Hence, to compensate these cognitive and haptical losses, such small-scale interventions have always been dependent on technical equipment, as e. g. endoscopes, light-sources, and extensions of manipulators.

Nevertheless, with the generation of instruments current available for “MIS“ or “NOTES“ interventions the steadily increasing possibilities of mechatronics, micro-electronics and information technologies are barely used. Furthermore, the instruments available lack the functional interfaces to acquire, store, receive and exchange information with other instruments and systems, and hence are not apt to use the possibilities of a higher-level information exchange, integration and complementary use of such systems.

Tackling these issues the “Whole´O´Hand“project was started where the Fraunhofer Institute for Integrated Circuits IIS, Fraunhofer Institute for Manufacturing Engineering and Automation IPA as well as the Fraunhofer Institute for Computer Graphics Research IGD as involved. The project was funded by the Fraunhofer-Gesellschaft since 2008 for three years.

The project “Whole´O´Hand“ (“Holistic interventional interaction system“) aims to support especially the surgeon´s vision, orientation, and dexterity with a novel concept for the integration and fusion of endoscopy, robotics, mechatronics, sensing and measuring devices, computer-assisted diagnosis, navigation and visual computing.

In order to achieve a more gentle as well as faster intervention for the patient, the project aims to enhance the pre-operative planning of the intervention using multi-slice CT images, to support the surgeon during the intervention by novel tele-manipulation and instrumentation-exchange concepts, and finally to assist the surgeon by locating and diagnosing the lesion and guiding the resection using multi-modal imaging and image analysis methods. Besides the conceptual and technical development and realization of all necessary sub-components and interfaces, the prototype of this system shall be used to demonstrate the potentials of computer-integrated surgical systems (CISS) for “MIS“ and “NOTES“ interventions, hence giving new impulses for future systems concepts and instruments.

As a primary application scenario for the presented closed-loop concept for this computer-integrated surgical system, minimally invasive liver tumor resection has been selected. Nevertheless, the presented concept and systems can be adapted to other scenarios as well, such as cystoscopic bladder surgery.

The main workflow of the “Whole´O´Hand“ system starts with the pre-operative surgical planning based on volumetric CT data. The extracted planning data is then sent to the navigation control system, which is part of the surgical control unit (SCU). Based on the SCU which is located at the operation table, all sub-systems can be monitored, coordinated and controlled. In order to register the patient to the coordinate system of the SCU, a 3D stripe projector is used, whose acquired surface grid is registered to the abdomen surface extracted from the CT data. Based on this registration as well as on further surgical planning data, the positions of the trocars on the surface of the patient´s abdomen are located and marked via light projection. Hence, an optimal pre-operative planned access can be set for the complete intervention, which furthermore relates to the working volume of the instrument carrier system.

Next, the two instrument carrier systems (ICS) are moved towards the operation table. The ICS supports the surgeon by hosting and guiding the instruments. Furthermore, an innovative revolver-based instrument exchange system (IES) allows the direct change of instruments and sensors without instrument withdrawal inside the patient´s abdomen, which reduces the time needed to remove and exchange the instrument through the trocar. After positioning of the trocars, a 3D ultrasound sensor is inserted to intra-operatively scan the liver surface and thus reconstruct the liver volume. The liver volume is then registered to the pre-operative planning data from the CT, thus transforming the planning data into the actual intra-operative (and possible by air inflation deformed) coordinate system.

During the resection the liver can be re-scanned permanently by means of the ultrasound sensor, thus allowing a constant matching between the planning data and the actual site. Furthermore the resection boundaries can continuously be controlled by the ultrasound device, thus allowing the detection of any dangerous changes. Possible accidents can be avoided by so-called active constraints-mechanisms. If necessary, the planning data can be adapted accordingly based on the constantly acquired volumetric ultrasound data.

The vessel trees of the liver are extracted and represented symbolically as a directed graph from pre-operatively obtained CT-scans of the abdomen as well as intraoperatively acquired 3D ultrasound image. To generate such a symbolical description of the liver vessel system, the 3D CT/US image volumes are manipulated using a distinct chain of image processing methods. The first step yields a segmentation of the vessel tree, which is transformed into a so-called graph-based skeleton, representing the center lines of the vessels. Finally the skeleton is converted into a graph-based symbolic description. Based on this representation, a registration of pre- and intra-operative image data is possible.

For the actual registration of the pre-operative CT and the intra-operative 3D ultrasound data, most of the correlating bifurcations between vessel trees are automatically detected and can be interactively refined to correct wrong matches. Based on this representative set of correlated landmarks between both image modalities, the deformation parameters can be obtained. Using this registration and the related deformation and transformation data, pre-operative computed resection-plans as well as resection boundaries can be transferred to the interventional site and be used and visualized intra-operatively together with the continuously acquired 3D ultrasound volumes. Hence an integrated visual control of the resection and the resection boundaries is possible. Furthermore, the current positions of the instruments can be augmented to the fused images.

The objectives of the “Whole´O´Hand“ instrument exchange system (IES) are simplification of use and the fast exchange of laparoscopic instruments. In standard laparoscopic procedures, instrument changes account for a significant amount of time; they interrupt the surgeon´s concentration phase and involve a reorientation of the new instrument at the region of interest. Within the “Whole´O´Hand“ instrument exchange system newly developed capsular instruments are stored outside the body in a cylindrical magazine and are automatically positioned at situ via a hollow cylindrical shaft using a hydraulic mechanism. At the proximal end of the shaft inside the patient´s abdomen the capsular instruments are electrically contacted in order to establish electrosurgical functionalities or allow electrical feedback from sensors. Up to six different graspers, forceps, HF-instruments or tip-chip endoscopic cameras can be stocked and automatically exchanged within a few seconds.

Navigation as well as intra-operatively localizing tumors a sweeping sonography system is used, which produces three-dimensional images of the liver. These volumes are digitally processed and registered to the pre-operatively acquired planning data. Thus, the system provides visualization for the resection process to the surgeon and gives notice of a possible constraint violation.

Placed at a single position, the sonography system samples several cross sections by means of a sweeping mechanism. These registered images are interpolated to a small volume part. During an initial scan across the liver surface a set of overlapping ultrasound volumes is acquired and subsequently merged to a coherent volume of the entire organ. During resection the ultrasound volume is dynamically updated by frequent rescans of the area of surgical interest. While the sweeping mechanism provides pose information of each of the cross sections to map them into the polume, the instrument carrier system delivers necessary position information for volume fusion. As sophisticated approach a miniaturized video sensor is laterally attached to the ultrasound probe to provide precise position information in case of organ motion. By processing the video stream, the relative motion between instrument and organ is detected and used to support the positioning of the resection instruments. Based on an automatic or interactive segmentation of the vessel tree, the navigation system obtains a registration of the ultrasound data with the pre-operative CT planning data. Hence, the planning data can be transferred to the current operative site. Thus, the surgeon can be provided with an augmented visualization of relevant constraints and additional information. By means of frequent rescans the current surgical activities are monitored and guidance is provided to assure compliance with transaction constraints.

The correct diagnosis of lesions usually requires a conventional biopsy followed by a histological analysis. To reduce the related costs, improve the accuracy whilst reducing the number of biopsies taken in a specific area and to accelerate the process of diagnosis by analyzing the area of interest right during the intervention, an approach for image-based computer-assisted diagnostics (CAD) is developed. The most important characteristics are the multimodal image acquisition and the semi-automatic online image processing.

Using a high-resolution endoscopic camera with a multi-modal (autofluorescent and whitelight) illumination, characteristical tissue features of the lesions like textual, morphological, colorimetric and functional parameters can be calculated. By means of a content-based image-retrieval approach, images with similar characteristics can be selected from a histologically validated reference data base. They can be used for early detection, a more confident diagnosis and to clarify the necessity of a biopsy. Furthermore, the search for similar reference images and the presentation of similar cases can support the approach of case-based reasoning.

The proposed system can be integrated in the established diagnostic workflow where the acquired images are pre-sented on a high-resolution monitor. If lesions are depicted on the monitor, a snapshot of the current display can be taken by using a foot-switch. Then the questionable lesion can be delineated interactively by using a touch-screen Based on the segmented region of interest the system presents the most similar reference images and provides a suggestion for the diagnosis, which can then be used for treatment planning.