Navigations- und AR - Visualisierungssystem für den Einsatz bei minimalinvasiven, unfallchirurgischen Operationen (NARVIS)

The NARVIS Project has been successfully completed in 2007.

From the annual report of Bayerische Forschungsstiftung (BFS) (2007):

Ziel der Entwicklung des Navigations- und Visualsierungssystems NARVIS war, eine dreidimensionale In-situ-Visualisierung für die minimalinvasive Unfallchirurgie zu schaffen. Dabei soll ein Head-Mounted Display (HMD) dem Arzt präoperative Bilddaten im Arbeitsbereich darstellen. Im Gegensatz zu klassischen Mehrschirmlösungen erlaubt die Erweiterung (Augmentierung) der Sicht auf den Operationssitus dem Chirurgen, sich auf die Operation zu konzentrieren. Dazu bietet sie ihm neben erweiterten Ansichtsmöglichkeiten auch Zusatzinformationen und Navigationshilfen bei schwierigen Eingriffen. Im Projekt sollte geprüft werden, inwieweit sich die HMD-Visualisierungstechnologie für derartige Anwendungen eignet und in den klinischen Arbeitsablauf integrieren lässt.

Nach der Systemplanung durch Mediziner und Ingenieure wurde ein erster Prototyp gebaut, der bei einer Reihe von experimentellen Operationen erprobt und in Design, Handhabung und Funktionalität nach den Verbesserungsvorschlägen der Mediziner weiter optimiert wurde. Dabei stellte sich heraus, dass die derzeitige HMD-Technik aufgrund von Gewicht und mangels Tragekomfort nicht durchgehend während einer OP getragen werden kann. Genauere Workflow-Analysen halfen bei der Identifizierung von Arbeitsabschnitten, in denen NARVIS nützlich sein kann. Auch wenn sich die HMD-Technik für einen direkten klinischen Einsatz noch besser an die klinischen Anforderungen anpassen muss, kann NARVIS in der chirurgischen Ausbildung schon wertvolle Dienste leisten, etwa wenn es von einem auszubildenden Mediziner getragen wird, der die Operation vor Ort verfolgt. Das rege Interesse an NARVIS bei Konferenzen und Vorführungen legt nahe, dass der Ansatz weiterverfolgt werden sollte – von technischen Verbesserungen würde er unmittelbar profitieren.

NARVIS sketch

NARVIS Related Publications

Find here a list of selected publications that are related to the NARVIS Project.

NARVIS Related Projects

The following projects have been started based on the objectives of the NARVIS Project that has been successfully completed in 2007.

NARVIS - navigated augmented reality visualization system
Advanced visualization is getting increasingly important for the operation room of the future. The increasing number of available medical images must be presented to surgery team in new ways in order to support them rather than overloading with more information. In our project NARVIS we integrate an HMD-based (head mounted display) AR system into the operation room for 3D in situ visualization of computed tomography (CT) images. The final system aims at spinal surgery. The work is in close collaboration with our project partners “Klinikum für Unfallchirgie” at LMU, A.R.T. Weilheim, and Siemens Corporate Research in Princeton. The project is funded by Bayerische Forschungsstiftung.

ARAV Augmented Reality Aided Vertebroplasty
In today’s ORs more and more operations are performed employing minimally invasive procedures. Surgical instruments are inserted through a tiny cut on the patient’s skin, the port to the inside of the patient. In some cases endoscope cameras record video images of the operation site that are presented on a monitor. As a consequence of this technique, the surgeon’s field of view is divided into several work spaces, the monitor, the patient and information of medical imaging data presented on a third station. The missing direct view on the workspace complicates intuitive control of surgical tools. In contrast to open surgery the physician has to collect information from several fields of view at the same time and fuse information mentally to create a complete model of his working space, the operation site. The minimally invasive intervention vertebroplasty was determined as a suitable medical application to bring an Head Mounted Display (HMD) into the OR for augmentation of surgical instruments and medical imaging data. In-situ visualization with an HMD presents all available imaging data and navigational information in one field of view. The objective of vertebroplasty is the insertion of cement into weak and brittle vertebrae through a trocar for stabilization. In this case the view on the inside of the patient is not provided by an endoscope camera. However, since the operation is performed under a CT scanner, imaging data is permanently updated to check position of the trocar and amount of inserted cement. Imaging data is presented on a monitor and has to be mapped mentally by the surgeon on the real operation site.

Improving Depth Perception and Perception of Layout for In-Situ Visualization in Medical Augmented Reality
In-situ visualization in medical augmented reality (AR) using for instance a video see-through head mounted display (HMD) and an optical tracking system enables the stereoscopic view on visualized CT data registered with the real anatomy of a patient. Data can aligned with the required accuracy and the surgeons do not have to analyze data on an external monitor or images attached to the wall somewhere in the operating room. Thanks to a medical AR system like mentioned before, surgeons get a direct view onto and also ”into” the patient. Mental registration of medical imagery with the operation site is not necessary anymore. In addition surgical instruments can be augmented inside the human body. Bringing medical imagery and surgical instruments in the same field of action provides the most intuitive way to understand the patient’s anatomy within the region of interest and allows for the development of completely new generations of surgical navigation systems.
Unfortunately, this method of presenting medical data suffers from a serious lack. Virtual imagery, such as a volume rendered spinal column, can only be displayed superimposed on real objects. If virtual entities of the scene are expected behind real ones, like the virtual spinal column beneath the real skin surface, this problem implicates incorrect perception of the viewed objects respective their distance to the observer. The strong visual depth cue interposition is responsible for misleading depth perception. This project aims at the development and evaluation of methods to improve depth perception for in-situ visualization in medical AR. Its intention is to provide an extended view onto the human body that allows an intuitive localization of visualized bones and tissue.

Virtual Mirror: Interaction Paradigm for Augmented Reality Applications
Augmented Reality offers a higher degree of freedom for the programmer than classical visualization of volume data on a screen. The existing paradigms for interaction with 3D objects are not satisfactory for particular applications since the majority of them rotate and move the object of interest. The classic manipulation of virtual objects cannot be used while keeping real and virtual spaces in alignment within an AR environment. This project introduces a simple and efficient interaction paradigm allowing the users to interact with 3D objects and visualize them from arbitrary viewpoints without disturbing the in-situ visualization, or requiring the user to change the viewpoint. We present a virtual, tangible mirror as a new paradigm for interaction with 3D models. The concept borrows its visualization paradigm in some sense from methodology used by dentists to examine the oral cavity without constantly changing their own viewpoint or moving the patients head. The virtual mirror improves the understanding of complex structures, enables completely new concepts to support navigational aid for different tasks and provides the user with intuitive views on physically restricted areas.

Augmented Reality Supported Patient Education and Consultation
The project Augmented Reality Supported Patient Education and Consultation (Augmented Reality unterstützte Operationsaufklärung) aims at developing Augmented Reality (AR) supported communication tools for patient education. The development of the targeted systems involves disciplines reaching from image registration, human computer interaction and in-situ visualization to instructional design and perceptual psychology. As a primary clinical application, we determined breast reconstruction in plastic surgery.

MeTaTop A Multi Sensory Table Top System for Medical Procedures
A tabletop system in medical environments can be used for interactive and collaborative analysis of patient data but also as a multimedia user interface within sterile space. For preoperative planning physicians in charge with a particular patient meet to discuss the medical case and plan further steps for therapy. For this reason, they could collaboratively view and browse through all kind of available medical imaging data with the tabletop system. Alternatively such a system could be a central interaction device for all kind of equippment within the OR requiring user input, however, can not be operated by the sterile surgeon. We believe that the projection of all kind of user and information interfaces on a sterile glass plane would facilitate the clinical workflow.
This project is strongly related to the Tangible Interaction Surface for Collaboration between Humans project.

3D user interfaces for medical interventions
This work group aims at practical user interfaces for 3D imaging data in surgery and medical interventions. The usual monitor based visualization and mouse based interaction with 3D data will not present acceptable solutions. Here we study the use of head mounted displays and advanced interaction techniques as alternative solutions. Different issues such as depth perception in augmented reality environment and optimal data representation for a smooth and efficient integration into the surgical workflow are the focus of our research activities. Furthermore appropriate ways of interaction within the surgical environment are investigated.

Laparoscope Augmentation for Minimally Invasive Liver Resection
In recent years, an increasing number of liver tumor indications were treated by minimally invasive laparoscopic resection. Besides the restricted view, a major issue in laparoscopic liver resection is the precise localization of the vessels to be divided. To navigate the surgeon to these vessels, pre-operative imaging data can hardly be used due to intra-operative organ deformations caused by appliance of carbon dioxide pneumoperitoneum and respiratory motion.

Therefore, we propose to use an optically tracked mobile C-arm providing cone-beam computed tomography imaging capability intra-operatively. After patient positioning, port placement, and carbon dioxide insufflation, the liver vessels are contrasted and a 3D volume is reconstructed during patient exhalation. Without any further need for patient registration, the volume can be directly augmented on the live laparoscope video. This augmentation provides the surgeon with essential aid in the localization of veins, arteries, and bile ducts to be divided or sealed.

Current research focuses on the intra-operative use and tracking of mobile C-arms as well as laparoscopic ultrasound, augmented visualization on the laparoscope's view, and methods to synchronize respiratory motion.

2020
	A. Martin-Gomez Evaluation of Different Visualization Techniques for Perception-Based Alignment in Medical AR 19th IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2020) (bib)
	A. Martin-Gomez, J. Fotouhi, U. Eck, N. Navab Gain A New Perspective: Towards Exploring Multi-View Alignment in Mixed Reality 19th IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2020) (bib)
	A. Martin-Gomez, A. Winkler, K. Yu, T. Roth, U. Eck, N. Navab Augmented Mirrors 19th IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2020) (bib)
	A. Winkler, U. Eck, N. Navab Spatially-Aware Displays for Computer Assisted Interventions Proceedings of the 23rd International Conference on Medical Image Computing and Computer Assisted Interventions (MICCAI), Lima, October 2020 (bib)
	A. Martin-Gomez, J. Fotouhi, N. Navab Towards Exploring the Benefits of Augmented Reality for Patient Support During Radiation Oncology Interventions Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization (bib)
	A. Martin-Gomez, U. Eck, J. Fotouhi, N. Navab Looking Also From Another Perspective: Exploring the Benefits of Alternative Views for Alignment Tasks 27th IEEE Virtual Reality Conference, Atlanta, Georgia, USA, Mar 2020 (bib)
2019
	F. Bork, A. Lehner, D. Kugelmann, U. Eck, J. Waschke, N. Navab VesARlius: An Augmented Reality System for Large-Group Co-Located Anatomy Learning International Symposium on Mixed and Augmented Reality (ISMAR), 2019 (bib)
	F. Bork, U. Eck, N. Navab Birds vs. Fish: Visualizing Out-Of-View Objects in Augmented Reality using 3D Minimaps International Symposium on Mixed and Augmented Reality (ISMAR), 2019 (bib)
	A. Martin-Gomez, U. Eck, N. Navab Visualization Techniques for Precise Alignment in VR. A Comparative Study 26th IEEE Virtual Reality Conference, Osaka, Japan, Mar 2019 (bib)
2018
	F. Bork, A. Schnelzer, U. Eck, N. Navab Towards Efficient Visual Guidance in Limited Field-of-View Head-Mounted Displays IEEE Transactions on Visualization and Computer Graphics (bib)
	S. Matinfar, A. Nasseri, U. Eck, , A. Roodaki, Navid Navab, C. Lohmann, M. Maier, N. Navab Surgical soundtracks: automatic acoustic augmentation of surgical procedures International Journal of Computer Assisted Radiology and Surgery. The final publication is available here (bib)
	U. Eck, T. Sielhorst Display Technologies Mixed and Augmented Reality in Medicine, CRC Press, 1st edition. 14pp, July 2018 (bib)
	U. Eck, A. Winkler Display-Technologien fuer Augmented Reality in der Medizin Der Unfallchirurg, Springer International, Volume 121, Number 4 pp. 278--285, Feb 2018 (bib)
2017
	F. Bork, R. Barmaki, U. Eck, K. Yu, C. Sandor, N. Navab Empirical Study of Non-Reversing Magic Mirrors for Augmented Reality Anatomy Learning International Symposium on Mixed and Augmented Reality (ISMAR), 2017 (bib)
	S. Matinfar, A. Nasseri, U. Eck, A. Roodaki, Navid Navab, C. Lohmann, M. Maier, N. Navab Surgical Soundtracks: Towards Automatic Musical Augmentation of Surgical Procedures Proceedings of the 20th International Conference on Medical Image Computing and Computer Assisted Interventions (MICCAI), Quebec, Canada, September 2017. This work won the young scientist award at MICCAI 2017. The final publication is available here (bib)
	S. Andress, U. Eck, C. Becker, A. Greiner, B. Rubenbauer, C. Linhart, S. Weidert Automatic Surface Model Reconstruction to Enhance Treatment of Acetabular Fracture Surgery with 3D Printing 17TH ANNUAL MEETING OF THE INTERNATIONAL SOCIETY FOR COMPUTER ASSISTED ORTHOPAEDIC SURGERY, Aachen, Germany, pp. TBA, Jun 2017 (bib)
	F. Bork, R. Barmaki, U. Eck, P. Fallavollita, B. Fuerst, N. Navab Exploring Non-reversing Magic Mirrors for Screen-Based Augmented Reality Systems 24th IEEE Virtual Reality Conference, Los Angeles, CA, USA, pp. 373-374, Mar 2017 (bib)
2016
	Alana Da Gama, Thiago Menezes Chaves, Lucas Silva Figueiredo, Adriana Baltar, Ma Meng, N. Navab, Veronica Teichrieb, P. Fallavollita MirrARbilitation: A clinically-related gesture recognition interactive tool for an AR rehabilitation system Computer Methods and Programs in Biomedicine (bib)
	Xiang Wang, S. Habert, C. Schulte-zu-Berge, P. Fallavollita, N. Navab Inverse visualization concept for RGB-D augmented C-arms Computers in Biology and Medicine (bib)
	Ma Meng, Philipp Jutzi, F. Bork, Ina Seelbach, A. von der Heide, N. Navab, P. Fallavollita Interactive mixed reality for muscle structure and function learning 7th International Conference on Medical Imaging and Augmented Reality (to appear), Aug 2016 The first three authors contribute equally to this paper. (bib)
	U. Eck, P. Stefan, H. Laga, C. Sandor, P. Fallavollita, N. Navab Exploring Visuo-Haptic Augmented Reality User Interfaces for Stereo-Tactic Neurosurgery Planning 7th International Conference on Medical Imaging and Augmented Reality, Bern, Switzerland, pp. 208-220, Aug 2016 (bib)
	Ma Meng, P. Fallavollita, S. Habert, Simon Weider, N. Navab Device and System Independent Personal Touchless User Interface for Operating Rooms International Conference on Information Processing in Computer-Assisted Interventions (IPCAI), 2016 (bib)
	P. Fallavollita, A. Brand, L. Wang, E. Euler, P. Thaller, N. Navab, S. Weidert An augmented reality C-arm for intraoperative assessment of the mechanical axis - a preclinical study International Journal of Computer Assisted Radiology and Surgery (bib)
	Xiang Wang, S. Habert, Ma Meng, C.-H. Huang, P. Fallavollita, N. Navab Precise 3D/2D calibration between a RGB-D sensor and a C-arm fluoroscope International Journal of Computer Assisted Radiology and Surgery (bib)
	Ma Meng, P. Fallavollita, Ina Seelbach, A. von der Heide, E. Euler, J. Waschke, N. Navab Personalized augmented reality for anatomy education Clinical Anatomy (bib)
	P. Fallavollita, L. Wang, S. Weidert, N. Navab Augmented Reality in Orthopaedic Interventions and Education Computational Radiology for Orthopaedic Interventions, Springer International Publishing 2016 (bib)
2015
	M. Zweng, P. Fallavollita, S. Demirci, M. Kowarschik, N. Navab, D. Mateus Automatic Guide-Wire Detection for Neurointerventions Using Low-Rank Sparse Matrix Decomposition and Denoising MICCAI 2015 Workshop on Augmented Environments for Computer-Assisted Interventions (bib)
	Hagen Kaiser, P. Fallavollita, N. Navab Real-Time Markerless Respiratory Motion Management Using Thermal Sensor Data MICCAI 2015 Workshop on Augmented Environments for Computer-Assisted Interventions (bib)
	Hagen Kaiser, P. Fallavollita, N. Navab 'On the Fly' Reconstruction and Tracking System for Patient Setup in Radiation Therapy MICCAI 2015 Workshop on Augmented Environments for Computer-Assisted Interventions (bib)
	R. Londei, M. Esposito, B. Diotte, S. Weidert, E. Euler, P. Thaller, N. Navab, P. Fallavollita Intra-operative augmented reality in distal locking The International Journal for Computer Assisted Radiology and Surgery (IJCARS) (bib)
	Xiang Wang, S. Habert, Ma Meng, C.-H. Huang, P. Fallavollita, N. Navab RGB-D/C-arm Calibration and Application in Medical Augmented Reality International Symposium on Mixed and Augmented Reality (ISMAR), 2015 (bib)
	Ma Meng, , P. Fallavollita, N. Navab Natural user interface for ambient objects International Symposium on Mixed and Augmented Reality (ISMAR), 2015 (bib)
	N. Leucht, S. Habert, P. Wucherer, S. Weidert, N. Navab, P. Fallavollita Augmented Reality for Radiation Awareness International Symposium on Mixed and Augmented Reality (ISMAR), 2015 (bib)
	S. Habert, Ma Meng, W. Kehl, Xiang Wang, F. Tombari, P. Fallavollita, N. Navab Augmenting mobile C-arm fluoroscopes via Stereo-RGBD sensors for multimodal visualization International Symposium on Mixed and Augmented Reality (ISMAR), 2015 (bib)