NARVIS - navigated augmented reality visualization system Advanced visualization is getting increasingly important for the operation room of the future. The increasing number of available medical images must be presented to surgery team in new ways in order to support them rather than overloading with more information. In our project NARVIS we integrate an HMD-based (head mounted display) AR system into the operation room for 3D in situ visualization of computed tomography (CT) images. The final system aims at spinal surgery. The work is in close collaboration with our project partners “Klinikum für Unfallchirgie” at LMU, A.R.T. Weilheim, and Siemens Corporate Research in Princeton. The project is funded by Bayerische Forschungsstiftung.

NARVIS

In a phantom experiment we have used a 3D augmented reality (AR) workspace, in which the user wears a video-feed-through head-mounted display to perform operative tasks in thoracoscopic spine surgery. The set up was used to observe and fuse preoperative 3D CT-data and intraoperative 3D fluoroscopy-data as well as 2D endoscopic images in-situ, i.e. in their actual location, overlaid onto a stereoscopic video view. In-situ visualization of rendered three-dimensional volumes (e.g. implants, vascular and neural structures) facilitates the understanding of the object's three-dimensional structure. Furthermore, the system has the ability to track hand-held surgical instruments in 2 different techniques. Graphically enhanced instruments improve the perception of tool position and orientation in respect to targets in the rendered volume. With an intuitive augmented reality user interface the surgeon controls the visualization of the volumetric data. In the phantom experiment the system has proven to effectively enhance information in minimally-invasive spinal surgery.



NARVIS sketch

Group coordinator

Tobias Sielhorst

Related publications

	A. Winkler, U. Eck, N. Navab Spatially-Aware Displays for Computer Assisted Interventions Proceedings of the 23rd International Conference on Medical Image Computing and Computer Assisted Interventions (MICCAI), Lima, October 2020 (bib)
	F. Bork, A. Lehner, D. Kugelmann, U. Eck, J. Waschke, N. Navab VesARlius: An Augmented Reality System for Large-Group Co-Located Anatomy Learning International Symposium on Mixed and Augmented Reality (ISMAR), 2019 (bib)
	F. Bork, U. Eck, N. Navab Birds vs. Fish: Visualizing Out-Of-View Objects in Augmented Reality using 3D Minimaps International Symposium on Mixed and Augmented Reality (ISMAR), 2019 (bib)
	F. Bork, A. Schnelzer, U. Eck, N. Navab Towards Efficient Visual Guidance in Limited Field-of-View Head-Mounted Displays IEEE Transactions on Visualization and Computer Graphics (bib)
	F. Bork, R. Barmaki, U. Eck, K. Yu, C. Sandor, N. Navab Empirical Study of Non-Reversing Magic Mirrors for Augmented Reality Anatomy Learning International Symposium on Mixed and Augmented Reality (ISMAR), 2017 (bib)
	Alana Da Gama, Thiago Menezes Chaves, Lucas Silva Figueiredo, Adriana Baltar, Ma Meng, N. Navab, Veronica Teichrieb, P. Fallavollita MirrARbilitation: A clinically-related gesture recognition interactive tool for an AR rehabilitation system Computer Methods and Programs in Biomedicine (bib)
	Xiang Wang, S. Habert, C. Schulte-zu-Berge, P. Fallavollita, N. Navab Inverse visualization concept for RGB-D augmented C-arms Computers in Biology and Medicine (bib)
	Ma Meng, P. Fallavollita, S. Habert, Simon Weider, N. Navab Device and System Independent Personal Touchless User Interface for Operating Rooms International Conference on Information Processing in Computer-Assisted Interventions (IPCAI), 2016 (bib)
	P. Fallavollita, A. Brand, L. Wang, E. Euler, P. Thaller, N. Navab, S. Weidert An augmented reality C-arm for intraoperative assessment of the mechanical axis - a preclinical study International Journal of Computer Assisted Radiology and Surgery (bib)
	Xiang Wang, S. Habert, Ma Meng, C.-H. Huang, P. Fallavollita, N. Navab Precise 3D/2D calibration between a RGB-D sensor and a C-arm fluoroscope International Journal of Computer Assisted Radiology and Surgery (bib)
	Ma Meng, P. Fallavollita, Ina Seelbach, A. von der Heide, E. Euler, J. Waschke, N. Navab Personalized augmented reality for anatomy education Clinical Anatomy (bib)
	P. Fallavollita, L. Wang, S. Weidert, N. Navab Augmented Reality in Orthopaedic Interventions and Education Computational Radiology for Orthopaedic Interventions, Springer International Publishing 2016 (bib)
	Xiang Wang, S. Habert, Ma Meng, C.-H. Huang, P. Fallavollita, N. Navab RGB-D/C-arm Calibration and Application in Medical Augmented Reality International Symposium on Mixed and Augmented Reality (ISMAR), 2015 (bib)
	Ma Meng, , P. Fallavollita, N. Navab Natural user interface for ambient objects International Symposium on Mixed and Augmented Reality (ISMAR), 2015 (bib)
	N. Leucht, S. Habert, P. Wucherer, S. Weidert, N. Navab, P. Fallavollita Augmented Reality for Radiation Awareness International Symposium on Mixed and Augmented Reality (ISMAR), 2015 (bib)
	S. Habert, Ma Meng, W. Kehl, Xiang Wang, F. Tombari, P. Fallavollita, N. Navab Augmenting mobile C-arm fluoroscopes via Stereo-RGBD sensors for multimodal visualization International Symposium on Mixed and Augmented Reality (ISMAR), 2015 (bib)
	S. Habert, J. Gardiazabal, P. Fallavollita, N. Navab RGBDX: first design and experimental validation of a mirror-based RGBD Xray imaging system International Symposium on Mixed and Augmented Reality (ISMAR), 2015 (bib)
	F. Bork, B. Fuerst, A.K. Schneider, F. Pinto, C. Graumann, N. Navab Auditory and Visio-Temporal Distance Coding for 3-Dimensional Perception in Medical Augmented Reality International Symposium on Mixed and Augmented Reality (ISMAR), 2015 (bib)
	F. Milletari, V. Belagiannis, N. Navab, P. Fallavollita Fully automatic catheter localization in C-arm images using l1- Sparse Coding Proceedings of the 17th International Conference on Medical Image Computing and Computer Assisted Interventions (MICCAI), Boston, September 2014 (bib)
	P. Fallavollita, A. Winkler, S. Habert, P. Wucherer, P. Stefan, Riad Mansour, R. Ghotbi, N. Navab 'Desired-View' controlled positioning of angiographic C-arms Proceedings of the 17th International Conference on Medical Image Computing and Computer Assisted Interventions (MICCAI), Boston, September 2014 (bib)
	Xiang Wang, S. Demirci, C. Schulte-zu-Berge, P. Fallavollita, N. Navab Improved Interventional X-ray Appearance International Symposium on Mixed and Augmented Reality (ISMAR), 2014 (bib)
	Alana Da Gama, P. Fallavollita, Veronica Teichrieb, N. Navab Motor rehabilitation using Kinect: a systematic review Games for Health Journal (bib)
	Jian Wang, L. Wang, Matthias Kreiser, N. Navab, P. Fallavollita Augmented Depth Perception Visualization in 2D/3D Image Fusion Computerized Medical Imaging and Graphics (bib)
	X. Chen, Hemal Naik, L. Wang, N. Navab, P. Fallavollita Video Guided Calibration of an Augmented Reality mobile C-arm International Journal of Computer Assisted Radiology and Surgery (IJCARS) (bib)
	P. Fallavollita Detection, Tracking and Related Costs of Ablation Catheters in the Treatment of Cardiac Arrhythmias Cardiac Arrhythmias - Mechanisms, Pathophysiology, and Treatment, Publisher: INTECH, Editor: Wilbert S. Aronow (bib)
	P. Stefan, P. Wucherer, Y. Oyamada, Ma Meng, A. Schoch, , , , , M. Weigl, , P. Fallavollita, H. Saito, N. Navab An AR Edutainment System Supporting Bone Anatomy Learning IEEE Virtual Reality (bib)
	C. Amat di San Filippo, G. Fichtinger, William Morris, Tim Salcudean, E. Dehghan, P. Fallavollita Intraoperative Segmentation of Iodine and Palladium Radioactive Sources in C-arm images International Journal for Computer Assisted Radiology and Surgery (IJCARS) (bib)
	P. Thaller, B. Diotte, P. Fallavollita, L. Wang, S. Weidert, E. Euler, W. Mutschler, N. Navab Video-assisted interlocking of intramedullary nails Injury, 2013 (bib)
	P. Thaller, B. Diotte, P. Fallavollita, L. Wang, S. Weidert, W. Mutschler, N. Navab, E. Euler Video-navigated targeting unit for interlocking of intramedullary nails – experimental study on a new technique. Kongress der Gesellschaft für Extremitätenverlängerung und Rekonstruktion, 2013 (bib)
	C. Amat di San Filippo, G. Fichtinger, William Morris, Tim Salcudean, E. Dehghan, P. Fallavollita Declustering n-connected components: an example case for the segmentation of iodine implants in C-arm images 4th International Conference on Information Processing in Computer-Assisted Interventions (IPCAI), Heidelberg, Germany, June 26, 2013 (bib)
	S. Weidert, P. Wucherer, P. Stefan, P. Fallavollita, E. Euler, N. Navab Vertebroplasty medical simulation learning environment International Society for Computer Assisted Orthopaedic Surgery, Orlando, June 2013 (bib)
	S. Pati, O. Erat, L. Wang, S. Weidert, E. Euler, N. Navab, P. Fallavollita Accurate pose estimation using a single marker camera calibration system SPIE Medical Imaging, Orlando, Florida, USA, February 2013 (bib)
	O. Erat, O. Pauly, S. Weidert, P. Thaller, E. Euler, W. Mutschler, N. Navab, P. Fallavollita How a Surgeon becomes Superman by visualization of intelligently fused multi-modalities SPIE Medical Imaging, Orlando, Florida, USA, February 2013 (bib)
	Ma Meng, P. Fallavollita, T. Blum, U. Eck, C. Sandor, S. Weidert, J. Waschke, N. Navab Kinect for Interactive AR Anatomy Learning The 12th IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR), Adelaide, Australia, Oct. 1 - 4, 2013. (bib)
	Y. Oyamada, P. Fallavollita, N. Navab Single Camera Calibration using partially visible calibration objects based on Random Dots Marker Tracking Algorithm The IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR), Workshop on Tracking Methods and Applications, 2012 (bib)
	Jian Wang, P. Fallavollita, L. Wang, Matthias Kreiser, N. Navab Augmented Reality during Angiography: Integration of a Virtual Mirror for Improved 2D/3D Visualization The 11th IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR), Atlanta, USA, Nov. 5 - 8, 2012. (bib)
	O. Pauly, A. Katouzian, A. Eslami, P. Fallavollita, N. Navab Supervised Classification for Customized Intraoperative Augmented Reality Visualization The 11th IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR), Atlanta, USA, Nov. 5 - 8, 2012. (bib)
	T. Blum, R. Stauder, E. Euler, N. Navab Superman-like X-ray Vision: Towards Brain-Computer Interfaces for Medical Augmented Reality The 11th IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR), Atlanta, USA, Nov. 5 - 8, 2012. The original publication is available online at ieee.org. (bib)
	A. Aichert, M. Wieczorek, Jian Wang, L. Wang, Matthias Kreiser, P. Fallavollita, N. Navab The Colored X-rays MICCAI 2012 Workshop on Augmented Environments for Computer-Assisted Interventions (bib)
	L. Wang, P. Fallavollita, A. Brand, O. Erat, S. Weidert, P. Thaller, E. Euler, N. Navab Intra-op measurement of the mechanical axis deviation: an evaluation study on 19 human cadaver legs Proceedings of the 15th International Conference on Medical Image Computing and Computer Assisted Interventions (MICCAI), Nice, France, October 2012 (bib)
	E. Dehghan, J. Lee, P. Fallavollita, N. Kuo, A. Deguet, Yi Le, C. Burdette, D. Song, J. Prince, G. Fichtinger Ultrasound-Fluoroscopy Registration for Prostate Brachytherapy Dosimetry Medical Image Analysis, 2012 (bib)
	A. Brand, L. Wang, P. Fallavollita, Philip Sandner, N. Navab, P. Thaller, E. Euler, S. Weidert Intraoperative measurement of mechanical axis alignment by automatic image stitching: a human cadaver study International Society for Computer Assisted Orthopaedic Surgery (CAOS), South Korea, June 2012 (bib)
	P. Fallavollita The Future of Cardiac Mapping Cardiac Arrhythmias - New Considerations, Publisher: INTECH, Editor: Francisco R. Breijo-Marquez (bib)
	C. Wachinger, M. Yigitsoy, E. Rijkhorst, N. Navab Manifold Learning for Image-Based Breathing Gating in Ultrasound and MRI Medical Image Analysis, Volume 16, Issue 4, May 2012, Pages 806–818. (bib)
	J. Traub New Concepts for Design and Workflow Driven Evaluation of Computer Assisted Surgery Solutions Dissertation an der Fakultät für Informatik, Technische Universität München, 2008 ( online version available here) (bib)
	T. Sielhorst New Methods for Medical Augmented Reality Dissertation an der Fakultät für Informatik, Technische Universität München, 2008 ( online version available here) (bib)
	T. Sielhorst, Wu Sa, A. Khamene, F. Sauer, N. Navab Measurement of absolute latency for video see through augmented reality Sixth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'07), Nara, Japan, November 2007 (bib)
	T. Sielhorst, M. Bauer, O. Wenisch, G. Klinker, N. Navab Online Estimation of the Target Registration Error for n-ocular Optical Tracking Systems to appear Proceedings of Medical Image Computing and Computer-Assisted Intervention (MICCAI 2007), Brisbane, Australia, October 2007, pp. 652-659. (bib)
	T. Sielhorst, C. Bichlmeier, S.M. Heining, N. Navab Depth perception a major issue in medical AR: Evaluation study by twenty surgeons Proceedings of Medical Image Computing and Computer-Assisted Intervention (MICCAI 2006), Copenhagen, Denmark, October 2006, pp. 364-372 The original publication is available online at www.springerlink.com (bib)
	T. Sielhorst, M. Feuerstein, J. Traub, O. Kutter, N. Navab CAMPAR: A software framework guaranteeing quality for medical augmented reality International Journal of Computer Assisted Radiology and Surgery, 2006; Vol. 1 Suppl. 1, page 29 - 30. The original publication is available online at www.springerlink.com (bib)