ISMAR 2007 Tutorial in Computer Vision for AR: Rigid and Deformable Tracking using Markers or Scene Features

Summary
Program
Organizers
Additional Speakers

Summary

In this tutorial, we aim to give a survey of the most recent developments on visual tracking in Augmented Reality applications. First, latest marker-based approaches will be recalled and discussed. Then, we will focus on marker-less concepts for tracking using features and/or image alignments. Once the newest methods have been introduced, we will see how computer vision and machine learning techniques can be combined in order to improve the quality and the efficiency of the detection and the tracking. Finally, after having considered the camera environment as rigid, performing virtual augmentation on deformable objects in the scene will be described.

Program

Date & Time: November 13, 2007 at 09:15-13:15
Location: Nara-Ken New Public Hall, Nara, Japan (details will be provided soon)

09:15 - 09:30 Prof. Nassir Navab Introduction Problem Statement in Industrial and Medical AR.

09:35 - 10:20 Dr. Mark Fiala Digital Techniques for Fiducial Marker-based tracking Fiducial markers are objects inserted into a scene to enable the pose tracking required in augmented reality. Using planar patterns and passive computer vision removes the need for specialized tracking hardware, and allows low cost scalable systems to be available to the average user. However, reliable AR requires the careful design of both the patterns and the computer vision for robust tracking. The use of lighting invariant image processing and optimized digital coding in the ARTag system is shown, along with several AR implementation examples.

10:10 - 10:55 Dr. Selim Benhimane General Methods for Feature- and Intensity-based tracking In some AR applications, the use of markers should be avoided. The tracking, permitting the determination of the camera position with respect to the scene observed, is then based either on a set of features extracted from the images or on the direct use of the pixel intensities for image alignment. In this course, we will give an overview of the newest methods of such marker-less tracking. We will detail the advantages, the limitations and the different applications of each approach.

11:15 - 11:30 Coffee Break

11:30 - 12:15 Dr. Vincent Lepetit Randomized Trees and Ferns for Detection, Tracking and Pose Estimation As described in the previous course, 3D tracking approaches can achieve very high accuracy even without markers. However, this often comes at the price of lack of robustness: The system must either be initialized by hand or require the camera to be very close to a specified position, and is very fragile if something goes wrong between two consecutive frames. A solution is to rely on feature point recognition to initialize and reinitialize the tracking system but such approach can be computationally expensive. In this course, we will review state-of-the-art methods for feature point recognition and describe a fast method suitable for interactive AR applications based on an extended version of Randomized Trees.

12:20 - 13:05 Dr. Adrien Bartoli Tracking and 3D Reconstruction for Deformable Surfaces This part of the tutorial focuses on the case of deformable surfaces such as clothes, rugs and paper sheets. It is organized into two main parts: image-based tracking and 3D reconstruction from multiple cameras. Computing the camera position and orientation from a monocular video stream of a deforming surface is in general ill-posed. 2D registration enables surface augmentation, though. The basic and most generic assumption that we make is surface and deformation smoothness. We show how feature-based and direct image registration algorithms can be formulated. The surface occlusion problem will be touched. Finally, we briefly show recent results on generic, non-rigid Structure-from-Motion based on the low-rank shape model and the non-rigid factorization paradigm.

Organizers

Selim BenHimane, CAMP, Technische Universität München (Germany)

Selim Benhimane obtained the degree of Engineering systems, Automation and Vision from the National School of Higher Education in Physics of Strasbourg, France in 2002. In the same year, he received a postgraduate advanced diploma in Photonics, Imaging et Cybernetics from Louis Pasteur University of Strasbourg, France. Then, he prepared his doctor thesis at the ICARE team of INRIA Sophia Antipolis and obtained his Ph.D. from the École Nationale Supérieure des Mines de Paris, France. Since 2005, he is a Research Associate at the CAMP-AR Chair of Prof. Nassir Navab in TU München, Germany. His research interests include computer vision, robotics, vision-based control.

Vincent Lepetit, CVlab, École Polytechnique Fédérale de Lausanne (Switzerland)

Vincent Lepetit received the engineering and master degrees in Computer Science from the ESIAL in 1996. He received the Ph.D. degree in Computer Vision in 2001 from the University of Nancy, France, after working in the ISA INRIA team. He then joined the Virtual Reality Lab at EPFL (Swiss Federal Institute of Technology) as a post-doctoral fellow and became a founding member of the Computer Vision Laboratory. He has received several awards in computer vision including the best paper award at CVPR 2005. His research interests include Augmented Reality, 3D camera tracking, and object recognition.

Adrien Bartoli, LASMEA, Université Blaise Pascal (France)

Adrien Bartoli is a permanent CNRS research scientist at the LASMEA laboratory in Clermont-Ferrand, France, since October 2004 and a visiting professor at DIKU in Copenhagen, Denmark for 2006-2009. Before that, he was a postdoctoral researcher at the University of Oxford, UK, in the Visual Geometry Group, under the supervisation of Prof. Andrew Zisserman. He did his Ph.D. in the Perception group, in Grenoble at INRIA, France, under the supervisation of Dr. Peter Sturm and Prof. Radu Horaud. Since September 2006, he is co-leading the Com-See research team. His main research interests are in structure-from-motion for rigid and non-rigid scenarios, in the field of computer vision.

Additional Speakers

Mark Fiala, IIT, National Research Council (Canada)

Nassir Navab, CAMP, Technische Universität München (Germany)

Nassir Navab is a full professor and director of the institute for Computer Aided Medical Procedures and Augmented Reality (CAMP-AR) at TU München. He has also a secondary faculty appointment at the Medical School of the TU München. He was recently elected as a member of board of directors of MICCAI society. He has been serving on the Steering Committee of the ISMAR since 2001. Before joining the TU München, he was a distinguished member at Siemens Corporate Research in Princeton, New Jersey. He received the prestigious Siemens Inventor of the Year Award in 2001 for his work in interventional imaging. He has served on the program committee of over 20 international conferences. He is the author of over hundred scientific papers and over 40 US and international patents. His main fields of interest include: computer vision, Augmented Reality, computer-aided surgery and medical image registration.

09:15 - 09:30	Prof. Nassir Navab	Introduction	Problem Statement in Industrial and Medical AR.
09:35 - 10:20	Dr. Mark Fiala	Digital Techniques for Fiducial Marker-based tracking	Fiducial markers are objects inserted into a scene to enable the pose tracking required in augmented reality. Using planar patterns and passive computer vision removes the need for specialized tracking hardware, and allows low cost scalable systems to be available to the average user. However, reliable AR requires the careful design of both the patterns and the computer vision for robust tracking. The use of lighting invariant image processing and optimized digital coding in the ARTag system is shown, along with several AR implementation examples.
10:10 - 10:55	Dr. Selim Benhimane	General Methods for Feature- and Intensity-based tracking	In some AR applications, the use of markers should be avoided. The tracking, permitting the determination of the camera position with respect to the scene observed, is then based either on a set of features extracted from the images or on the direct use of the pixel intensities for image alignment. In this course, we will give an overview of the newest methods of such marker-less tracking. We will detail the advantages, the limitations and the different applications of each approach.
11:15 - 11:30		Coffee Break
11:30 - 12:15	Dr. Vincent Lepetit	Randomized Trees and Ferns for Detection, Tracking and Pose Estimation	As described in the previous course, 3D tracking approaches can achieve very high accuracy even without markers. However, this often comes at the price of lack of robustness: The system must either be initialized by hand or require the camera to be very close to a specified position, and is very fragile if something goes wrong between two consecutive frames. A solution is to rely on feature point recognition to initialize and reinitialize the tracking system but such approach can be computationally expensive. In this course, we will review state-of-the-art methods for feature point recognition and describe a fast method suitable for interactive AR applications based on an extended version of Randomized Trees.
12:20 - 13:05	Dr. Adrien Bartoli	Tracking and 3D Reconstruction for Deformable Surfaces	This part of the tutorial focuses on the case of deformable surfaces such as clothes, rugs and paper sheets. It is organized into two main parts: image-based tracking and 3D reconstruction from multiple cameras. Computing the camera position and orientation from a monocular video stream of a deforming surface is in general ill-posed. 2D registration enables surface augmentation, though. The basic and most generic assumption that we make is surface and deformation smoothness. We show how feature-based and direct image registration algorithms can be formulated. The surface occlusion problem will be touched. Finally, we briefly show recent results on generic, non-rigid Structure-from-Motion based on the low-rank shape model and the non-rigid factorization paradigm.