research-article

KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera

Authors:

Otmar Hilliges,

David Molyneaux,

Richard Newcombe,

Pushmeet Kohli,

Dustin Freeman,

Andrew Davison, and

Andrew FitzgibbonAuthors Info & Claims

UIST '11: Proceedings of the 24th annual ACM symposium on User interface software and technology

October 2011

Pages 559 - 568

https://doi.org/10.1145/2047196.2047270

Published: 16 October 2011 Publication History

Abstract

KinectFusion enables a user holding and moving a standard Kinect camera to rapidly create detailed 3D reconstructions of an indoor scene. Only the depth data from Kinect is used to track the 3D pose of the sensor and reconstruct, geometrically precise, 3D models of the physical scene in real-time. The capabilities of KinectFusion, as well as the novel GPU-based pipeline are described in full. Uses of the core system for low-cost handheld scanning, and geometry-aware augmented reality and physics-based interactions are shown. Novel extensions to the core GPU pipeline demonstrate object segmentation and user interaction directly in front of the sensor, without degrading camera tracking or reconstruction. These extensions are used to enable real-time multi-touch interactions anywhere, allowing any planar or non-planar reconstructed physical surface to be appropriated for touch.

References

[1]

P. J. Besl and N. D. McKay. A method for registration of 3D shapes. IEEE Trans. Pattern Anal. Mach. Intell., 14:239--256, February 1992.

Digital Library

[2]

X. Cao and R. Balakrishnan. Interacting with dynamically defined information spaces using a handheld projector and a pen. In UIST, pages 225--234, 2006.

Digital Library

[3]

Y. Chen and G. Medioni. Object modeling by registration of multiple range images. Image and Vision Computing (IVC), 10(3):145--155, 1992.

Digital Library

[4]

Y. Cui et al. 3d shape scanning with a time-of-flight camera. In Computer Vision and Pattern Recognition (CVPR), pages 1173 --1180, June 2010.

[5]

B. Curless and M. Levoy. A volumetric method for building complex models from range images. ACM Trans. Graph., 1996.

Digital Library

[6]

S. Farsiu et al. Fast and robust multiframe super resolution. IEEE Transactions on Image Processing, 13(10):1327--1344, 2004.

Digital Library

[7]

J. Frahm et al. Building Rome on a cloudless day. In Proc. Europ. Conf. on Computer Vision (ECCV), 2010.

Digital Library

[8]

B. Freedman, A. Shpunt, M. Machline, and Y. Arieli. Depth Mapping Using Projected Patterns. Patent Application, 10 2008. WO 2008/120217 A2.

[9]

S. L. Grand. Broad-phase collision detection with CUDA. In GPU Gems 3. Addison-Wesley, 2007.

[10]

T. Harada. Real-time rigid body simulation on gpus. In GPU Gems 3. Addison-Wesley Professional, 2007.

[11]

R. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, second edition, 2004.

Digital Library

[12]

P. Henry et al. RGB-D mapping: Using depth cameras for dense 3D modeling of indoor environments. In Proc. of the Int. Symposium on Experimental Robotics (ISER), 2010.

[13]

B. Huhle et al. Fusion of range and color images for denoising and resolution enhancement with a non-local filter. Computer Vision and Image Understanding, 114(12):1336--1345, 2010.

Digital Library

[14]

M. Kazhdan, M. Bolitho, and H. Hoppe. Poisson surface reconstruction. In Proc. of the Eurographics Symposium on Geometry Processing, 2006.

Digital Library

[15]

G. Klein and D. W. Murray. Parallel tracking and mapping for small AR workspaces. In ISMAR, 2007.

Digital Library

[16]

M. Levoy et al. The digital Michelangelo Project: 3D scanning of large statues. ACM Trans. Graph., 2000.

Digital Library

[17]

K. Low. Linear least-squares optimization for point-to-plane icp surface registration. Technical report, TR04-004, University of North Carolina, 2004.

[18]

P. Merrell et al. Real-time visibility-based fusion of depth maps. In Proc. of the Int. Conf. on Computer Vision (ICCV), 2007.

[19]

R. A. Newcombe and A. J. Davison. Live dense reconstruction with a single moving camera. In Proc. of the IEEE (CVPR), 2010.

[20]

R. A. Newcombe, S. Lovegrove, and A. J. Davison. Dense tracking and mapping in real-time. In Proc. of the Int. Conf. on Computer Vision (ICCV), 2011.

Digital Library

[21]

R. A. Newcombe et al. Real-Time Dense Surface Mapping and Tracking with Kinect. In ISMAR, 2011.

Digital Library

[22]

S. Osher and R. Fedkiw. Level Set Methods and Dynamic Implicit Surfaces. Springer, 2002.

[23]

S. Rusinkiewicz, O. Hall-Holt, and M. Levoy. Real-time 3D model acquisition. ACM Trans. Graph., 2002.

Digital Library

[24]

S. Rusinkiewicz and M. Levoy. Efficient variants of the ICP algorithm. 3D Digital Imaging and Modeling, Int. Conf. on, 0:145, 2001.

[25]

S. Thrun. Robotic mapping: A survey. In Exploring Artificial Intelligence in the New Millenium. 2002.

Digital Library

[26]

D. Vlasic et al. Dynamic shape capture using multi-view photometric stereo. ACM Trans. Graph., 28(5), 2009.

Digital Library

[27]

D. Wagner, T. Langlotz, and D. Schmalstieg. Robust and unobtrusive marker tracking on mobile phones. In ISMAR, pages 121--124, 2008.

Digital Library

[28]

T. Weise, T. Wismer, B. Leibe, and L. V. Gool. In-hand scanning with online loop closure. In IEEE Int. Workshop on 3-D Digital Imaging and Modeling, 2009.

[29]

K. Zhou, M. Gong, X. Huang, and B. Guo. Data-parallel octrees for surface reconstruction. IEEE Trans. on Visualization and Computer Graphics, 17, 2011.

Digital Library

Cited By

Kamran-Pishhesari AMoniri-Morad ASattarvand J(2024)Applications of 3D Reconstruction in Virtual Reality-Based Teleoperation: A Review in the Mining IndustryTechnologies10.3390/technologies1203004012:3(40)Online publication date: 15-Mar-2024
https://doi.org/10.3390/technologies12030040
Mora PGarcia CIvorra EOrtega MAlcañiz M(2024)Virtual Experience Toolkit: An End-to-End Automated 3D Scene Virtualization Framework Implementing Computer Vision TechniquesSensors10.3390/s2412383724:12(3837)Online publication date: 13-Jun-2024
https://doi.org/10.3390/s24123837
Zhuang LZhong XXu LTian CYu W(2024)Visual SLAM for Unmanned Aerial Vehicles: Localization and PerceptionSensors10.3390/s2410298024:10(2980)Online publication date: 8-May-2024
https://doi.org/10.3390/s24102980
Show More Cited By

Index Terms

KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Image and video acquisition
        3D imaging
  2. Computer graphics
    1. Animation

Recommendations

KinectFusion: real-time dynamic 3D surface reconstruction and interaction
SIGGRAPH '11: ACM SIGGRAPH 2011 Talks

We present KinectFusion, a system that takes live depth data from a moving Kinect camera and in real-time creates high-quality, geometrically accurate, 3D models. Our system allows a user holding a Kinect camera to move quickly within any indoor space, ...
Read More
Heterogeneous Sensor Data Fusion: How Many Cameras are Needed for an Accurate 3D Reconstruction of Indoor Scene?
ISCID '14: Proceedings of the 2014 Seventh International Symposium on Computational Intelligence and Design - Volume 02

Vision systems, consisting of conventional color cameras and depth cameras have proved capable of solving the problem of indoor scene reconstruction with sufficient detail and satisfactory accuracy. However, there are few guidelines for how many cameras ...
Read More
Real-Time High Resolution Fusion of Depth Maps on GPU
CADGRAPHICS '13: Proceedings of the 2013 International Conference on Computer-Aided Design and Computer Graphics

A system for live high quality surface reconstruction using a single moving depth camera on a commodity hardware is presented. High accuracy and real-time frame rate is achieved by utilizing graphics hardware computing capabilities via OpenCLTM and by ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '11: Proceedings of the 24th annual ACM symposium on User interface software and technology

October 2011

654 pages

ISBN:9781450307161

DOI:10.1145/2047196

General Chair:
Jeff Pierce
IBM Research, USA
,
Program Chairs:
Maneesh Agrawala
University of California, Berkeley, USA
,
Scott Klemmer
Stanford University, USA

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 October 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

UIST '11

Sponsor:

UIST '11: The 24th Annual ACM Symposium on User Interface Software and Technology

October 16 - 19, 2011

California, Santa Barbara, USA

Acceptance Rates

UIST '11 Paper Acceptance Rate 67 of 262 submissions, 26%;

Overall Acceptance Rate 842 of 3,967 submissions, 21%

Upcoming Conference

UIST '24

Sponsor:
sigchi
sigchi

The 37th Annual ACM Symposium on User Interface Software and Technology

October 13 - 16, 2024

Pittsburgh , PA , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1,489
Total Citations
View Citations
17,258
Total Downloads

Downloads (Last 12 months)701
Downloads (Last 6 weeks)55

Other Metrics

View Author Metrics

Citations

Cited By

Kamran-Pishhesari AMoniri-Morad ASattarvand J(2024)Applications of 3D Reconstruction in Virtual Reality-Based Teleoperation: A Review in the Mining IndustryTechnologies10.3390/technologies1203004012:3(40)Online publication date: 15-Mar-2024
https://doi.org/10.3390/technologies12030040
Mora PGarcia CIvorra EOrtega MAlcañiz M(2024)Virtual Experience Toolkit: An End-to-End Automated 3D Scene Virtualization Framework Implementing Computer Vision TechniquesSensors10.3390/s2412383724:12(3837)Online publication date: 13-Jun-2024
https://doi.org/10.3390/s24123837
Zhuang LZhong XXu LTian CYu W(2024)Visual SLAM for Unmanned Aerial Vehicles: Localization and PerceptionSensors10.3390/s2410298024:10(2980)Online publication date: 8-May-2024
https://doi.org/10.3390/s24102980
Wang DHao HZhang J(2024)HALNet: Partial Point Cloud Registration Based on Hybrid Attention and Deep Local FeaturesSensors10.3390/s2409276824:9(2768)Online publication date: 26-Apr-2024
https://doi.org/10.3390/s24092768
Zhou LWu GZuo YChen XHu H(2024)A Comprehensive Review of Vision-Based 3D Reconstruction MethodsSensors10.3390/s2407231424:7(2314)Online publication date: 5-Apr-2024
https://doi.org/10.3390/s24072314
Wang RWang JLi YMa TZhang X(2024)Research Advances and Prospects of Underwater Terrain-Aided NavigationRemote Sensing10.3390/rs1614256016:14(2560)Online publication date: 12-Jul-2024
https://doi.org/10.3390/rs16142560
Nguyen TLe VDo HTe TPhan V(2024)TQU-SLAM Benchmark Dataset for Comparative Study to Build Visual Odometry Based on Extracted Features from Feature Descriptors and Deep LearningFuture Internet10.3390/fi1605017416:5(174)Online publication date: 17-May-2024
https://doi.org/10.3390/fi16050174
Liu LLiu YLv YLi X(2024)A Novel Approach for Simultaneous Localization and Dense Mapping Based on Binocular Vision in Forest Ecological EnvironmentForests10.3390/f1501014715:1(147)Online publication date: 10-Jan-2024
https://doi.org/10.3390/f15010147
Liu PZhong LWang RZhu JZhai XZhang J(2024)GraM: Geometric Structure Embedding into Attention Mechanisms for 3D Point Cloud RegistrationElectronics10.3390/electronics1310199513:10(1995)Online publication date: 20-May-2024
https://doi.org/10.3390/electronics13101995
Ezhilarasu APey JMuthugala MBudig MElara M(2024)Enhancing Robot Inclusivity in the Built Environment: A Digital Twin-Assisted Assessment of Design Guideline ComplianceBuildings10.3390/buildings1405119314:5(1193)Online publication date: 23-Apr-2024
https://doi.org/10.3390/buildings14051193
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents

-