article

Unstructured Light Fields

Authors:

Marc Levoy, and

Fredo DurandAuthors Info & Claims

Computer Graphics Forum, Volume 31, Issue 2pt1

Pages 305 - 314

https://doi.org/10.1111/j.1467-8659.2012.03009.x

Published: 01 May 2012 Publication History

Abstract

We present a system for interactively acquiring and rendering light fields using a hand-held commodity camera. The main challenge we address is assisting a user in achieving good coverage of the 4D domain despite the challenges of hand-held acquisition. We define coverage by bounding reprojection error between viewpoints, which accounts for all 4 dimensions of the light field. We use this criterion together with a recent Simultaneous Localization and Mapping technique to compute a coverage map on the space of viewpoints. We provide users with real-time feedback and direct them toward under-sampled parts of the light field. Our system is lightweight and has allowed us to capture hundreds of light fields. We further present a new rendering algorithm that is tailored to the unstructured yet dense data we capture. Our method can achieve piecewise-bicubic reconstruction using a triangulation of the captured viewpoints and subdivision rules applied to reconstruction weights. © 2012 Wiley Periodicals, Inc.

References

[1]

Aliaga D. G., Funkhouser T., Yanovsky D., Carlbom I.: Sea of images. In Proc IEEE VIS-02 (2002), pp. 331–338. 1

Digital Library

[2]

Adelson E. H., Wang J. Y. A.: Single lens stereo with a plenoptic camera. IEEE PAMI 14, 2 (Feb. 1992), 99–106. 2

Digital Library

[3]

Buehler C., Bosse M., McMillan L., Gortler S., Cohen M.: Unstructured lumigraph rendering. In SIGGRAPH '01 (2001), ACM, pp. 425–432. 3, 5, 7

Digital Library

[4]

Blake A., Torr P. H. S., Cox I. I., Criminisi A.: Estimating uncertainty in dense stereo disparity maps. Tech. Rep. MSR-TR-2003–93, Microsoft Research (MSR), Mar. 2003. URL: 5

[5]

Chai I.-X., Tong X., Chan S.-C, Shum H.-Y.: Plenoptic sampling. In SIGGRAPH 2000 (2000), pp. 307–318. URL: 2, 4, 8

Digital Library

[6]

Furukawa Y., Ponce J.: Patch-based multi-view stereo software, 3, 7

[7]

Furukawa Y., Ponce J.: Accurate, dense, and robust multi-view stereopsis. IEEE Trans, on Pattern Analysis and Machine Intelligence (2009). 3, 7

Digital Library

[8]

Gortler S. J., Grzeszczuk R., Szeliski R., Cohen M.F.: The lumigraph. In SIGGRAPH '96 (New York, NY, USA, 1996), ACM, pp. 43–54. 2

Digital Library

[9]

Georgiev T., Lumsdaine A.: Focused plenoptic camera and rendering. J. Electronic Imaging 19, 2 (2010), 021106. URL: 2

[10]

Georgiev T., Zheng K. C., Curless B., Salesin D., Nayar S., Intwala C.: Spatio-angular resolution tradeoffs in integral photography. In Proc. Eurographics Symposium on Rendering Techniques, 2006 (2006), Akenine-Möller T., Heidrich W., (Eds.), Eurographics Association, pp. 263–272. URL: 2

Digital Library

[11]

Heigl B., Koch R., Pollefeys M., Denzler L., Gool L.V.: Plenoptic modeling and rendering from image sequences taken by a hand-held camera. In DAGM (1999), pp. 94–101. 2

Digital Library

[12]

Isaksen A., McMillan L., Gortler S.J.: Dynamically reparameterized light fields. In SIGGRAPH 2000 (2000), Annual Conference Series, pp. 297–306. URL: 2, 4, 8

Digital Library

[13]

Kato Ft., Billinghurst M., B. Blanding, May R.: ARToolKit. Tech. rep., Hiroshima City University, December 1999. 4

[14]

Klein G., Murray D.: Parallel tracking and mapping for small AR workspaces. In Proc. (ISMAR'07) (Nara, Japan, November 2007). 2, 3

Digital Library

[15]

Koch R., Pollefeys M., Heigl B., Gool L. V., Niemann H.: Calibration of hand-held camera sequences for plenoptic modeling. In ICCV (1999), pp. 585–591. 2

[16]

Levoy M., Hanrahan P.: Light field rendering. In SIGGRAPH '96 (New York, NY, USA, 1996), ACM, pp. 31–42. 2, 3, 7, 8

Digital Library

[17]

Lipski C., Linz C., Berger K., Sellent A., Magnor M.: Virtual video camera: Image-based viewpoint navigation through space and time. Computer Graphics Forum 29, 8 (Dec. 2010), 2555–2568. 2

[18]

Loop C.: Smooth Subdivision Surfaces Based on Triangles. Master's thesis, University of Utah, Department of Mathematics, 1987. 6

[19]

Maver J., Bajcsy R.: Occlusions as a guide for planning the next view. IEEE Trans. Pattern Anal. Mach. Intell. 15 (May 1993), 417–433. URL:.

Digital Library

[20]

Ng R., Levoy M., Bredif M., Duval G., Horowitz M., Hanrahan P.: Light Field Photography with a Hand-Held Plenoptic Camera. Tech. Rep. CSTR 2005–02, Stanford University Computer Science, 2005. 2

[21]

Pito R.: A sensor-based solution to the next best view problem. In Proc. ICPR '96 (Washington, DC, USA, 1996), ICPR '96, IEEE Computer Society, pp. 941. URL: 2

Digital Library

[22]

Pollefeys M., Sinha S.N.: Iso-disparity surfaces for general stereo configurations. In ECCV (2004), pp. Vol III: 509–520. URL: 5

[23]

Rusinkiewicz S., Hall-Holt O., Levoy M.: Real-time 3D model acquisition. In SIGGRAPH 2002 (2002), Hughes J., (Ed.), ACM Press/ACM SIGGRAPH, pp. 438–446. URL: 2

Digital Library

[24]

Snavely N., Garg R., Seitz S. M., Szeliski R.: Finding paths through the world's photos. In SIGGRAPH '08 (New York, NY, USA, 2008), ACM, pp. 1–11. 3

Digital Library

[25]

Shewchuk J.R.: Delaunay Refinement Mesh Generation. PhD thesis, CMU, 1997. T. Report CMU-CS-97-137. 6

Digital Library

[26]

Stich T., Linz C., Albuquerque G., Magnor M.: View and time interpolation in image space. Computer Graphics Forum (Proc. of Pacific Graphics) 27, 7 (Feb. 2008), 1781–1787.

[27]

Stich T., Linz C., Wallraven C., Cunningham D., Magnor M.: Perception-motivated interpolation of image sequences. ACM Transactions on Applied Perception (TAP) 8, 2 (Jan. 2011), 1–25.

Digital Library

[28]

Snavely N., Seitz S. M., Szeliski R.: Photo tourism: exploring photo collections in 3d. In SIGGRAPH '06 (New York, NY, USA, 2006), ACM, pp. 835–846. 3

Digital Library

[29]

Stam J.: Exact evaluation of catmull-clark subdivision surfaces at arbitrary parameter values. In SIGGRAPH (1998), pp. 395–404. URL: 6

Digital Library

[30]

Taguchi Y., Agrawal A., Veeraraghavan A., Ramalingam S., Raskar R.: Axial-cones: modeling spherical catadioptric cameras for wide-angle light field rendering. ACM Trans. Graph 29, 6 (2010), 172. URL: 2

Digital Library

[31]

Veeraraghavan A., Raskar R., Agrawal A. K., Mohan A., Tumblin J.: Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing. ACM Trans. Graph 26, 3 (2007), 69. URL: 2

Digital Library

[32]

Volpel B., Theiner W.: Localization uncertainty in area-based stereo algorithms. Systems, Man and Cybernetics, IEEE Trans. 25, 12 (Dec. 1995), 1628–1634.

[33]

Warren J.: Subdivision methods for geometric design, Dec. 06 1995. URL: ; 6

Digital Library

[34]

Wilburn B., Joshi N., Vaish V., Talvala E.-V, Antunez E., Barth A., Adams A., Horowitz M., Levoy M.: High performance imaging using large camera arrays. ACM Trans. Graph. 24, 3 (2005), 765–776. 2, 8, 9

Digital Library

Cited By

Mahmoudpour SPagliari CSchelkens P(2024)Learning-based light field imaging: an overviewJournal on Image and Video Processing10.1186/s13640-024-00628-12024:1Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1186/s13640-024-00628-1
Wang GZhang JZhang KHuang RFang L(2024)GiganticNVS: Gigapixel Large-Scale Neural Rendering With Implicit Meta-Deformed ManifoldIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.332306946:1(338-353)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TPAMI.2023.3323069
Xu LAgrawal VLaney WGarcia TBansal AKim CRota Bulò SPorzi LKontschieder PBožič ALin DZollhöfer MRichardt C(2023)VR-NeRF: High-Fidelity Virtualized Walkable SpacesSIGGRAPH Asia 2023 Conference Papers10.1145/3610548.3618139(1-12)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.1145/3610548.3618139
Show More Cited By

Recommendations

Opacity light fields: interactive rendering of surface light fields with view-dependent opacity
I3D '03: Proceedings of the 2003 symposium on Interactive 3D graphics

We present new hardware-accelerated techniques for rendering surface light fields with opacity hulls that allow for interactive visualization of objects that have complex reflectance properties and elaborate geometrical details. The opacity hull is a ...
Read More
Visualization and Computer Graphics on Isotropically Emissive Volumetric Displays

The availability of commodity volumetric displays provides ordinary users with a new means of visualizing 3D data. Many of these displays are in the class of isotropically emissive light devices, which are designed to directly illumi-nate voxels in a 3D ...
Read More
Relighting with 4D incident light fields
SIGGRAPH '03: ACM SIGGRAPH 2003 Papers

We present an image-based technique to relight real objects illuminated by a 4D incident light field, representing the illumination of an environment. By exploiting the richness in angular and spatial variation of the light field, objects can be relit ...
Read More

Comments

Information & Contributors

Information

Published In

cover image Computer Graphics Forum

Computer Graphics Forum Volume 31, Issue 2pt1

May 2012

150 pages

ISSN:0167-7055

EISSN:1467-8659

Issue’s Table of Contents

Publisher

The Eurographs Association & John Wiley & Sons, Ltd.

Chichester, United Kingdom

Publication History

Published: 01 May 2012

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

55
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Other Metrics

View Author Metrics

Citations

Cited By

Mahmoudpour SPagliari CSchelkens P(2024)Learning-based light field imaging: an overviewJournal on Image and Video Processing10.1186/s13640-024-00628-12024:1Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1186/s13640-024-00628-1
Wang GZhang JZhang KHuang RFang L(2024)GiganticNVS: Gigapixel Large-Scale Neural Rendering With Implicit Meta-Deformed ManifoldIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.332306946:1(338-353)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TPAMI.2023.3323069
Xu LAgrawal VLaney WGarcia TBansal AKim CRota Bulò SPorzi LKontschieder PBožič ALin DZollhöfer MRichardt C(2023)VR-NeRF: High-Fidelity Virtualized Walkable SpacesSIGGRAPH Asia 2023 Conference Papers10.1145/3610548.3618139(1-12)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.1145/3610548.3618139
Lin KYang GXiao LLiu FRamamoorthi R(2023)View Synthesis of Dynamic Scenes Based on Deep 3D Mask VolumeIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.328933345:11(13250-13264)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1109/TPAMI.2023.3289333
Peng SGeng CZhang YXu YWang QShuai QZhou XBao H(2023)Implicit Neural Representations With Structured Latent Codes for Human Body ModelingIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.324581545:8(9895-9907)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1109/TPAMI.2023.3245815
Yan RSun JDeng LDavis A(2022)ReCapture: AR-Guided Time-lapse PhotographyProceedings of the 35th Annual ACM Symposium on User Interface Software and Technology10.1145/3526113.3545641(1-14)Online publication date: 29-Oct-2022
https://dl.acm.org/doi/10.1145/3526113.3545641
Zhang JLiu XYe XZhao FZhang YWu MZhang YXu LYu J(2021)Editable free-viewpoint video using a layered neural representationACM Transactions on Graphics10.1145/3450626.345975640:4(1-18)Online publication date: 19-Jul-2021
https://dl.acm.org/doi/10.1145/3450626.3459756
Tomoto YRao SBertel TChande KRichardt CHolzer SOrtiz-Cayon R(2020)Casual Real-World VR using Light FieldsSIGGRAPH Asia 2020 Posters10.1145/3415264.3425452(1-2)Online publication date: 4-Dec-2020
https://dl.acm.org/doi/10.1145/3415264.3425452
Broxton MFlynn JOverbeck RErickson DHedman PDuvall MDourgarian JBusch JWhalen MDebevec P(2020)Immersive light field video with a layered mesh representationACM Transactions on Graphics10.1145/3386569.339248539:4(86:1-86:15)Online publication date: 12-Aug-2020
https://dl.acm.org/doi/10.1145/3386569.3392485
Chen CSuma Rosenberg E(2020)Capture to Rendering Pipeline for Generating Dynamically Relightable Virtual Objects with Handheld RGB-D CamerasProceedings of the 26th ACM Symposium on Virtual Reality Software and Technology10.1145/3385956.3418952(1-11)Online publication date: 1-Nov-2020
https://dl.acm.org/doi/10.1145/3385956.3418952
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents

-