research-article

Parallax360: Stereoscopic 360° Scene Representation for Head-Motion Parallax

Authors:

Christian Richardt, and

Jun-Hai YongAuthors Info & Claims

IEEE Transactions on Visualization and Computer Graphics, Volume 24, Issue 4

Pages 1545 - 1553

https://doi.org/10.1109/TVCG.2018.2794071

Published: 01 April 2018 Publication History

Abstract

We propose a novel 360° scene representation for converting real scenes into stereoscopic 3D virtual reality content with head-motion parallax. Our image-based scene representation enables efficient synthesis of novel views with six degrees-of-freedom (6-DoF) by fusing motion fields at two scales: (1) disparity motion fields carry implicit depth information and are robustly estimated from multiple laterally displaced auxiliary viewpoints, and (2) pairwise motion fields enable real-time flow-based blending, which improves the visual fidelity of results by minimizing ghosting and view transition artifacts. Based on our scene representation, we present an end-to-end system that captures real scenes with a robotic camera arm, processes the recorded data, and finally renders the scene in a head-mounted display in real time (more than 40 Hz). Our approach is the first to support head-motion parallax when viewing real 360° scenes. We demonstrate compelling results that illustrate the enhanced visual experience — and hence sense of immersion-achieved with our approach compared to widely-used stereoscopic panoramas.

References

[1]

R. Anderson, D. Gallup, J.T. Barron, J. Kontkanen, N. Snavely, C. Hernandez, S. Agarwal, and S.M. Seitz. Jump: virtual reality video. ACM Transactions on Graphics, Volume 35 Issue 6: p. pp.198, 2016.

Digital Library

[2]

M. Arikan, R. Preiner, and M. Wimmer. Multi-depth-map raytracing for efficient large-scene reconstruction. IEEE Transactions on Visualization and Computer Graphics, Volume 22 Issue 2: pp. 1127–1137, 2016.

Digital Library

[3]

C. Birklbauer and O. Bimber. Panorama light-field imaging. Computer Graphics Forum, Volume 33 Issue 2: pp. 43–52, 2014.

Digital Library

[4]

M. Brown and D.G. Lowe. Automatic panoramic image stitching using invariant features. International Journal of Computer Vision, Volume 74 Issue 1: pp. 59–73, 2007.

Digital Library

[5]

C. Buehler, M. Bosse, L. McMillan, S. Gortler, and M. Cohen. Unstructured lumigraph rendering. In SIGGRAPH, pp. 425–432, 2001.

Digital Library

[6]

G. Chaurasia, S. Duchene, O. Sorkine-Hornung, and G. Drettakis. Depth synthesis and local warps for plausible image-based navigation. ACM Transactions on Graphics, Volume 32 Issue 3: p. pp.30, 2013.

Digital Library

[7]

S.E. Chen and L. Williams. View interpolation for image synthesis. In SIGGRAPH, pp. 279–288, 1993.

Digital Library

[8]

A. Collet, M. Chuang, P. Sweeney, D. Gillett, D. Evseev, D. Calabrese, H. Hoppe, A. Kirk, and S. Sullivan. High-quality streamable free-viewpoint video. ACM Transactions on Graphics, Volume 34 Issue 4: p. pp.69, 2015.

Digital Library

[9]

M. Dou, S. Khamis, Y. Degtyarev, P. Davidson, S.R. Fanello, A. Kowdle, S.O. Escolano, C. Rhemann, D. Kim, J. Taylor, P. Kohli, V. Tankovich, and S. Izadi. Fusion4D: real-time performance capture of challenging scenes. ACM Transactions on Graphics, Volume 35 Issue 4: p. pp.114, 2016.

Digital Library

[10]

M. Eisemann, B. De Decker, M. Magnor, P. Bekaert, E. De Aguiar, N. Ahmed, C. Theobalt, and A. Sellent. Floating textures. Computer Graphics Forum, Volume 27 Issue 2: pp. 409–418, 2008.

[11]

Y. Furukawa and J. Ponce. Accurate, dense, and robust multiview stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 32 Issue 8: pp. 1362–1376, 2010.

Digital Library

[12]

M. Goesele, J. Ackermann, S. Fuhrmann, C. Haubold, R. Klowsky, D. Steedly, and R. Szeliski. Ambient point clouds for view interpolation. ACM Transactions on Graphics, Volume 29 Issue 4: p. pp.95, 2010.

Digital Library

[13]

S.J. Gortler, R. Grzeszczuk, R. Szeliski, and M.F. Cohen. The lumigraph. In SIGGRAPH, pp. 43–54, 1996.

Digital Library

[14]

P. Hedman, S. Alsisan, R. Szeliski, and J. Kopf. Casual 3D photography. ACM Transactions on Graphics, Volume 36 Issue 6: pp. 234:1–15, 2017.

Digital Library

[15]

P. Hedman, T. Ritschel, G. Drettakis, and G. Brostow. Scalable inside-out image-based rendering. ACM Transactions on Graphics, Volume 35 Issue 6: pp. 231:1–11, 2016.

Digital Library

[16]

J. Huang, Z. Chen, D. Ceylan, and H. Jin. 6-DOF VR videos with a single 360-camera. In Proceedings of IEEE Virtual Reality (VR), pp. 37–44, 2017.

[17]

E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. FlowNet 2.0: Evolution of optical flow estimation with deep networks. In CVPR, 2017.

[18]

M. Innmann, M. Zollhöfer, M. Nießner, C. Theobalt, and M. Stamminger. VolumeDeform: Real-time volumetric non-rigid reconstruction. In ECCV, 2016.

[19]

S.B. Kang, R. Szeliski, and M. Uyttendaele. Seamless stitching using multi-perspective plane sweep . Technical Report, <institution>Microsoft Research</institution>, 2004.

[20]

J. Kopf, D. Lischinski, O. Deussen, D. Cohen-Or, and M. Cohen. Locally adapted projections to reduce panorama distortions. Computer Graphics Forum, Volume 28 Issue 4: pp. 1083–1089, 2009.

Digital Library

[21]

B. Krolla, M. Diebold, B. Goldlücke, and D. Stricker. Spherical light fields. In BMVC, 2014.

[22]

J. Lee, B. Kim, K. Kim, Y. Kim, and J. Noh. Rich360: optimized spherical representation from structured panoramic camera arrays. ACM Transactions on Graphics, Volume 35 Issue 4: p. pp.63, 2016.

Digital Library

[23]

A. Levin and F. Durand. Linear view synthesis using a dimensionality gap light field prior. In CVPR, 2010.

[24]

M. Levoy and P. Hanrahan. Light field rendering. In SIGGRAPH, pp. 31–42, 1996.

Digital Library

[25]

K. Marwah, G. Wetzstein, Y. Bando, and R. Raskar. Compressive light field photography using overcomplete dictionaries and optimized projections. ACM Transactions on Graphics, Volume 32 Issue 4: p. pp.46, 2013.

Digital Library

[26]

K. Matzen, M.F. Cohen, B. Evans, J. Kopf, and R. Szeliski. Low-cost 360 stereo photography and video capture. ACM Transactions on Graphics, Volume 36 Issue 4: p. pp.148, 2017.

Digital Library

[27]

K. Mitra and A. Veeraraghavan. Light field denoising, light field super-resolution and stereo camera based refocussing using a GMM light field patch prior. In CVPR Workshops, 2012.

[28]

R.A. Newcombe, D. Fox, and S.M. Seitz. DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time. In CVPR, 2015.

[29]

R.A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A.J. Davison, P. Kohi, J. Shotton, S. Hodges, and A. Fitzgibbon. KinectFusion: Real-time dense surface mapping and tracking. In ISMAR, 2011.

Digital Library

[30]

M. Nießner, M. Zollhöfer, S. Izadi, and M. Stamminger. Real-time 3D reconstruction at scale using voxel hashing. ACM Transactions on Graphics, Volume 32 Issue 6: pp. 169:1–11, 2013.

Digital Library

[31]

S. Peleg, M. Ben-Ezra, and Y. Pritch. Omnistereo: Panoramic stereo imaging. IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 23 Issue 3: pp. 279–290, 2001.

Digital Library

[32]

F. Perazzi, A. Sorkine-Hornung, H. Zimmer, P. Kaufmann, O. Wang, S. Watson, and M. Gross. Panoramic video from unstructured camera arrays. Computer Graphics Forum, Volume 34 Issue 2: pp. 57–68, 2015.

Digital Library

[33]

S. Philip, B. Summa, J. Tierny, P.-T. Bremer, and V. Pascucci. Distributed seams for gigapixel panoramas. IEEE Transactions on Visualization and Computer Graphics, Volume 21 Issue 3: pp. 350–362, 2015.

[34]

L. Piegl and W. Tiller. The NURBS book . Springer, 2012.

[35]

C. Richardt, Y. Pritch, H. Zimmer, and A. Sorkine-Hornung. Megastereo: Constructing high-resolution stereo panoramas. In CVPR, 2013.

Digital Library

[36]

L. Shi, H. Hassanieh, A. Davis, D. Katabi, and F. Durand. Light field reconstruction using sparsity in the continuous Fourier domain. ACM Transactions on Graphics, Volume 34 Issue 1: p. pp.12, 2014.

Digital Library

[37]

H.-Y. Shum, S.-C. Chan, and S.B. Kang. Image-Based Rendering . Springer, 2007.

Digital Library

[38]

H.-Y. Shum and L.-W. He. Rendering with concentric mosaics. In SIGGRAPH, pp. 299–306, 1999.

Digital Library

[39]

H.-Y. Shum and R. Szeliski. Stereo reconstruction from multiperspective panoramas. In ICCV, 1999.

[40]

H.-Y. Shum and R. Szeliski. Systems and experiment paper: Construction of panoramic image mosaics with global and local alignment. International Journal of Computer Vision, Volume 36 Issue 2: pp. 101–130, 2000.

Digital Library

[41]

R. Szeliski. Image alignment and stitching: A tutorial. Foundations and Trends® in Computer Graphics and Vision, Volume 2 Issue 1: pp. 1–104, 2006.

Digital Library

[42]

S. Wanner and B. Goldluecke. Variational light field analysis for disparity estimation and super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 36 Issue 3: pp. 606–619, 2014.

Digital Library

[43]

K. Xu, H. Huang, Y. Shi, H. Li, P. Long, J. Caichen, W. Sun, and B. Chen. Autoscanning for coupled scene reconstruction and proactive object analysis. ACM Transactions on Graphics, Volume 34 Issue 6: p. pp.177, 2015.

Digital Library

[44]

L. Xu, J. Jia, and Y. Matsushita. Motion detail preserving optical flow estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 34 Issue 9: pp. 1744–1757, 2012.

Digital Library

[45]

L. Zelnik-Manor, G. Peters, and P. Perona. Squaring the circle in panoramas. In ICCV, 2005.

Digital Library

[46]

Y. Zhang, W. Xu, Y. Tong, and K. Zhou. Online structure analysis for real-time indoor scene reconstruction. ACM Transactions on Graphics, Volume 34 Issue 5: p. pp.159, 2015.

Digital Library

[47]

Z. Zhang, Y. Liu, and Q. Dai. Light field from micro-baseline image pair. In CVPR, 2015.

Cited By

Pintore GJaspe-Villanueva AHadwiger MGobbetti ESchneider JAgus M(2023)PanoVerse: automatic generation of stereoscopic environments from single indoor panoramic images for Metaverse applicationsProceedings of the 28th International ACM Conference on 3D Web Technology10.1145/3611314.3615914(1-10)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3611314.3615914
da Silveira TPinto PMurrugarra-Llerena JJung C(2022)3D Scene Geometry Estimation from 360° Imagery: A SurveyACM Computing Surveys10.1145/351902155:4(1-39)Online publication date: 21-Nov-2022
https://dl.acm.org/doi/10.1145/3519021
Li JHe YJiao JHu YHan YWen JShen HZhuang YSmith JYang YCesar PMetze FPrabhakaran B(2021)Extending 6-DoF VR Experience Via Multi-Sphere Images InterpolationProceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3475339(4632-4640)Online publication date: 17-Oct-2021
https://dl.acm.org/doi/10.1145/3474085.3475339
Show More Cited By

Index Terms

Parallax360: Stereoscopic 360° Scene Representation for Head-Motion Parallax
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Image and video acquisition
        3D imaging

Index terms have been assigned to the content through auto-classification.

Recommendations

Catadioptric Stereo Using Planar Mirrors

By using mirror reflections of a scene, stereo images can be captured with a single camera (catadioptric stereo). In addition to simplifying data acquisition single camera stereo provides both geometric and radiometric advantages over traditional two ...
Read More
Omnistereo: Panoramic Stereo Imaging

An Omnistereo panorama consists of a pair of panoramic images, where one panorama is for the left eye and another panorama is for the right eye. The panoramic stereo pair provides a stereo sensation up to a full 360 degrees. Omnistereo panoramas cannot ...
Read More
Self-calibration of hybrid central catadioptric and perspective cameras

Hybrid central catadioptric and perspective cameras are desired in practice, because the hybrid camera system can capture large field of view as well as high-resolution images. However, the calibration of the system is challenging due to heavy ...
Read More

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Visualization and Computer Graphics

IEEE Transactions on Visualization and Computer Graphics Volume 24, Issue 4

April 2018

283 pages

ISSN:1077-2626

Issue’s Table of Contents

Copyright © 2018.

Publisher

IEEE Educational Activities Department

United States

Publication History

Published: 01 April 2018

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Other Metrics

View Author Metrics

Citations

Cited By

Pintore GJaspe-Villanueva AHadwiger MGobbetti ESchneider JAgus M(2023)PanoVerse: automatic generation of stereoscopic environments from single indoor panoramic images for Metaverse applicationsProceedings of the 28th International ACM Conference on 3D Web Technology10.1145/3611314.3615914(1-10)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3611314.3615914
da Silveira TPinto PMurrugarra-Llerena JJung C(2022)3D Scene Geometry Estimation from 360° Imagery: A SurveyACM Computing Surveys10.1145/351902155:4(1-39)Online publication date: 21-Nov-2022
https://dl.acm.org/doi/10.1145/3519021
Li JHe YJiao JHu YHan YWen JShen HZhuang YSmith JYang YCesar PMetze FPrabhakaran B(2021)Extending 6-DoF VR Experience Via Multi-Sphere Images InterpolationProceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3475339(4632-4640)Online publication date: 17-Oct-2021
https://dl.acm.org/doi/10.1145/3474085.3475339
Bertel TYuan MLindroos RRichardt C(2020)OmniPhotos: Casual 360° VR Photography with Motion ParallaxSIGGRAPH Asia 2020 Emerging Technologies10.1145/3415255.3422884(1-2)Online publication date: 4-Dec-2020
https://dl.acm.org/doi/10.1145/3415255.3422884
Bertel TYuan MLindroos RRichardt C(2020)OmniPhotosACM Transactions on Graphics10.1145/3414685.341777039:6(1-12)Online publication date: 27-Nov-2020
https://dl.acm.org/doi/10.1145/3414685.3417770
Mühlhausen MKappel MKassubeck MBittner PCastillo SMagnor M(2020)Temporal Consistent Motion Parallax for Omnidirectional Stereo Panorama VideoProceedings of the 26th ACM Symposium on Virtual Reality Software and Technology10.1145/3385956.3418965(1-9)Online publication date: 1-Nov-2020
https://dl.acm.org/doi/10.1145/3385956.3418965
Zhao SZhang HBhuyan SMishra CYing ZKandemir MSivasubramaniam ADas CMartínez JDuato JEeckhout L(2020)Déjà viewProceedings of the ACM/IEEE 47th Annual International Symposium on Computer Architecture10.1109/ISCA45697.2020.00030(241-253)Online publication date: 30-May-2020
https://dl.acm.org/doi/10.1109/ISCA45697.2020.00030
Youssef Sel Shehaby MFayed S(2020)A Smart multi-view panoramic imaging integrating stitching with geometric matrix relations among surveillance cameras (SMPI)Multimedia Tools and Applications10.1007/s11042-020-09432-179:41-42(30917-30981)Online publication date: 1-Nov-2020
https://dl.acm.org/doi/10.1007/s11042-020-09432-1
Lin KXu ZMildenhall BSrinivasan PHold-Geoffroy YDiVerdi SSun QSunkavalli KRamamoorthi R(2020)Deep Multi Depth Panoramas for View SynthesisComputer Vision – ECCV 202010.1007/978-3-030-58601-0_20(328-344)Online publication date: 23-Aug-2020
https://dl.acm.org/doi/10.1007/978-3-030-58601-0_20
Attal BLing SGokaslan ARichardt CTompkin J(2020)MatryODShka: Real-time 6DoF Video View Synthesis Using Multi-sphere ImagesComputer Vision – ECCV 202010.1007/978-3-030-58452-8_26(441-459)Online publication date: 23-Aug-2020
https://dl.acm.org/doi/10.1007/978-3-030-58452-8_26
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents

-