research-article

Supporting Responsive Cohabitation Between Virtual Interfaces and Physical Objects on Everyday Surfaces

Authors:

Scott Hudson, and

Chris HarrisonAuthors Info & Claims

Proceedings of the ACM on Human-Computer Interaction, Volume 1, Issue EICS

Article No.: 12, Pages 1 - 17

https://doi.org/10.1145/3095814

Published: 30 June 2017 Publication History

Abstract

Systems for providing mixed physical-virtual interaction on desktop surfaces have been proposed for decades, though no such systems have achieved widespread use. One major factor contributing to this lack of acceptance may be that these systems are not designed for the variety and complexity of actual work surfaces, which are often in flux and cluttered with physical objects. In this paper, we use an elicitation study and interviews to synthesize a list of ten interactive behaviors that desk-bound, digital interfaces should implement to support responsive cohabitation with physical objects. As a proof of concept, we implemented these interactive behaviors in a working augmented desk system, demonstrating their imminent feasibility.

References

[1]

Anand Agarawala and Ravin Balakrishnan. 2006. Keepin' it real: pushing the desktop metaphor with physics, piles and the pen. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '06). ACM, New York, NY, USA, 1283--1292.

Digital Library

[2]

Maneesh Agrawala and Chris Stolte. 2001. Rendering effective route maps: improving usability through generalization. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques (SIGGRAPH '01). ACM, New York, NY, USA, 241--249.

Digital Library

[3]

Toshifumi Arai, Kimiyoshi Machii, Soshiro Kuzunuki, and Hiroshi Shojima. 1995. InteractiveDESK: a computer-augmented desk which responds to operations on real objects. In Conference Companion on Human Factors in Computing Systems (CHI '95). ACM, New York, NY, USA, 141--142.

Digital Library

[4]

Hrvoje Benko, Ricardo Jota, and Andrew Wilson. 2012. MirageTable: freehand interaction on a projected augmented reality tabletop. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 199--208.

Digital Library

[5]

Xiaojun Bi, Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2011. Magic desk: bringing multi-touch surfaces into desktop work. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 2511--2520.

Digital Library

[6]

François Bodart, Anne-Marie Hennebert, Jean-Marie Leheureux, and Jean Vanderdonckt. 1994. Towards a dynamic strategy for computer-aided visual placement. In Proceedings of the workshop on Advanced visual interfaces (AVI '94). ACM, New York, NY, USA, 78--87.

Digital Library

[7]

Olha Bondarenko and Ruud Janssen. 2005. Documents at Hand: Learning from Paper to Improve Digital Technologies. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '05). ACM, New York, NY, USA, 121--130.

Digital Library

[8]

J Canny. 1986. A Computational Approach to Edge Detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 6 (June 1986), 679--698.

Digital Library

[9]

Xiang Cao, Andrew D.Wilson, Ravin Balakrishnan, Ken Hinckley, and Scott E. Hudson. ShapeTouch: Leveraging Contact Shape on Interactive Surfaces. In 3rd IEEE International Workshop on Horizontal Interactive Human Computer Systems, 2008 (TABLETOP '08), 129--136.

[10]

Jae Sik Chang, Eun Yi Kim, KeeChul Jung, and Hang Joon Kim. Real time hand tracking based on active contour model. In Proc. ICCSA '05, 999--1006.

Digital Library

[11]

Jason Cong, Lei He, Cheng-Kok Koh, and Patrick H. Madden. 1996. Performance optimization of VLSI interconnect layout. Integr. VLSI J. 21, 1--2 (November 1996), 1--94.

Digital Library

[12]

Franklin C. Crow. 1984. Summed-area tables for texture mapping. In Proceedings of the 11th annual conference on Computer graphics and interactive techniques (SIGGRAPH '84). ACM, New York, NY, USA, 207--212.

Digital Library

[13]

Giuseppe Di Battista, Peter Eades, Roberto Tamassia, and Ioannis G. Tollis. 1998. Graph Drawing: Algorithms for the Visualization of Graphs (1st ed.). Prentice Hall PTR, Upper Saddle River, NJ, USA.

Digital Library

[14]

Jerry Alan Fails and Dan Olsen Jr. 2002. Light widgets: interacting in every-day spaces. In Proceedings of the 7th international conference on Intelligent user interfaces (IUI '02). ACM, New York, NY, USA.

Digital Library

[15]

James Fogarty and Scott E. Hudson. 2003. GADGET: a toolkit for optimization-based approaches to interface and display generation. In Proceedings of the 16th annual ACM symposium on User interface software and technology (UIST '03). ACM, New York, NY, USA, 125--134.

Digital Library

[16]

Krzysztof Z. Gajos, Daniel S. Weld, and Jacob O. Wobbrock. 2008. Decision-theoretic user interface generation. In Proceedings of the 23rd national conference on Artificial intelligence - Volume 3 (AAAI'08), Anthony Cohn (Ed.), Vol. 3. AAAI Press 1532--1536.

Digital Library

[17]

Christoph Gebhardt, Roman Rädle, and Harald Reiterer. 2014. Integrative workplace: studying the effect of digital desks on users' working practices. In CHI '14 Extended Abstracts on Human Factors in Computing Systems (CHI EA '14). ACM, New York, NY, USA, 2155--2160.

Digital Library

[18]

John Hardy. 2012. Experiences: a year in the life of an interactive desk. In Proceedings of the Designing Interactive Systems Conference (DIS '12). ACM, New York, NY, USA, 679--688.

Digital Library

[19]

Chris Harrison, Hrvoje Benko, and Andrew D. Wilson. 2011. OmniTouch: wearable multitouch interaction everywhere. In Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST '11). ACM, New York, NY, USA, 441--450.

Digital Library

[20]

Hiroshi Ishii. 1990. TeamWorkStation: towards a seamless shared workspace. In Proceedings of the 1990 ACM conference on Computer-supported cooperative work (CSCW '90). ACM, New York, NY, USA, 13--26.

Digital Library

[21]

Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, and Andrew Fitzgibbon. 2011. KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST '11). ACM, New York, NY, USA, 559--568.

Digital Library

[22]

Sasa Junuzovic, Kori Inkpen, Tom Blank, and Anoop Gupta. 2012. IllumiShare: sharing any surface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 1919--1928.

Digital Library

[23]

Shaun K. Kane, Daniel Avrahami, Jacob O. Wobbrock, Beverly Harrison, Adam D. Rea, Matthai Philipose, and Anthony LaMarca. 2009. Bonfire: a nomadic system for hybrid laptop-tabletop interaction. In Proceedings of the 22nd annual ACM symposium on User interface software and technology (UIST '09). ACM, New York, NY, USA, 129--138.

Digital Library

[24]

Kourosh Khoshelham and Sander Oude Elberink. Accuracy and resolution of Kinect depth data for indoor mapping applications. Sensors, 12(2), 2012, 437--454.

[25]

Jeongyun Kim, Jonghoon Seo, and Tack-Don Han. 2014. AR Lamp: interactions on projection-based augmented reality for interactive learning. In Proceedings of the 19th international conference on Intelligent User Interfaces (IUI '14). ACM, New York, NY, USA, 353--358.

Digital Library

[26]

K. C. Knowlton. Computer Displays Optically Superimposed on Input Devices. Bell Systems Technical Journal, 53(3), 1977, 367--383.

[27]

Hideki Koike, Yoichi Sato, and Yoshinori Kobayashi. 2001. Integrating paper and digital information on EnhancedDesk: a method for realtime finger tracking on an augmented desk system. ACM Trans. Comput.-Hum. Interact. 8, 4 (December 2001), 307--322.

Digital Library

[28]

Eric Larson, Gabe Cohn, Sidhant Gupta, Xiaofeng Ren, Beverly Harrison, Dieter Fox, and Shwetak Patel. 2011. HeatWave: thermal imaging for surface user interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 2565--2574.

Digital Library

[29]

Joonhwan Lee, Jodi Forlizzi, and Scott E. Hudson. 2005. Studying the effectiveness of MOVE: a contextually optimized in-vehicle navigation system. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '05). ACM, New York, NY, USA, 571--580.

Digital Library

[30]

Natan Linder and Pattie Maes. 2010. LuminAR: portable robotic augmented reality interface design and prototype. In Adjunct proceedings of the 23nd annual ACM symposium on User interface software and technology (UIST '10). ACM, New York, NY, USA, 395--396.

Digital Library

[31]

Thomas W. Malone. 1983. How do people organize their desks?: Implications for the design of office information systems. ACM Trans. Inf. Syst. 1, 1 (January 1983), 99--112.

Digital Library

[32]

Daniel C. McFarlane and Steven M. Wilder. 2009. Interactive dirt: increasing mobile work performance with a wearable projector-camera system. In Proceedings of the 11th international conference on Ubiquitous computing (UbiComp '09). ACM, New York, NY, USA, 205--214.

Digital Library

[33]

Jurriaan D. Mulder, Jack Jansen, and Arjen van Rhijn. 2003. An affordable optical head tracking system for desktop VR/AR systems. In Proceedings of the workshop on Virtual environments 2003 (EGVE '03). ACM, New York, NY, USA, 215--223.

Digital Library

[34]

William Newman and Pierre Wellner. 1992. A desk supporting computer-based interaction with paper documents. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '92). ACM, New York, NY, USA, 587--592.

Digital Library

[35]

Michael Nielsen, Moritz Störring, Thomas B. Moeslund, and Erik Granum. 2004. A procedure for developing intuitive and ergonomic gesture interfaces for HCI. In Gesture Based Communication in Human-Computer Interaction 2915, 409--420. Springer Berlin Heidelberg.

[36]

Claudio S. Pinhanez. 2001. The Everywhere Displays Projector: A Device to Create Ubiquitous Graphical Interfaces. In Proceedings of the 3rd international conference on Ubiquitous Computing (UbiComp '01). Springer-Verlag, London, UK, UK, 315--331.

Digital Library

[37]

Ramesh Raskar, Jeroen van Baar, Paul Beardsley, Thomas Willwacher, Srinivas Rao, and Clifton Forlines. 2003. iLamps: geometrically aware and self-configuring projectors. In ACM SIGGRAPH 2003 Papers (SIGGRAPH '03). ACM, New York, NY, USA, 809--818.

Digital Library

[38]

Ramesh Raskar, Greg Welch, Matt Cutts, Adam Lake, Lev Stesin, and Henry Fuchs. 1998. The office of the future: a unified approach to image-based modeling and spatially immersive displays. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques (SIGGRAPH '98). ACM, New York, NY, USA, 179--188.

Digital Library

[39]

Jun Rekimoto. 2002. SmartSkin: an infrastructure for freehand manipulation on interactive surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '02). ACM, New York, NY, USA, 113--120.

Digital Library

[40]

Charles Robertson and John Robinson. 1999. Live paper: video augmentation to simulate interactive paper. In Proceedings of the seventh ACM international conference on Multimedia (Part 2) (MULTIMEDIA '99). ACM, New York, NY, USA, 167--170.

Digital Library

[41]

A. Sears. 1993. Layout Appropriateness: A Metric for Evaluating User Interface Widget Layout. IEEE Trans. Softw. Eng. 19, 7 (July 1993), 707--719.

Digital Library

[42]

Abigail J. Sellen and Richard H.R. Harper. 2003. The Myth of the Paperless Office. MIT Press, Cambridge, MA, USA.

Digital Library

[43]

Jürgen Steimle, Mohammadreza Khalilbeigi, Max Mühlhäuser, and James D. Hollan. 2010. Physical and digital media usage patterns on interactive tabletop surfaces. In ACM International Conference on Interactive Tabletops and Surfaces (ITS '10). ACM, New York, NY, USA, 167--176.

Digital Library

[44]

Yoshiki Takeoka, Takashi Miyaki, and Jun Rekimoto. 2010. Z-touch: an infrastructure for 3d gesture interaction in the proximity of tabletop surfaces. In ACM International Conference on Interactive Tabletops and Surfaces (ITS '10). ACM, New York, NY, USA, 91--94.

Digital Library

[45]

John Underkoffler and Hiroshi Ishii. 1998. Illuminating light: an optical design tool with a luminous-tangible interface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '98). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 542--549.

Digital Library

[46]

John Underkoffler, Brygg Ullmer, and Hiroshi Ishii. 1999. Emancipated pixels: real-world graphics in the luminous room. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques (SIGGRAPH '99). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 385--392.

Digital Library

[47]

Dhaval Vyas and Anton Nijholt. Artful surfaces: an ethnographic study exploring the use of space in design studios. Digital Creativity, 23(1), 2012, 1--20.

[48]

Pierre Wellner. 1993. Interacting with paper on the DigitalDesk. Commun. ACM 36, 7 (July 1993), 87--96.

Digital Library

[49]

Pierre Wellner. 1991. The DigitalDesk calculator: tangible manipulation on a desk top display. In Proceedings of the 4th annual ACM symposium on User interface software and technology (UIST '91). ACM, New York, NY, USA, 27--33.

Digital Library

[50]

Andrew D. Wilson and Hrvoje Benko. 2010. Combining multiple depth cameras and projectors for interactions on, above and between surfaces. In Proceedings of the 23nd annual ACM symposium on User interface software and technology (UIST '10). ACM, New York, NY, USA, 273--282.

Digital Library

[51]

Andrew D. Wilson. Depth sensing video cameras for 3D tangible tabletop interaction. In Proc. Tabletop '07, 201--204.

[52]

Raphael Wimmer, Fabian Hennecke, Florian Schulz, Sebastian Boring, Andreas Butz, and Heinrich Hußmann. 2010. Curve: revisiting the digital desk. In Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries (NordiCHI '10). ACM, New York, NY, USA, 561--570.

Digital Library

[53]

Robert Xiao, Chris Harrison, and Scott E. Hudson. 2013. WorldKit: rapid and easy creation of ad-hoc interactive applications on everyday surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 879--888.

Digital Library

[54]

Clemens Zeidler, Christof Lutteroth, Wolfgang Sturzlinger, and Gerald Weber. 2013. The auckland layout editor: an improved GUI layout specification process. In Proceedings of the 26th annual ACM symposium on User interface software and technology (UIST '13). ACM, New York, NY, USA, 343--352.

Digital Library

Cited By

Huang XXiao R(2024)SurfShareProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314187:4(1-24)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631418
Dupré CAppert CRey SSaidi HPietriga E(2024)TriPad: Touch Input in AR on Ordinary Surfaces with Hand Tracking OnlyProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642323(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642323
Johns CEvangelista Belo JFeit AKlokmose CPfeuffer K(2023)Towards Flexible and Robust User Interface Adaptations With Multiple ObjectivesProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606799(1-17)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606799
Show More Cited By

Index Terms

Supporting Responsive Cohabitation Between Virtual Interfaces and Physical Objects on Everyday Surfaces
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Graphics input devices
      2. Mixed / augmented reality
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Graphical user interfaces

Recommendations

Preliminary Investigation of Text Entry Method with Haptic Feedback from Real Object Surfaces Estimated Using Hand Tracking on HMD
Asian CHI '21: Proceedings of the Asian CHI Symposium 2021

In virtual reality systems, users enter text by selecting virtual keys with the fingers. A virtual keyboard is displayed in mid-air and thus does not provide haptic feedback. To address this problem, we present a text entry method that uses the surfaces ...
Read More
A study on touch & hover based interaction for zooming
CHI EA '12: CHI '12 Extended Abstracts on Human Factors in Computing Systems

Proximity is a useful medium for interaction with high interactive digital contents. It can be used in different contexts such as for navigation through depth in 3D space in zoomable interfaces. In this paper, we propose hover-based zoom interaction as ...
Read More
Bezel swipe: conflict-free scrolling and multiple selection on mobile touch screen devices
CHI '09: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Zooming user interfaces are increasingly popular on mobile devices with touch screens. Swiping and pinching finger gestures anywhere on the screen manipulate the displayed portion of a page, and taps open objects within the page. This makes navigation ...
Read More

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction

Proceedings of the ACM on Human-Computer Interaction Volume 1, Issue EICS

June 2017

249 pages

EISSN:2573-0142

DOI:10.1145/3120954

Editors:
Clifford Lampe
University of Michigan
,
Jeff Nichols
Google
,
Karrie Karahalios
University of Illinois Urbana-Champaign
,
Geraldine Fitzpatrick
Vienna University of Technology
,
Uichin Lee
KAIST
,
Andres Monroy-Hernandez
Microsoft Research
,
Wolfgang Stuerzlinger
Simon Fraser University

Issue’s Table of Contents

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 June 2017

Published in PACMHCI Volume 1, Issue EICS

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
661
Total Downloads

Downloads (Last 12 months)65
Downloads (Last 6 weeks)3

Other Metrics

View Author Metrics

Citations

Cited By

Huang XXiao R(2024)SurfShareProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314187:4(1-24)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631418
Dupré CAppert CRey SSaidi HPietriga E(2024)TriPad: Touch Input in AR on Ordinary Surfaces with Hand Tracking OnlyProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642323(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642323
Johns CEvangelista Belo JFeit AKlokmose CPfeuffer K(2023)Towards Flexible and Robust User Interface Adaptations With Multiple ObjectivesProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606799(1-17)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606799
Xu VNakagaki K(2023)Xs: Interactive Scissor Mechanisms as Portable and Customizable Shape-Changing InterfacesProceedings of the Seventeenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3569009.3572737(1-10)Online publication date: 26-Feb-2023
https://dl.acm.org/doi/10.1145/3569009.3572737
Johns CEvangelista Belo JKlokmose CPfeuffer K(2023)Pareto Optimal Layouts for Adaptive Mixed RealityExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585732(1-7)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3585732
Huang YLi CChen FZhang QWu K(2023)MM-Tap: Adaptive and Scalable Tap Localization on Ubiquitous Surfaces With mm-Level AccuracyIEEE Internet of Things Journal10.1109/JIOT.2023.328440410:23(20789-20802)Online publication date: 1-Dec-2023
https://doi.org/10.1109/JIOT.2023.3284404
Drey TJanek JLang JPuschmann DRietzler MRukzio E(2022)SpARklingPaperProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35503376:3(1-29)Online publication date: 7-Sep-2022
https://dl.acm.org/doi/10.1145/3550337
Evangelista Belo JLystbæk MFeit APfeuffer KKán POulasvirta AGrønbæk K(2022)AUIT – the Adaptive User Interfaces Toolkit for Designing XR ApplicationsProceedings of the 35th Annual ACM Symposium on User Interface Software and Technology10.1145/3526113.3545651(1-16)Online publication date: 29-Oct-2022
https://dl.acm.org/doi/10.1145/3526113.3545651
Pei SChari PWang XYang XKadambi AZhang Y(2022)ForceSight: Non-Contact Force Sensing with Laser Speckle ImagingProceedings of the 35th Annual ACM Symposium on User Interface Software and Technology10.1145/3526113.3545622(1-11)Online publication date: 29-Oct-2022
https://dl.acm.org/doi/10.1145/3526113.3545622
Shen VSpann JHarrison C(2021)FarOut Touch: Extending the Range of ad hoc Touch Sensing with Depth CamerasProceedings of the 2021 ACM Symposium on Spatial User Interaction10.1145/3485279.3485281(1-12)Online publication date: 9-Nov-2021
https://dl.acm.org/doi/10.1145/3485279.3485281
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents

-