skip to main content
10.1145/3485279.3485281acmconferencesArticle/Chapter ViewAbstractPublication PagessuiConference Proceedingsconference-collections
research-article
Open access

FarOut Touch: Extending the Range of ad hoc Touch Sensing with Depth Cameras

Published: 09 November 2021 Publication History
  • Get Citation Alerts
  • Abstract

    The ability to co-opt everyday surfaces for touch interactivity has been an area of HCI research for several decades. Ideally, a sensor operating in a device (such as a smart speaker) would be able to enable a whole room with touch sensing capabilities. Such a system could allow for software-defined light switches on walls, gestural input on countertops, and in general, more digitally flexible environments. While advances in depth sensors and computer vision have led to step-function improvements in the past, progress has slowed in recent years. We surveyed the literature and found that the very best ad hoc touch sensing systems are able to operate at ranges up to around 1.5 m. This limited range means that sensors must be carefully positioned in an environment to enable specific surfaces for interaction. In this research, we set ourselves the goal of doubling the sensing range of the current state of the art system. To achieve this goal, we leveraged an interesting finger ”denting” phenomena and adopted a marginal gains philosophy when developing our full stack. When put together, these many small improvements compound and yield a significant stride in performance. At 3 m range, our system offers a spatial accuracy of 0.98 cm with a touch segmentation accuracy of 96.1%, in line with prior systems operating at less than half the range. While more work remains to be done to achieve true room-scale ubiquity, we believe our system constitutes a useful advance over prior work.

    Supplementary Material

    MP4 File (3485279.3485281.mp4)
    Supplemental video

    References

    [1]
    Yomna Abdelrahman, Alireza Sahami Shirazi, Niels Henze, and Albrecht Schmidt. 2015. Investigation of material properties for thermal imaging-based interaction. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 15–18.
    [2]
    Cyrus S Bamji, Patrick O’Connor, Tamer Elkhatib, Swati Mehta, Barry Thompson, Lawrence A Prather, Dane Snow, Onur Can Akkaya, Andy Daniel, Andrew D Payne, 2014. A 0.13 μm CMOS system-on-chip for a 512 × 424 time-of-flight image sensor with multi-frequency photo-demodulation up to 130 MHz and 2 GS/s ADC. IEEE Journal of Solid-State Circuits 50, 1 (2014), 303–319.
    [3]
    Hrvoje Benko and Andrew D Wilson. 2009. DepthTouch: using depthsensing camera to enable freehand interactions on and above the interactive surface. In In Proceedings of the IEEE Workshop on Tabletops and Interactive Surfaces. Citeseer.
    [4]
    Timo Breuer, Christoph Bodensteiner, and Michael Arens. 2014. Low-cost commodity depth sensor comparison and accuracy analysis. In Electro-Optical Remote Sensing, Photonic Technologies, and Applications VIII; and Military Applications in Hyperspectral Imaging and High Spatial Resolution Sensing II, Vol. 9250. International Society for Optics and Photonics, 92500G.
    [5]
    Leah Buechley, David Mellis, Hannah Perner-Wilson, Emily Lovell, and Bonifaz Kaufmann. 2010. Living wall: programmable wallpaper for interactive spaces. In Proceedings of the 18th ACM international conference on Multimedia. 1401–1402.
    [6]
    Alex Butler, Shahram Izadi, and Steve Hodges. 2008. SideSight: multi-” touch” interaction around small devices. In Proceedings of the 21st annual ACM symposium on User interface software and technology. 201–204.
    [7]
    Arturo Cadena, Rubén Carvajal, Bruno Guamán, Roger Granda, Enrique Peláez, and Katherine Chiluiza. 2016. Fingertip detection approach on depth image sequences for interactive projection system. In 2016 IEEE Ecuador Technical Chapters Meeting (ETCM). 1–6. https://doi.org/10.1109/ETCM.2016.7750827
    [8]
    David Capel. 2004. Image mosaicing. In Image Mosaicing and super-resolution. Springer, 47–79.
    [9]
    Chongyu Chen, Jianfei Cai, Jianmin Zheng, Tat-Jen Cham, and Guangming Shi. 2013. A color-guided, region-adaptive and depth-selective unified framework for Kinect depth recovery. In 2013 IEEE 15th International Workshop on Multimedia Signal Processing (MMSP). IEEE, 007–012.
    [10]
    James Clear. 2018. Atomic habits: Tiny changes, remarkable results: An easy & proven way to build good habits & break bad ones. Avery.
    [11]
    Beginner’s Mind Collective and David Shaw. 2012. Makey Makey: improvising tangible and nature-based user interfaces. In Proceedings of the sixth international conference on tangible, embedded and embodied interaction. 367–370.
    [12]
    Andrea Corti, Silvio Giancola, Giacomo Mainetti, and Remo Sala. 2016. A metrological characterization of the Kinect V2 time-of-flight camera. Robotics and Autonomous Systems 75 (2016), 584–594.
    [13]
    Kyis Essmaeel, Luigi Gallo, Ernesto Damiani, Giuseppe De Pietro, and Albert Dipanda. 2012. Temporal Denoising of Kinect Depth Data. 8th International Conference on Signal Image Technology and Internet Based Systems, SITIS 2012r, 47–52. https://doi.org/10.1109/SITIS.2012.18
    [14]
    Georgios D Evangelidis and Emmanouil Z Psarakis. 2008. Parametric image alignment using enhanced correlation coefficient maximization. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 10(2008), 1858–1865.
    [15]
    Jerry Alan Fails and Dan Olsen Jr. 2002. Light widgets: interacting in every-day spaces. In Proceedings of the 7th international conference on Intelligent user interfaces. 63–69.
    [16]
    Martin Fischbach, Hendrik Striepe, Marc Erich Latoschik, and Birgit Lugrin. 2016. A low-cost, variable, interactive surface for mixed-reality tabletop games. In Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology. 297–298.
    [17]
    William T Freeman, Egon C Pasztor, and Owen T Carmichael. 2000. Learning low-level vision. International journal of computer vision 40, 1 (2000), 25–47.
    [18]
    Markus Funk, Stefan Schneegass, Michael Behringer, Niels Henze, and Albrecht Schmidt. 2015. An interactive curtain for media usage in the shower. In Proceedings of the 4th International Symposium on Pervasive Displays. 225–231.
    [19]
    Peter Fürsattel, Simon Placht, Michael Balda, Christian Schaller, Hannes Hofmann, Andreas Maier, and Christian Riess. 2015. A comparative error analysis of current time-of-flight sensors. IEEE Transactions on Computational Imaging 2, 1 (2015), 27–41.
    [20]
    Chunle Guo, Chongyi Li, Jichang Guo, Runmin Cong, Huazhu Fu, and Ping Han. 2018. Hierarchical features driven residual learning for depth map super-resolution. IEEE Transactions on Image Processing 28, 5 (2018), 2545–2557.
    [21]
    David Hall, Derek James, and Nick Marsden. 2012. Marginal gains: Olympic lessons in high performance for organisations. HR Bulletin: Research and Practice 7, 2 (2012), 9–13.
    [22]
    EJHBR Harrell. 2015. How 1% performance improvements led to Olympic gold. Harvard Business Review 30 (2015).
    [23]
    Chris Harrison, Hrvoje Benko, and Andrew D Wilson. 2011. OmniTouch: wearable multitouch interaction everywhere. In Proceedings of the 24th annual ACM symposium on User interface software and technology. 441–450.
    [24]
    Chris Harrison and Scott E Hudson. 2008. Scratch input: creating large, inexpensive, unpowered and mobile finger input surfaces. In Proceedings of the 21st annual ACM symposium on User interface software and technology. 205–208.
    [25]
    Nadia Haubner, Ulrich Schwanecke, Ralf Dörner, Simon Lehmann, and Johannes Luderschmidt. 2013. Detecting interaction above digital tabletops using a single depth camera. Machine vision and applications 24, 8 (2013), 1575–1587.
    [26]
    Christian Holz and Patrick Baudisch. 2010. The generalized perceived input point model and how to double touch accuracy by extracting fingerprints. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 581–590.
    [27]
    Michal Irani and Shmuel Peleg. 1991. Improving resolution by image registration. CVGIP: Graphical models and image processing 53, 3(1991), 231–239.
    [28]
    Hiroshi Ishii, Craig Wisneski, Ben Orbanes, Julian andChun, and Joe Paradiso. 1999. PingPongPlus: design of an athletic-tangible interface for computer-supported cooperative play. In Proceedings of the 1999 CHI Conference on Human Factors in Computing Systems. 394–401.
    [29]
    Daisuke Iwai and Kosuke Sato. 2005. Heat sensation in image creation with thermal vision. In Proceedings of the 2005 ACM SIGCHI International Conference on Advances in computer entertainment technology. 213–216.
    [30]
    Daisuke Iwai and Kosuke Sato. 2011. Document search support by making physical documents transparent in projection-based mixed reality. Virtual reality 15, 2-3 (2011), 147–160.
    [31]
    Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, 2011. KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In Proceedings of the 24th annual ACM symposium on User interface software and technology. 559–568.
    [32]
    Zhongyu Jiang, Huanjing Yue, Yu-Kun Lai, Jingyu Yang, Yonghong Hou, and Chunping Hou. 2021. Deep edge map guided depth super resolution. Signal Processing: Image Communication 90 (2021), 116040.
    [33]
    Shaun K Kane, Daniel Avrahami, Jacob O Wobbrock, Beverly Harrison, Adam D Rea, Matthai Philipose, and Anthony LaMarca. 2009. Bonfire: a nomadic system for hybrid laptop-tabletop interaction. In Proceedings of the 22nd annual ACM symposium on User interface software and technology. 129–138.
    [34]
    Kourosh Khoshelham and Sander Oude Elberink. 2012. Accuracy and resolution of kinect depth data for indoor mapping applications. Sensors 12, 2 (2012), 1437–1454.
    [35]
    Daniel Kurz. 2014. Thermal touch: Thermography-enabled everywhere touch interfaces for mobile augmented reality applications. In 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 9–16.
    [36]
    Gierad Laput and Chris Harrison. 2019. SurfaceSight: a new spin on touch, user, and object sensing for IoT experiences. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.
    [37]
    Eric Larson, Gabe Cohn, Sidhant Gupta, Xiaofeng Ren, Beverly Harrison, Dieter Fox, and Shwetak Patel. 2011. Heatwave: thermal imaging for surface user interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2565–2574.
    [38]
    Felix Järemo Lawin, Per-Erik Forssén, and Hannes Ovrén. 2016. Efficient Multi-Frequency Phase Unwrapping using Kernel Density Estimation. In European Conference on Computer Vision (ECCV). Springer International Publishing AG, Amsterdam. VR Projects: Learnable Camera Motion Models, 2014-5928, Energy Models for Computational Cameras, 2014-6227.
    [39]
    Joe Marshall, Tony Pridmore, Mike Pound, Steve Benford, and Boriana Koleva. 2008. Pressing the flesh: Sensing multiple touch and finger pressure on arbitrary surfaces. In International Conference on Pervasive Computing. Springer, 38–55.
    [40]
    Sundar Murugappan, Niklas Elmqvist, and Karthik Ramani. 2012. Extended multitouch: recovering touch posture and differentiating users using a depth camera. In Proceedings of the 25th annual ACM symposium on User interface software and technology. 487–496.
    [41]
    J. A. Paradiso, Che King Leo, N. Checka, and Kaijen Hsiao. 2002. Passive acoustic sensing for tracking knocks atop large interactive displays. In SENSORS, 2002 IEEE, Vol. 1. 521–527 vol.1. https://doi.org/10.1109/ICSENS.2002.1037150
    [42]
    Joseph A Paradiso and Che King Leo. 2005. Tracking and characterizing knocks atop large interactive displays. Sensor Review (2005).
    [43]
    DT Pham, M Al-Kutubi, Z Ji, M Yang, Z Wang, and S Catheline. 2005. Tangible acoustic interface approaches. In Proceedings of IPROMS 2005 Virtual Conference. Citeseer, 497–502.
    [44]
    DT Pham, Z Wang, Z Ji, M Yang, M Al-Kutubi, and Stefan Catheline. 2005. Acoustic pattern registration for a new type of human-computer interface. In Proc. Virtual Conf., IPROMs.
    [45]
    Duc Truong Pham, Ze Ji, Ming Yang, Zuobin Wang, and Mostafa Al-Kutubi. 2007. A novel human-computer interface based on passive acoustic localisation. In International Conference on Human-Computer Interaction. Springer, 901–909.
    [46]
    Fei Qi, Junyu Han, Pengjin Wang, Guangming Shi, and Fu Li. 2013. Structure guided fusion for depth map inpainting. Pattern Recognition Letters 34, 1 (2013), 70–76.
    [47]
    Hamed Sarbolandi, Damien Lefloch, and Andreas Kolb. 2015. Kinect range sensing: Structured-light versus Time-of-Flight Kinect. Computer vision and image understanding 139 (2015), 1–20.
    [48]
    Munehiko Sato, Ivan Poupyrev, and Chris Harrison. 2012. Touché: enhancing touch interaction on humans, screens, liquids, and everyday objects. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 483–492.
    [49]
    Katie Seggerman and Andy Perez. 2014. Table Top Computer Interface Using A Depth Sensor.
    [50]
    John Sell and Patrick O’Connor. 2014. The xbox one system on a chip and kinect sensor. IEEE Micro 34, 2 (2014), 44–53.
    [51]
    Adam Smith, Hari Balakrishnan, Michel Goraczko, and Nissanka Priyantha. 2004. Tracking moving devices with the cricket location system. In Proceedings of the 2nd international conference on Mobile systems, applications, and services. 190–202.
    [52]
    Wanbin Song, Anh Vu Le, Seokmin Yun, Seung-Won Jung, and Chee Sun Won. 2017. Depth completion for kinect v2 sensor. Multimedia Tools and Applications 76, 3 (2017), 4357–4380.
    [53]
    Jean-Luc Starck, Jalal Fadili, and Fionn Murtagh. 2007. The undecimated wavelet decomposition and its reconstruction. IEEE transactions on image processing 16, 2 (2007), 297–309.
    [54]
    Joshua Strickon and Joseph Paradiso. 1998. Tracking hands above large interactive surfaces with a low-cost scanning laser rangefinder. In CHI 98 Conference Summary on Human Factors in Computing Systems. 231–232.
    [55]
    Naoki Sugita, Daisuke Iwai, and Kosuke Sato. 2008. Touch sensing by image analysis of fingernail. In 2008 SICE Annual Conference. IEEE, 1520–1525.
    [56]
    Yoshiki Takeoka, Takashi Miyaki, and Jun Rekimoto. 2010. Z-touch: an infrastructure for 3d gesture interaction in the proximity of tabletop surfaces. In ACM International Conference on Interactive Tabletops and Surfaces. 91–94.
    [57]
    Carlo Tomasi, Abbas Rafii, and Ilhami Torunoglu. 2003. Full-size projection keyboard for handheld devices. Commun. ACM 46, 7 (2003), 70–75.
    [58]
    Feng Wang and Xiangshi Ren. 2009. Empirical evaluation for finger input properties in multi-touch interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1063–1072.
    [59]
    Oliver Wasenmüller and Didier Stricker. 2016. Comparison of kinect v1 and v2 depth images in terms of accuracy and precision. In Asian Conference on Computer Vision. Springer, 34–45.
    [60]
    Pierre Wellner. 1993. Interacting with paper on the DigitalDesk. Commun. ACM 36, 7 (1993), 87–96.
    [61]
    Andrew D Wilson. 2004. TouchLight: an imaging touch screen and display for gesture-based interaction. In Proceedings of the 6th international conference on Multimodal interfaces. 69–76.
    [62]
    Andrew D Wilson. 2005. PlayAnywhere: a compact interactive tabletop projection-vision system. In Proceedings of the 18th annual ACM symposium on User interface software and technology. 83–92.
    [63]
    Andrew D Wilson. 2007. Depth-sensing video cameras for 3d tangible tabletop interaction. In Second Annual IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP’07). IEEE, 201–204.
    [64]
    Andrew D Wilson. 2010. Using a depth camera as a touch sensor. In ACM international conference on interactive tabletops and surfaces. 69–72.
    [65]
    Andrew D Wilson and Hrvoje Benko. 2010. Combining multiple depth cameras and projectors for interactions on, above and between surfaces. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. 273–282.
    [66]
    Bartlomiej Wronski, Ignacio Garcia-Dorado, Manfred Ernst, Damien Kelly, Michael Krainin, Chia-Kai Liang, Marc Levoy, and Peyman Milanfar. 2019. Handheld multi-frame super-resolution. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1–18.
    [67]
    Robert Xiao, Chris Harrison, and Scott E Hudson. 2013. WorldKit: rapid and easy creation of ad-hoc interactive applications on everyday surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 879–888.
    [68]
    Robert Xiao, Scott Hudson, and Chris Harrison. 2016. Direct: Making touch tracking on ordinary surfaces practical with hybrid depth-infrared sensing. In Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces. 85–94.
    [69]
    Robert Xiao, Scott Hudson, and Chris Harrison. 2017. Supporting responsive cohabitation between virtual interfaces and physical objects on everyday surfaces. Proceedings of the ACM on Human-Computer Interaction 1, EICS(2017), 1–17.
    [70]
    Robert Xiao, Greg Lew, James Marsanico, Divya Hariharan, Scott Hudson, and Chris Harrison. 2014. Toffee: enabling ad hoc, around-device interaction with acoustic time-of-arrival correlation. In Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services. 67–76.
    [71]
    Robert Xiao, Julia Schwarz, Nick Throm, Andrew D Wilson, and Hrvoje Benko. 2018. MRTouch: Adding touch input to head-mounted mixed reality. IEEE transactions on visualization and computer graphics 24, 4(2018), 1653–1660.
    [72]
    Jingyu Yang, Xinchen Ye, Kun Li, Chunping Hou, and Yao Wang. 2014. Color-guided depth recovery from RGB-D data using an adaptive autoregressive model. IEEE transactions on image processing 23, 8 (2014), 3443–3458.
    [73]
    Ying Yin. 2012. A hierarchical approach to continuous gesture analysis for natural multi-modal interaction. In Poster in Proceedings of the 14th ACM international conference on Multimodal interaction. 357–360. https://groups.csail.mit.edu/mug/projects/gesture_kinect/docs/fingertip-tracking.pdf
    [74]
    Guoshen Yu and Guillermo Sapiro. 2011. DCT image denoising: a simple and effective image denoising algorithm. Image Processing On Line 1 (2011), 292–296.
    [75]
    Cheng Zhang, Qiuyue Xue, Anandghan Waghmare, Ruichen Meng, Sumeet Jain, Yizeng Han, Xinyu Li, Kenneth Cunefare, Thomas Ploetz, Thad Starner, 2018. Fingerping: Recognizing fine-grained hand poses using active acoustic on-body sensing. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–10.
    [76]
    Yang Zhang, Gierad Laput, and Chris Harrison. 2017. Electrick: Low-cost touch sensing using electric field tomography. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 1–14.
    [77]
    Yang Zhang, Chouchang Yang, Scott E Hudson, Chris Harrison, and Alanson Sample. 2018. Wall++ room-scale interactive and context-aware sensing. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–15.

    Cited By

    View all
    • (2023)Structured Light Speckle: Joint Ego-Centric Depth Estimation and Low-Latency Contact Detection via Remote VibrometryProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606749(1-12)Online publication date: 29-Oct-2023
    • (2023)Xs: Interactive Scissor Mechanisms as Portable and Customizable Shape-Changing InterfacesProceedings of the Seventeenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3569009.3572737(1-10)Online publication date: 26-Feb-2023
    • (2022)Reducing the Latency of Touch Tracking on Ad-hoc SurfacesProceedings of the ACM on Human-Computer Interaction10.1145/35677306:ISS(489-499)Online publication date: 14-Nov-2022

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SUI '21: Proceedings of the 2021 ACM Symposium on Spatial User Interaction
    November 2021
    206 pages
    ISBN:9781450390910
    DOI:10.1145/3485279
    This work is licensed under a Creative Commons Attribution International 4.0 License.

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 09 November 2021

    Check for updates

    Badges

    • Best Paper

    Author Tags

    1. Depth sensing
    2. finger tracking.
    3. on-world computing
    4. projection
    5. ubiquitous computing

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    SUI '21
    SUI '21: Symposium on Spatial User Interaction
    November 9 - 10, 2021
    Virtual Event, USA

    Acceptance Rates

    Overall Acceptance Rate 86 of 279 submissions, 31%

    Upcoming Conference

    SUI '24
    ACM Symposium on Spatial User Interaction
    October 7 - 8, 2024
    Trier , Germany

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)219
    • Downloads (Last 6 weeks)23

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)Structured Light Speckle: Joint Ego-Centric Depth Estimation and Low-Latency Contact Detection via Remote VibrometryProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606749(1-12)Online publication date: 29-Oct-2023
    • (2023)Xs: Interactive Scissor Mechanisms as Portable and Customizable Shape-Changing InterfacesProceedings of the Seventeenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3569009.3572737(1-10)Online publication date: 26-Feb-2023
    • (2022)Reducing the Latency of Touch Tracking on Ad-hoc SurfacesProceedings of the ACM on Human-Computer Interaction10.1145/35677306:ISS(489-499)Online publication date: 14-Nov-2022

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Get Access

    Login options

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media

    -