research-article

PrivacEye: privacy-preserving head-mounted eye tracking using egocentric scene image and eye movement features

Authors:

Andreas BullingAuthors Info & Claims

ETRA '19: Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications

Article No.: 26, Pages 1 - 10

https://doi.org/10.1145/3314111.3319913

Published: 25 June 2019 Publication History

Abstract

Eyewear devices, such as augmented reality displays, increasingly integrate eye tracking, but the first-person camera required to map a user's gaze to the visual scene can pose a significant threat to user and bystander privacy. We present PrivacEye, a method to detect privacy-sensitive everyday situations and automatically enable and disable the eye tracker's first-person camera using a mechanical shutter. To close the shutter in privacy-sensitive situations, the method uses a deep representation of the first-person video combined with rich features that encode users' eye movements. To open the shutter without visual input, PrivacEye detects changes in users' eye movements alone to gauge changes in the "privacy level" of the current situation. We evaluate our method on a first-person video dataset recorded in daily life situations of 17 participants, annotated by themselves for privacy sensitivity, and show that our method is effective in preserving privacy in this challenging setting.

Supplementary Material

ZIP File (a26-steil.zip)

Supplemental files.

Download
5.88 MB

References

[1]

Paarijaat Aditya, Rijurekha Sen, Peter Druschel, Seong Joon Oh, Rodrigo Benenson, Mario Fritz, Bernt Schiele, Bobby Bhattacharjee, and Tong Tong Wu. 2016. I-pic: A platform for Privacy-compliant Image Capture. In Annual International Conference on Mobile Systems, Applications, and Services (MobiSys). ACM, 235--248.

Digital Library

[2]

Olivier Aubert, Yannick Prié, and Daniel Schmitt. 2012. Advene As a Tailorable Hypervideo Authoring Tool: A Case Study. In Proceedings of the 2012 ACM Symposium on Document Engineering (DocEng '12). ACM, New York, NY, USA, 79--82.

Digital Library

[3]

Lemi Baruh and Zeynep Cemalcilar. 2014. It is more than personal: Development and validation of a multidimensional privacy orientation scale. Personality and Individual Differences 70 (2014), 165--170. https://doi.org/

[4]

Jürgen Bohn, Vlad Coroamă, Marc Langheinrich, Friedemann Mattern, and Michael Rohs. 2005. Social, economic, and ethical implications of ambient intelligence and ubiquitous computing. In Ambient Intelligence. Springer, 5--29.

[5]

Andreas Bulling and Kai Kunze. 2016. Eyewear Computers for Human-Computer Interaction. ACM Interactions 23, 3 (2016), 70--73.

Digital Library

[6]

Andreas Bulling, Jamie A. Ward, and Hans Gellersen. 2012. Multimodal Recognition of Reading Activity in Transit Using Body-Worn Sensors. ACM Transactions on Applied Perception 9, 1 (2012), 2:1--2:21.

Digital Library

[7]

Andreas Bulling, Jamie A. Ward, Hans Gellersen, and Gerhard Tröster. 2011. Eye Movement Analysis for Activity Recognition Using Electrooculography. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 4 (April 2011), 741--753.

Digital Library

[8]

Andreas Bulling, Christian Weichel, and Hans Gellersen. 2013. EyeContext: Recognition of High-level Contextual Cues from Human Visual Behaviour. In Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI). 305--308.

Digital Library

[9]

Kelly Caine. 2009. Exploring everyday privacy behaviors and misclosures. Georgia Institute of Technology.

[10]

Liwei Chan and Kouta Minamizawa. 2017. FrontFace: Facilitating Communication Between HMD Users and Outsiders Using Front-facing-screen HMDs. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '17). ACM, New York, NY, USA, Article 22, 5 pages.

Digital Library

[11]

Soumyadeb Chowdhury, Md Sadek Ferdous, and Joemon M Jose. 2016. Exploring Lifelog Sharing and Privacy. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct (UbiComp '16). ACM, New York, NY, USA, 553--558.

Digital Library

[12]

Tamara Denning, Zakariya Dehlawi, and Tadayoshi Kohno. 2014. In situ with Bystanders of Augmented Reality Glasses: Perspectives on Recording and Privacy-mediating Technologies. In Proceedings of the Conference on Human Factors in Computing Systems (CHI). ACM, 2377--2386.

Digital Library

[13]

Barrett Ens, Tovi Grossman, Fraser Anderson, Justin Matejka, and George Fitzmaurice. 2015. Candid interaction: Revealing hidden mobile and wearable computing activities. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. ACM, 467--476.

Digital Library

[14]

Zackory Erickson, Jared Compiano, and Richard Shin. 2014. Neural Networks for Improving Wearable Device Security. (2014).

[15]

Charles W Eriksen and Yei-yu Yeh. 1985. Allocation of attention in the visual field. Journal of Experimental Psychology: Human Perception and Performance 11, 5 (1985), 583.

[16]

Md Sadek Ferdous, Soumyadeb Chowdhury, and Joemon M Jose. 2017. Analysing privacy in visual lifelogging. Pervasive and Mobile Computing (2017).

Digital Library

[17]

Gerhard Fischer. 2001. User modeling in human-computer interaction. User modeling and user-adapted interaction 11, 1--2 (2001), 65--86.

Digital Library

[18]

Jonna Häkkilä, Farnaz Vahabpour, Ashley Colley, Jani Väyrynen, and Timo Koskela. 2015. Design Probes Study on User Perceptions of a Smart Glasses Concept. In Proceedings of the 14th International Conference on Mobile and Ubiquitous Multimedia (MUM '15). ACM, New York, NY, USA, 223--233.

Digital Library

[19]

John Paulin Hansen, Anders Sewerin Johansen, Dan Witzner Hansen, Kenji Itoh, and Satoru Mashino. 2003. Command without a click: Dwell time typing by mouse and gaze selections. In Proceedings of Human-Computer Interaction-INTERACT. 121--128.

[20]

Adam Harvey. 2010. Camoflash-anti-paparazzi clutch. (2010). http://ahprojects.com/projects/camoflash/ accessed 13/09/2017.

[21]

Adam Harvey. 2012. CVDazzle: Camouflage from Computer Vision. Technical report (2012).

[22]

Sabrina Hoppe, Tobias Loetscher, Stephanie Morey, and Andreas Bulling. 2018. Eye Movements During Everyday Behavior Predict Personality Traits. Frontiers in Human Neuroscience 12 (2018).

[23]

Roberto Hoyle, Robert Templeman, Denise Anthony, David Crandall, and Apu Kapadia. 2015. Sensitive lifelogs: A privacy analysis of photos from wearable cameras. In Proceedings of the 33rd Annual ACM conference on human factors in computing systems. ACM, 1645--1648.

Digital Library

[24]

Roberto Hoyle, Robert Templeman, Steven Armes, Denise Anthony, David Crandall, and Apu Kapadia. 2014. Privacy Behaviors of Lifeloggers using Wearable Cameras. In International Joint Conference on Pervasive and Ubiquitous Computing (Ubicomp). ACM, 571--582.

Digital Library

[25]

Laurent Itti and Christof Koch. 2001. Computational modelling of visual attention. Nature reviews neuroscience 2, 3 (2001), 194.

[26]

Moritz Kassner, William Patera, and Andreas Bulling. 2014. Pupil: an open source platform for pervasive eye tracking and mobile gaze-based interaction. In Adj. Proc. UbiComp. 1151--1160.

Digital Library

[27]

Marion Koelle, Wilko Heuten, and Susanne Boll. 2017. Are You Hiding It?: Usage Habits of Lifelogging Camera Wearers. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '17). ACM, New York, NY, USA, Article 80, 8 pages.

Digital Library

[28]

Marion Koelle, Matthias Kranz, and Andreas Möller. 2015. Don't look at me that way!: Understanding User Attitudes Towards Data Glasses Usage. In Proceedings of the 17th international conference on human-computer interaction with mobile devices and services. ACM, 362--372.

Digital Library

[29]

Marion Koelle, Katrin Wolf, and Susanne Boll. 2018. Beyond LED Status Lights-Design Requirements of Privacy Notices for Body-worn Cameras. In Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction. ACM, 177--187.

Digital Library

[30]

Mohammed Korayem, Robert Templeman, Dennis Chen, David Crandall, and Apu Kapadia. 2014. Screenavoider: Protecting computer screens from ubiquitous cameras. arXiv preprint arXiv:1412.0008 (2014).

[31]

Mohammed Korayem, Robert Templeman, Dennis Chen, David Crandall, and Apu Kapadia. 2016. Enhancing lifelogging privacy by detecting screens. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4309--4314.

Digital Library

[32]

Katharina Krombholz, Adrian Dabrowski, Matthew Smith, and Edgar Weippl. 2015. Ok glass, leave me alone: towards a systematization of privacy enhancing technologies for wearable computing. In International Conference on Financial Cryptography and Data Security. Springer, 274--280.

[33]

Daniel J Liebling and Sören Preibusch. 2014. Privacy considerations for a pervasive eye tracking world. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication. ACM, 1169--1177.

Digital Library

[34]

Philipp Mayring. 2014. Qualitative content analysis: theoretical foundation, basic procedures and software solution. 143 pages.

[35]

Vivian Genaro Motti and Kelly Caine. 2016. Towards a Visual Vocabulary for Privacy Concepts. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 60. SAGE Publications Sage CA: Los Angeles, CA, 1078--1082.

[36]

Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. 2017. Towards a Visual Privacy Advisor: Understanding and Predicting Privacy Risks in Images. In International Conference on Computer Vision (ICCV 2017). Venice, Italy.

[37]

Alfredo J Perez, Sherali Zeadally, and Scott Griffith. 2017. Bystanders' Privacy. IT Professional 19, 3 (2017), 61--65.

Digital Library

[38]

Rebecca S Portnoff, Linda N Lee, Serge Egelman, Pratyush Mishra, Derek Leung, and David Wagner. 2015. Somebody's watching me?: Assessing the effectiveness of Webcam indicator lights. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 1649--1658.

Digital Library

[39]

Zachary Pousman, Giovanni Iachello, Rachel Fithian, Jehan Moghazy, and John Stasko. 2004. Design iterations for a location-aware event planner. Personal and Ubiquitous Computing 8, 2 (2004), 117--125.

[40]

Sören Preibusch. 2014. Eye-tracking. Privacy interfaces for the next ubiquitous modality. In 2014 W3C Workshop on Privacy and User-Centric Controls. https://www.w3.org/2014/privacyws/pp/Preibusch.pdf

[41]

Blaine A. Price, Avelie Stuart, Gul Calikli, Ciaran Mccormick, Vikram Mehta, Luke Hutton, Arosha K. Bandara, Mark Levine, and Bashar Nuseibeh. 2017. Logging You, Logging Me: A Replicable Study of Privacy and Sharing Behaviour in Groups of Visual Lifeloggers. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 2, Article 22 (June 2017), 18 pages.

Digital Library

[42]

Halley Profita, Reem Albaghli, Leah Findlater, Paul Jaeger, and Shaun K Kane. 2016. The AT Effect: How Disability Affects the Perceived Social Acceptability of Head-Mounted Display Use. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4884--4895.

Digital Library

[43]

Nisarg Raval, Animesh Srivastava, Kiron Lebeck, Landon Cox, and Ashwin Machanavajjhala. 2014. Markit: Privacy markers for protecting visual secrets. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication. ACM, 1289--1295.

Digital Library

[44]

Jeremy Schiff, Marci Meingast, Deirdre K Mulligan, Shankar Sastry, and Ken Goldberg. 2007. Respectful cameras: Detecting visual markers in real-time to address privacy concerns. In Intelligent Robots and Systems, 2007. IROS 2007. IEEE/RSJ International Conference on. IEEE, 971--978.

[45]

Roman Schlegel, Apu Kapadia, and Adam J Lee. 2011. Eyeing your exposure: quantifying and controlling information sharing for improved privacy. In Proceedings of the Seventh Symposium on Usable Privacy and Security. ACM, 14.

Digital Library

[46]

Jiayu Shu, Rui Zheng, and Pan Hui. 2016. Cardea: Context-aware visual privacy protection from pervasive cameras. arXiv preprint arXiv:1610.00889 (2016).

[47]

Julian Steil and Andreas Bulling. 2015. Discovery of Everyday Human Activities From Long-term Visual Behaviour Using Topic Models. In Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp). 75--85.

Digital Library

[48]

Julian Steil, Philipp Müller, Yusuke Sugano, and Andreas Bulling. 2018. Forecasting user attention during everyday mobile interactions using device-integrated and wearable sensors. In Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM, 1.

Digital Library

[49]

Yusuke Sugano and Andreas Bulling. 2015. Self-Calibrating Head-Mounted Eye Trackers Using Egocentric Visual Saliency. In Proc. of the 28th ACM Symposium on User Interface Software and Technology (UIST 2015). 363--372.

Digital Library

[50]

Yusuke Sugano, Xucong Zhang, and Andreas Bulling. 2016. Aggregaze: Collective estimation of audience attention on public displays. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 821--831.

Digital Library

[51]

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going Deeper with Convolutions. In Computer Vision and Pattern Recognition (CVPR). http://arxiv.org/abs/1409.4842

[52]

Robert Templeman, Mohammed Korayem, David J Crandall, and Apu Kapadia. 2014. PlaceAvoider: Steering First-Person Cameras away from Sensitive Spaces. In NDSS.

[53]

Marc Tonsen, Julian Steil, Yusuke Sugano, and Andreas Bulling. 2017. InvisibleEye: Mobile Eye Tracking Using Multiple Low-Resolution Cameras and Learning-Based Gaze Estimation. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT) 1, 3 (2017), 106:1--106:21.

Digital Library

[54]

Khai Truong, Shwetak Patel, Jay Summet, and Gregory Abowd. 2005. Preventing camera recording by designing a capture-resistant environment. UbiComp 2005: Ubiquitous Computing (2005), 903--903.

Digital Library

[55]

Roel Vertegaal et al. 2003. Attentive user interfaces. Commun. ACM 46, 3 (2003), 30--33.

Digital Library

[56]

Alan F Westin. 2003. Social and political dimensions of privacy. Journal of social issues 59, 2 (2003), 431--453.

[57]

Takayuki Yamada, Seiichi Gohshi, and Isao Echizen. 2013. Privacy visor: Method based on light absorbing and reflecting properties for preventing face image detection. In Systems, Man, and Cybernetics (SMC), 2013 IEEE International Conference on. IEEE, 1572--1577.

Digital Library

[58]

Jun Yu, Baopeng Zhang, Zhengzhong Kuang, Dan Lin, and Jianping Fan. 2017. iPrivacy: image privacy protection by identifying sensitive objects via deep multi-task learning. IEEE Transactions on Information Forensics and Security 12, 5 (2017), 1005--1016.

Digital Library

Cited By

Shalawadi SGetschmann Cvan Berkel NEchtler F(2024)Manual, Hybrid, and Automatic Privacy Covers for Smart Home CamerasProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661569(3453-3470)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661569
Xu AZhou ZMiyazaki KYoshikawa RHosio SYatani K(2024)DIPA2Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314397:4(1-30)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631439
Bhardwaj DPonticello ATomar SDabrowski AKrombholz K(2024)In Focus, Out of Privacy: The Wearer's Perspective on the Privacy Dilemma of Camera GlassesProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642242(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642242
Show More Cited By

Index Terms

PrivacEye: privacy-preserving head-mounted eye tracking using egocentric scene image and eye movement features
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interactive systems and tools
  2. Ubiquitous and mobile computing
    1. Ubiquitous and mobile devices

Recommendations

Privacy-aware eye tracking using differential privacy
ETRA '19: Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications

With eye tracking being increasingly integrated into virtual and augmented reality (VR/AR) head-mounted displays, preserving users' privacy is an ever more important, yet under-explored, topic in the eye tracking community. We report a large-scale ...
Fixation detection for head-mounted eye tracking based on visual similarity of gaze targets
ETRA '18: Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications

Fixations are widely analysed in human vision, gaze-based interaction, and experimental psychology research. However, robust fixation detection in mobile settings is profoundly challenging given the prevalence of user and gaze target motion. These ...
Dealing with head-mounted eye-tracking data: comparison of a frame-by-frame and a fixation-based analysis
ETSA '13: Proceedings of the 2013 Conference on Eye Tracking South Africa

Although analysing software for eye-tracking data has improved a lot during the last decennia, analysis of gaze behaviour recorded using a head-mounted device is still challenging. In this paper gaze behaviour of six participants, cycling on four ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ETRA '19: Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications

June 2019

623 pages

ISBN:9781450367097

DOI:10.1145/3314111

Conference Chairs:
Krzysztof Krejtz
SWPS University, Poland
,
Bonita Sharif
University of Nebraska-Lincoln

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 June 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

JST CREST

Conference

ETRA '19

Sponsor:

ETRA '19: 2019 Symposium on Eye Tracking Research and Applications

June 25 - 28, 2019

Colorado, Denver

Acceptance Rates

Overall Acceptance Rate 69 of 137 submissions, 50%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

50
Total Citations
View Citations
682
Total Downloads

Downloads (Last 12 months)132
Downloads (Last 6 weeks)10

Reflects downloads up to 25 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Shalawadi SGetschmann Cvan Berkel NEchtler F(2024)Manual, Hybrid, and Automatic Privacy Covers for Smart Home CamerasProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661569(3453-3470)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661569
Xu AZhou ZMiyazaki KYoshikawa RHosio SYatani K(2024)DIPA2Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314397:4(1-30)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631439
Bhardwaj DPonticello ATomar SDabrowski AKrombholz K(2024)In Focus, Out of Privacy: The Wearer's Perspective on the Privacy Dilemma of Camera GlassesProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642242(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642242
Bektaş KStrecker JMayer SGarcia K(2024)Gaze-enabled activity recognition for augmented reality feedbackComputers and Graphics10.1016/j.cag.2024.103909119:COnline publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1016/j.cag.2024.103909
Kurzhals K(2024)Anonymizing eye-tracking stimuli with stable diffusionComputers & Graphics10.1016/j.cag.2024.103898119(103898)Online publication date: Apr-2024
https://doi.org/10.1016/j.cag.2024.103898
Plizzari CGoletto GFurnari ABansal SRagusa FFarinella GDamen DTommasi T(2024)An Outlook into the Future of Egocentric VisionInternational Journal of Computer Vision10.1007/s11263-024-02095-7Online publication date: 28-May-2024
https://doi.org/10.1007/s11263-024-02095-7
Matsui TMisaki SSuwa HYasumoto K(2023)Privacy Filtering Using Word Embedding for 3D Point Cloud Based Spatial Sharing Systems2023 Fourteenth International Conference on Mobile Computing and Ubiquitous Network (ICMU)10.23919/ICMU58504.2023.10412252(1-7)Online publication date: 29-Nov-2023
https://doi.org/10.23919/ICMU58504.2023.10412252
Grootjen JSipatchin AWahl SMachulla TChuang LKosch T(2023)Assessing Eye Tracking for Continuous Central Field Loss MonitoringProceedings of the 22nd International Conference on Mobile and Ubiquitous Multimedia10.1145/3626705.3627776(54-64)Online publication date: 3-Dec-2023
https://dl.acm.org/doi/10.1145/3626705.3627776
Kurzhals K(2023)Privacy in Eye Tracking Research with Stable DiffusionProceedings of the 2023 Symposium on Eye Tracking Research and Applications10.1145/3588015.3589842(1-7)Online publication date: 30-May-2023
https://dl.acm.org/doi/10.1145/3588015.3589842
Bektaş KStrecker JMayer SGarcia DHermann JJenß KAntille YSolèr M(2023)GEAR: Gaze-enabled augmented reality for human activity recognitionProceedings of the 2023 Symposium on Eye Tracking Research and Applications10.1145/3588015.3588402(1-9)Online publication date: 30-May-2023
https://dl.acm.org/doi/10.1145/3588015.3588402
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents

-