A systematic review of extended reality (XR) for understanding and augmenting vision loss

XR for simulating the perception of people with low vision

Seventy-nine of the 90 identified low-vision perception studies (87.7%) relied on simulated low vision. An inexpensive means to simulate low vision for a sighted participant is the use of nonelectronic wearables, such as specially designed glasses, goggles, filters, and more (e.g., Kobashi, Kamiya, Shimizu, Kawamorita, & Uozato, 2012; Morris, Chaparro, Downs, & Wood, 2012; Scott, Atkins, Bentzen, & Barlow, 2012; Kanzler, Barth, Klucken, & Eskofier, 2016; Latham, Waller, & Schaitel, 2011). Modern alternatives include desktop displays or head-mounted displays that update the view based on where the user is looking (“gaze-contingent display”), which can be used to simulate specific eye conditions in real time (e.g., Kwon, Nandy, & Tjan, 2013; Seitz et al., 2020; Jones et al., 2020; also known as “altered reality,” Bao & Engel, 2019). While VR and AR headsets allow for similar experimental designs, researchers have direct control of the environment when using VR. The primary advantage of this approach is the ability to flexibly remove, add, or modify many different features of visual input.

Simulations are a valuable experimental tool for studying performance in tasks such as visual search (Addleman, Legge, & Jiang, 2021; Jones et al., 2020), face perception (Liu & Kwon, 2016; Tsank & Eckstein, 2017), reading (Huang et al., 2019; Latham et al., 2011), and navigation (Barhorst-Cates, Rand, & Creem-Regehr, 2019; Freedman, Achtemeier, Baek, & Legge, 2019; Zult et al., 2019). A prime example of this is OpenVisSim (Jones et al., 2020), which can track eye movements and simulate different gaze-contingent impairments in real time (Figure 4). To demonstrate the utility of OpenVisSim, Jones et al. (2020) simulated a central scotoma in VR (based on perimetric data from a person with glaucoma) and had sighted participants perform a visual search task with a Fove 0 headset and a mobility task with the HTC Vive. They demonstrated that the scotoma led to impaired performance in both tasks and found that a scotoma located in the upper visual field (inferior retina) led to worse performance, more eye movements, and more head movements.

Open in a separate window

Figure 4.

OpenVisSim conditions. (A) For a given fixation location (red cross), an example of simulated peripheral vision loss (“tunnel vision”) is shown. (B) Examples of visual changes associated with various low-vision conditions (reprinted under CC-BY from Jones et al., 2020).

However, the most commonly studied topic concerned the consequences of visual field loss on eye movements and associated behavior. It is well known that people with central visual field loss shift their oculomotor reference location from the fovea to an eccentric area known as the preferred retinal locus (PRL) (Bronstad, Bowers, Albu, Goldstein, & Peli, 2013). This happens gradually over time. Understanding PRL development and the behavioral consequences could potentially help people with low vision improve their oculomotor control in tasks such as reading and visual search (Kwon et al., 2013).

Many studies have thus trained sighted participants on a simulated scotoma with the help of the above-mentioned gaze-contingent displays, hoping that participants would develop a PRL. However, whereas some studies reported a shift in PRL with simulated low vision (SLV) in as fast as 3 hours (Kwon et al., 2013; Maniglia, Visscher, & Seitz, 2020; Seitz et al., 2020), others did not (David, Beitner, & Võ, 2020; Copolillo, Christopher, & Lyons, 2017; Almutleb et al., 2018). Longer explicit training times (i.e., 15 to 25 additional hours) have been shown to refine these effects to where oculomotor behavior was comparable to unimpaired controls (Kwon et al., 2013). Maniglia et al. (2020) conducted a systematic analysis of measures to understand how sighted participants can develop multiple PRLs and how individual participant re-referencing behavior is not consistent trial to trial even when trained. They found that roughly half of the participants exhibited saccadic re-referencing even without being instructed to do so (Maniglia et al., 2020). Kwon et al. (2013) showed that explicit training of a scotoma by highlighting estimated PRL locations effectively reduced the variance of fixation, much more than when participants were allowed to free-view the stimulus (i.e., the less variance, the more consistent their fixation locations).

Eye movements under SLV can vary drastically depending on how the impairment is presented to the participant (Kwon et al., 2013; David et al., 2020; Chow-Wing-Bom, Dekker, & Jones, 2020) and on what task is being studied (Tsank & Eckstein, 2017). For example, David et al. (2020) were able to show that saccade amplitude and fixation duration were significantly larger and longer with a simulated central scotoma, whereas the opposite effect was seen for a peripheral scotoma. Tsank and Eckstein (2017) showed that saccade patterns changed for an object-following and a visual-search task but not when identifying faces. In another study, McIlreavy, Fiser, and Bex (2012) were able to show that search times for targets and spatial distribution of gaze increased as the size of the simulated scotoma increased, while saccade amplitude and fixation duration remained unaffected.

While both Tsank and Eckstein (2017) and McIlreavy et al. (2012) provide insight into the effects of short-term impairment, it remains to be explored to which extent these simulations generalize to real people with low vision. Simulations of central vision loss with sighted participants have shown that PRLs can develop with training and that eye movements, while initially highly variable, can be refined over time (Liu & Kwon, 2016; Kwon et al., 2013; Maniglia et al., 2020; Seitz et al., 2020; Tsank & Eckstein, 2017). However, PRLs develop much quicker with sighted participants than with real AMD patients (Geringswald & Pollmann, 2015; Kwon et al., 2013), especially if the scotoma design includes a border and/or visual cueing for reference (Liu & Kwon, 2016; Seitz et al., 2020; Walsh & Liu, 2014). In contrast, people with AMD are often unaware of their scotoma location (Kwon et al., 2013).

XR for studying low-vision participants

To our surprise, only 20 of the 90 low-vision studies recruited participants with low vision (notable examples include Bowman & Liu, 2017; Powell, Powell, & Cook, 2020; Miura et al., 2018; Hoppe et al., 2020; Lin, Jan, Lay, Huang, & Chen, 2014; for the full list, please refer to the annotated spreadsheet in the Supplementary Materials), all of which were interested in studying how their oculomotor behavior differed from that of sighted people. Nine of 20 papers were interested in understanding how VR could be used to assess the behavior of BLV users. For example, Bowman and Liu (2017) trained low-vision participants in a street-crossing task. Four out of 12 participants were trained with real streets while the other 8 were trained with virtual streets using a three-screen VR projection system (subtending 168 × 35 degrees of visual angle). Both groups were tested on their street-crossing ability in real streets both before and after training. Before training, all participants demonstrated poor street-crossing skills (more than half of the responses were during “unsafe” times to cross). After training, over 90% of crossing responses were “safe.” Training with the VR system was comparable to training in real life, demonstrating how VR can be a powerful tool for practicing tasks that would otherwise be too dangerous or unfeasible within a laboratory setting (Bowman & Liu, 2017).

Lin et al. (2014) undertook the VR study with the largest sample size by recruiting 21 participants to perform a reading task while wearing a VR headset integrated with closed-circuit television magnification software. The head-mounted display with CCTV was used to obtain better depth of field and a higher modulation transfer function from the video camera. By sensing the parameters of the environment (e.g., ambient light level) and collecting the user’s specific characteristics, the system could make adjustments according to the user’s needs, which allowed participants to read more efficiently.

In sum, these studies highlight how VR headsets can be a tool for training and rehabilitation of improving reading skills for people with low vision.

XR for augmenting simulated low vision

A small number of studies in our corpus (n = 17) focused on digital image processing that may one day improve the behavioral performance of low-vision participants across different practical tasks. These visual augmentations were often added in real time to a gaze-contingent or a head-mounted display. For instance, low-level image manipulations such as increased text magnification and contrast were found to lead to faster reading speeds (Christen & Abegg, 2017), and enhancing the contours of faces and objects in a visual search task led to faster search times for older participants (Kwon et al., 2012). Other studies did not involve low-vision participants but instead added the visual augmentations on top of simulated low-vision conditions that were viewed by sighted participants. Christen and Abegg (2017) found that magnification was more beneficial for simulated blurry vision compared to a simulated scotoma, whereas contrast enhancement affected reading speed equally across simulated conditions. Similarly, temporal subsampling of an image (“image jitter”) was shown to improve peripheral acuity, word recognition, and facial emotion discrimination (Patrick, Roach, & McGraw, 2019; Watson et al., 2012).

Many of these studies aimed to understand how smartglasses could be used to benefit people with low vision (e.g., Hwang & Peli, 2014; Huang et al., 2019; Zhao, Hu, et al., 2017). See-through head-mounted displays such as Google Glass and Microsoft Hololens are systems that are commercially available for testing. Hwang and Peli (2014) measured contrast sensitivity for two conditions with three sighted participants: with or without AR edge enhancement and with or without a heavy diffuse film (Hwang & Peli, 2014). The enhancement being tested was the Laplacian edge detection method, where a positive method enhanced edges while a negative method enhanced the surrounding of edges. Contrast sensitivity thresholds had improved with the enhancement method (Hwang & Peli, 2014). Huang et al. (2019) tested 24 sighted participants on a navigation task with a voice-based sign reading application for the Hololens. All participants wore goggles modified with occlusion foils during the task to simulate reduced acuity. Results indicated that participants walked more slowly and took more time with the sign-reading application. Overall, participants walked on more direct paths and were more confident with the application.

XR for studying augmentations for low-vision participants

Although results from the above simulation studies are notable, the ultimate goal of an XR accessibility aid should be to improve the residual vision of real people with low vision. In line with this goal, the majority of studies in this category thus evaluated their prototypes on appropriate end users.

Several groups have explored how XR may benefit people with low vision perform different activities of daily living, such as navigating in unfamiliar environments or identifying objects of interest. For instance, “RealSense” (Yang, Wang, Hu, & Bai, 2016) is an AR application that automatically detects and highlights the traversable area in both indoor and outdoor environments (Figure 5A). Rather than highlighting nearby obstacles, the authors argued that highlighting the traversable area would better allow participants to plan their paths around obstacles. A related idea was presented by Zhao, Kupferstein, Rojnirun, Findlater, and Azenkot (2020), who used AR smartglasses to annotate the natural scene with “turn-by-turn” instructions for wayfinding akin to a car navigation system. Using a control condition that provided only audio feedback, the authors were able to demonstrate that low-vision participants made fewer mistakes and walked faster when using visual feedback.

Open in a separate window

Figure 5.

Examples of augmented reality in a head-mounted display. (A) “RealSense” is able to detect and highlight the traversable area in a variety of structured indoor environments (reprinted under CC-BY from Yang et al., 2016). (B) A depth camera designed for detecting people and obstacles while walking (reprinted under CC-BY from Hicks et al., 2013).

Houston et al. (2018) tested the ability of people with visual field loss to navigate a virtual mall while wearing specially designed glasses (peripheral prisms) that could expand the binocular visual field by up to 40 degrees. Twenty-four participants were asked to report obstacles and pedestrians while navigating the virtual mall. Interestingly, the detection of hazards on the same side of the visual field defect improved significantly for most participants, even without training (Houston et al., 2018). In another study, participants with various diagnosed forms of visual impairment were able to safely complete a stair navigation task with the help of an AR headset designed to highlight stair edges (Zhao, Kupferstein, Castro, Feiner, & Azenkot, 2019).

Few studies in our bibliography focused on improving reading. A notable exception is “ForeSee” (Zhao, Szpiro, & Azenkot, 2015; Zhao, Kupferstein, et al., 2019), an AR application that uses a combination of general low-level image enhancement methods (e.g., magnification, contrast enhancement, edge enhancement) for reading text in near- and far-distance viewing conditions. The benefit of “ForeSee” is that users can choose any of the enhancements presented in two display modes (full view or windowed), customizing the viewing experience as they see fit. Magnification and the windowed mode were the methods most preferred by participants, but the ability to use a combination of enhancements in real time was reported to have the strongest influence on a user’s viewing experience (Zhao et al., 2015).

Assistive devices are designed to help users see details (such as in reading), but often these devices are not designed to assist in other visual tasks. Enhancement of visual search was the main motivation for developing “CueSee”: an AR application to enhance recognition of targets with the help of five different attention enhancement cues, including magnification, color enhancement, flashing bounding boxes, and rotation. The researchers designed a search task with a mock grocery shelf in which different items were marked using AR tags (“Chilitags”; https://github.com/chili-epfl/chilitags), though a future iteration of the application may rely on real-time object recognition. Participants identified items on a grocery shelf significantly faster and more accurately using CueSee than without it. More importantly, participants preferred the CueSee enhancements over traditional cues (Zhao, Szpiro, Knighten, & Azenkot, 2016).

Whereas most studies focused on enhancing a user’s residual vision, others built custom head-mounted displays to simplify the visual scene. A notable example is the work by van Rheede et al. (2015), which built a headset integrated with an infrared depth camera to create a depth map, which was relayed to the user as a grayscale map: The closer the obstacle, the brighter the representation on the display (Figure 5B). Participants were then instructed to avoid foam obstacles while navigating a hallway; 6 of the 11 participants completed the obstacle course without any collisions.

Lastly, with XR systems becoming more common in low-vision research, Zhao, Cutrell, et al. (2019) asked how the user experience in VR itself may be improved. To address this question, they developed “SeeingVR” (Zhao, Cutrell, et al., 2019), a set of 14 visual enhancement tools that include digital magnification, brightness and contrast controls, edge enhancement, peripheral remapping, text augmentation, depth measuring, and text to speech, which can be overlaid post hoc onto any existing VR application. When asked to use a virtual keyboard, navigate an options menu on the screen, search for an object, or shoot a moving target while wearing a HTC VIVE, 11 participants with low vision completed the tasks much faster and more accurately with SeeingVR than without the overlay application (Zhao, Cutrell, et al., 2019). Moreover, users reported finding VR more enjoyable when using SeeingVR, making this work a promising first step toward the design of general accessibility standards for VR.

Although most studies in this subcategory focused on technology development, some also assessed the usability of the proposed XR systems. One study reported that most people with low vision preferred a compact device similar to a regular pair of glasses with buttons for inconspicuous interactions (Hoogsteen, Osinga, Steenbekkers, & Szpiro, 2020). Another study pointed to the portability of a head-mounted system paired with a smartphone as a camera as the preferred form factor for a reading aid (Stearns, Findlater, & Froehlich, 2018). The ForeSee work (Zhao et al., 2015) highlighted the need to give users the option to choose from several enhancement modes. Another AR study found that alphanumeric representation of information may be better for those with relatively higher acuity, whereas symbolic representation may be better suited for those with worse acuity (Lang, Schmidt, & Machulla, 2020). Audio feedback was generally liked by participants as well (Zhao et al., 2020): Although some participants preferred audio feedback because of shorter learning curves, all participants reportedly wanted to combine visual and audio features to refine their wayfinding experience.

XR for simulating prosthetic vision

To investigate functional recovery and experiment with different implant designs, researchers have been developing XR prototypes that rely on simulated prosthetic vision (SPV). The classical method relies on sighted subjects wearing a VR headset, who are then deprived of natural viewing and only perceive phosphenes displayed in the head-mounted display. This viewing mode has been termed “transformative reality” (Lui, Browne, Kleeman, Drummond, & Li, 2011, 2012) (as opposed to “altered reality,” which is typically used to describe simulated low-vision approaches; Bao & Engel, 2019). This allows sighted participants to “see” through the eyes of the bionic eye user, taking into account their head and/or eye movements as they explore a virtual environment (Kasowski, Wu, & Beyeler, 2021).

One application of SPV is assessing low-level visual function, and three studies were placed in this category. These studies focused on aspects like phosphene size (Lu et al., 2012) and shape (Cao, Li, Lu, Chai, & Wang, 2017) by varying stimulus and model parameters. Stimuli for these tasks are typically presented on a monitor (Lu et al., 2012), via AR glasses (Caspi & Zivotofsky, 2015), or in a head-mounted display (Cao et al., 2017). One prominent example (Caspi & Zivotofsky, 2015) used sighted volunteers to complete a Landolt-C visual acuity task using SPV. Participants wore custom AR glasses to view webcam input that was converted to an 8 × 8 pixel image, meant to represent the limited resolution of current retinal implants. The authors found that well-performing participants developed similar strategies to those employed by real prosthesis users, such as scanning the image using strategic head movements. Interestingly, by utilizing head movements, the participants were able to surpass the theoretical acuity limit. This phenomenon had previously been identified in real prosthesis users (Humayun et al., 2012), and the authors hypothesized it was the accumulation of information over time. By utilizing a simple low-level visual function XR experiment, Caspi and Zivotofsky (2015) were able to confirm this hypothesis.

The majority of studies in this section focused on slightly more complex tasks such as letter (Zhao et al., 2011), word (Fornos, Sommerhalder, & Pelizzone, 2011), face (Denis, Jouffrais, Vergnieux, & Macé, 2013; Chang, Kim, Shin, & Park, 2012), and object recognition (Zhao et al., 2010; Wang, Sharifian, Napp, Nath, & Pollmann, 2018; Macé, Guivarch, Denis, & Jouffrais, 2015). This group of studies had the highest average number of subjects (μ = 21.06 ± 12.34) when compared to other areas of SPV studies. In most setups, participants would view SPV stimuli in a conventional VR headset, but a large portion used a monitor with some sort of eye tracking. Surprisingly, although the majority of the tasks used head-mounted displays, none of the studies allowed for a fully immersive experience that would allow the subject to walk around and interact with the environment. Studies in this category primarily used SPV to study basic behavior in these tasks, but some studies also used these tasks to focus on another behavior. One example is the work by Sanchez Garcia, Martinez-Cantin, Bermudez-Cameo, and Guerrero-Campo (2020). In this work, participants were tasked to find and recognize objects in a scene with different fields of view (20°, 40°, or 60°) and number of phosphenes (200 or 500). The authors showed counterintuitive results, with a higher field of view resulting in significantly worse performance and longer recognition times. However, they argued that phosphene density may be more important for object recognition than field of view, which is consistent with earlier findings (van Rheede, Kennard, & Hicks, 2010). Ho, Boffa, and Palanker (2019) relied on AR smartglasses to simulate the artificial vision provided by the PRIMA subretinal implant (Lorach et al., 2015) (Figure 6A). This device was developed for people with geographic atrophy as commonly experienced with AMD, where vision is first lost in the macula. To simulate this, the authors needed to combine SPV in the macula and natural vision in the periphery. The authors accomplished this by using AR smartglasses with black tape occluding the central field of view so only the LED overlay was visible in this area. With this setup, they were able to make testable predictions about the visual acuity to be expected from PRIMA (Lorach et al., 2015), which is currently in clinical trials.

Open in a separate window

Figure 6.

Examples of augmented reality systems used to simulate prosthetic vision with sighted participants. (A) AR glasses for mimicking the prosthetic vision seen by a participant with geographic atrophy (reprinted under CC-BY from Ho et al., 2019). The front camera of the AR glasses captured the video stream, while custom software preloaded on the glasses adjusted the video quality to mimic prosthetic vision (bottom). (B) AR system to evaluate the benefit of gaze compensation on hand–eye coordination (reprinted under CC-BY from Titchener, Shivdasani, Fallon, & Petoe, 2018). Phosphenes were rendered as Gaussian blobs (top). Participants wore a simulated prosthetic vision headset that included a front-facing camera, head motion tracker, and eye tracker (bottom). (C) Simulated prosthetic vision in retinitis pigmentosa. Residual vision covers the central 10^o field of view, and simulated electrode arrays provide bionic vision in the degenerated periphery (reprinted under CC-BY from Zapf, Boon, Matteucci, Lovell, & Suaning, 2015).

Lastly, a number of studies focused on spatial cognition tasks, such as obstacle avoidance (Zapf, Boon, Lovell, & Suaning, 2015, 2016; Endo et al., 2019) and wayfinding (Vergnieux, Macé, & Jouffrais, 2014). By design, these tasks require a more immersive setup that allows for the incorporation of head and eye movements as well as locomotion (Kasowski & Beyeler, 2022). Most of these studies incorporated a fully immersive design for their task, although a few used VR headsets but required their subjects to sit/stand in place and instead use a keyboard/controller to move. The majority of tasks were simply “proof-of-concept” experiments showing that users were able to navigate effectively with SPV. A notable example is Zapf, Boon, Matteucci, et al. (2015), who simulated tunnel vision as typically encountered during retinitis pigmentosa by restricting participants to their central 10° field of view in a virtual environment. Eleven participants completed a variety of tasks consisting of low-lying obstacle circumvention (avoiding traffic cones), static/moving pedestrian avoidance (navigating a corridor with stationary/moving virtual characters), and path following (following a path through parked cars). The authors then wanted to know how behavioral performance might improve when visual cues in the (presumed degenerated) periphery were provided by a simulated retinal implant (see Figure 6). Although behavioral performance improved for avoiding low-lying obstacles and following paths, the simulated prosthetic vision in the periphery could not help participants avoid stationary head-level targets.

XR for studying prosthesis users

Rachitskaya et al. (2020) was the only study to use XR for visual rehabilitation of real bionic eye users. It was also the only study to mention consultation with the BLV community during development, having utilized an interdisciplinary team that incorporated ophthalmologists and rehabilitation specialists. Since there currently is no standardized procedure for vision rehabilitation across different Argus II implantation centers, Rachitskaya et al. (2020) developed a Computer-Assisted Rehabilitation Environment (CAREN), which consists of a motion capture system, control software with a 180° curved projection screen, a motion platform, and a treadmill. Participants donned a harness, had access to handrails on the treadmill, and were accompanied by a physical therapist. After using CAREN twice a week for 4 weeks, participants showed significant improvements in walking speed and object localization, demonstrating that immersive technology may provide a solution for the standardization of effective rehabilitation approaches to augment bionic eye performance.