Glow in the Dark: Using a Heat-Sensitive Camera for Blind Individuals with Prosthetic Vision

Abstract

Introduction

Retinal prosthetic implants are targeted at patients with end-stage outer retinal degenerative diseases such as retinitis pigmentosa (RP) and age-related macular degeneration. These diseases primarily affect photoreceptors and the retinal pigment epithelium, and only indirectly the bipolar cells and ganglion cells. More specifically, due to the loss of input from photoreceptors in RP, the other layers of the retina change in such a way that at the end stage of the disease the affected retina is no longer recognizable as an orderly layered structure of parallel processing units (Jones et al., 2003; Marc et al., 2007). Changes in the retina include new connections between bipolar cells, which result in signals spreading out horizontally through the retina. This particularly limits the achievable resolution using electrical stimulation (Dagnelie, 2012; Marc et al., 2007). To date, there is no cure to prevent or stop the degeneration progress that happens slowly over decades. Yet even in the end-stage of these diseases there are surviving ganglion cells that can send signals to the brain through the optic nerve in response to electrical stimulation (Humayun et al., 1996; Santos et al., 1997).

To deliver the most effective stimulation to the remaining ganglion cells, one challenge is to get close to these cells. Surface electrodes have been shown to be safe and provide visual perception, but if the implant is not perfectly apposed to the retina they may remain far from the ganglion cell layer and thus, require a large surface area in order to sufficiently stimulate neurons without exceeding safe charge density limitations (De Balthasar et al., 2008). To date, the Alpha AMS by Retina Implant AG in Germany (Stingl et al., 2017; no longer in production) and Argus II by Second Sight Medical Products in CA, USA (Cruz et al., 2016; no longer in production) are the only retinal implants that have been approved for clinical use, but they offer only limited functional vision to blind subjects, best described as “moving shadows.”

The Argus II consists of a 6 × 10 array of round platinum-coated 200 μm diameter electrodes that stimulate the surviving ganglion cells in the retina, and it covers 11° × 19°, or ~ 22° diagonally, of the visual field. The stimulation is based on head mounted-camera imagery that is fed into a video processing unit (VPU). The VPU converts the image into grayscale, reduces the resolution to that of the electrode array, and generates the signals to stimulate the 60 electrodes through a wireless telemetry link (Stronks & Dagnelie, 2014).

The Argus II has been implanted in more than 300 blind individuals worldwide. It has been shown to improve performance for locating and identifying the direction of motion of an object, and is helpful for orientation and mobility in high contrast settings in laboratory experiments (Dagnelie, 2006; Dagnelie et al., 2017; Humayun, de Juan, & Dagnelie, 2016; Humayun et al., 2009, 2003; Stronks & Dagnelie, 2014). However, under normal circumstances, the visual field of an Argus II wearer may be full of people and objects at multiple distances. The complex scene, low resolution, and spread of elicited activity through the retina combine to yield a poorly structured blurry image.

In real life, Argus II users will want to find and communicate with people, detect and avoid obstacles, and manipulate objects. With the regular Argus II camera, stimulation is based on light intensity throughout the visible image. Due to the charge spread mentioned above, in low contrast and/or brightness situations, using a visible-light camera may cause the target to blur into the background requiring Argus II users to spend more time and energy to perform even a simple task. In many situations, it may not be possible to compensate by using other senses, e.g., hearing and touch, by hand or cane. For example, when handling a hot pot on a stove, caution is required using only touch, which may result in a dangerous accident. When looking for someone to ask for assistance in a public place or looking for an empty seat in a crowded auditorium, one needs to distinguish people from inanimate objects. Thus, the use of other senses alone may not eliminate safety hazards or improve performance. A better solution may be to simplify the input image before presenting it to the user. The approach investigated in this study was to utilize a thermal imager as an alternative to the normal Argus II visible-light camera. A thermal camera filters out cooler objects and presents only warmer objects in a visual scene regardless of scene illumination and target contrast.

Thermography or thermal imaging has been successfully used in a wide range of fields that require safety monitoring; these include applications in firefighting to detect smoke and heat (Cope, Arias, Williams, Bahm, & Ngwazini, 2019), road safety in vehicles with an autonomous operation mode to detect pedestrians (Miethig, Liu, Habibi, & Mohrenschildt, 2019), military and defense for target recognition (Stout, Madineni, Tremblay, & Tane, 2019), and even disease control by screening people at airports or hospitals for elevated body temperatures related to epidemics such as Covid-19, SARS and bird flu (Lee et al., 2020; Ng, Kaw, & Chang, 2004). Moreover, thermal imaging has found new uses in medical imaging such as corneal thermography to assess corneal integrity in the study of dry eye, corneal ulcers and diabetic retinopathy (Konieczka, Schoetzau, Koch, Hauenstein, & Flammer, 2018), and a recent case report discusses the utility of thermal imaging in Horner Syndrome showing anhidrosis (Henderson, Ramulu, & Lawler, 2019).

The ability to detect warm objects could be helpful to blind individuals for locating hazards such as a stove or toaster, or desired objects such as a hot cup of coffee or a meal. In addition, people and pets are visible in the thermal image, which makes it easier for users to locate and differentiate them from inanimate objects. We first reported the use of thermal imaging for Argus II users during ARVO 2016 (Dagnelie et al., 2016). In two recently published studies, Argus II users performed two stationary tasks, a mobility task, and an orientation/mobility task with a thermal camera; the results suggested some benefit of using a thermal camera for Argus II users and potentially other low vision individuals (He, Sun, Roy, Caspi, & Montezuma, 2020; Montezuma et al., 2020). The present study bears similarities to those of Montezuma et al. and He et al., but differs in three important respects: 1) We chose our tasks to be more diverse, and more similar to what an Argus II user would encounter in the real world; 2) all tasks were structured as m-alternative forced choice tasks (m-AFC) – on each trial there were m ≥ 2 possible response alternatives with precisely one of the response alternatives defined as “correct”; 3) we used a novel analysis framework that allows quantification of subject ability and task difficulty regardless of the number of response alternatives m. Subject ability and task difficulty are latent variables, and to estimate these we used an extension of signal detection theory (SDT) to latent variables (Bradley & Massof, 2019) that transforms percent correct scores into d′ units and thus, enables comparison of subject ability across tasks with different chance performance levels. The new method provides calibrated task difficulties which can be used to estimate person abilities on the same scale in different studies; it also provides estimated person abilities which can be compared to future performance, for example during a treatment trial.

There are standardized tests in the literature to measure visual function in individuals with ultra-low and prosthetic vision, (Bach, Wilke, Wilhelm, Zrenner, & Wilke, 2010; Bailey, Jackson, Minto, Greer, & Chu, 2012) and a set of lab-based functional vision measures has been developed for Argus II users (Ahuja et al., 2011; Caspi & Zivotofsky, 2015; Dorn et al., 2013). However, such measures are not informative for device functionality in daily life, and more representative tests have not yet been developed for the level of vision afforded by the Argus II; nor did previous reports take thermal imaging into account. Thus, the present study aimed to quantitatively compare the benefits offered by thermal and regular Argus II cameras in a set of standardized situations approximating real life.

Methods

Thermal Camera System

In our experiment, we used a miniature heat-sensitive infrared camera (Lepton; FLIR Corp.) in a custom-designed frame. The heat-sensitive camera feeds radiometric information to an electronics box that generates and transfers the thermal image of the scene to the Argus II VPU. The VPU communicates stimulation parameters based on the local thermal image intensity to each electrode through the transmitter coil (Fig. 1). This real-time stimulation results in visual percepts corresponding to the location of warm objects in the scene.

Open in a separate window

Fig. 1.

Side-by-side views of the external Argus II system components and the heat-sensitive adaptation. A) A miniature head-mounted video camera sends a grayscale image to the VPU, and the VPU transfers energy and the real-time stimulation parameters to the implant array through the transmitter coil. B) Heat-sensitive camera connected to the electronics box that processes and transfers the thermal camera images to the VPU for transmission to the implant through the external transmitter coil.

Study Design

Four experienced Argus II users participated in this study (Table 1) and were asked to perform 5 tasks while wearing either the normal Argus II glasses with a visible-light camera or a glasses frame with the heat-sensitive camera. All tasks in our experiment were designed to approximate real-life scenarios such as identifying stationary and moving targets. Target sizes as small as a cup and as large as a human were chosen to assess how useful information from the thermal camera was for Argus II users compared to information from the visible-light camera. The study was approved by the Johns Hopkins Medical Institutions IRB and followed the tenets of the Declaration of Helsinki. All subjects signed an informed consent form to participate after being informed of the goals and methods of the study.

Table 1.

Demographics table for the four subjects who participated in this study.

Subject	Age	Gender	Years of using Argus II	Number of Active Electrodes
S1	87	Male	12	55
S2	80	Male	10	56
S3	67	Female	5	57
S4	58	Male	4	57

Open in a separate window

Fig. 2 shows example image frames with visible-light and heat-sensitive cameras that were used in this study. The image frame of the thermal camera only contains the face and arms and removes all other cooler objects that are visible in the normal view. This potentially helps subjects perceive the location of warm targets more accurately.

Open in a separate window

Fig. 2.

Image frame examples A) from a visible-light camera and B) a thermal camera. Approximation of Argus II user percept from the image frame example with C) visible-light camera and D) a thermal camera.

As mentioned above, all 5 tasks were structured as m-AFC. The tasks were designed to cover a range of difficulty levels encountered in daily life (Table 2). In all 5 tasks, subjects were seated with no constraints on head movement, and no time limits were imposed. Prior to this study, all four subjects had participated in thermal camera system testing, on similar tasks, so they were familiar with the characteristics of thermal imaging. At the beginning of each task, subjects were given 15 minutes to practice locating the target with both cameras.

Table 2.

Tasks performed using the thermal and the visible-light camera. These tasks were designed to approximate real-life scenarios to investigate the possible benefit of using the heat-sensitive camera.

Task Number	Task Name	Detail	Data Collection	Trial Count per subject, per camera
1	Cups	Report the side of closer cup	2-AFC	40
2	Bowls	Report the location of the missing bowl if there is any	5-AFC	80
3	Distance discrimination	Report the distance of the standing person	3-AFC	60
4	Direction of person walking	Report the direction of the person’s movement	2-AFC	100 (opportunity trials)
5	Direction of closer escalator	After 10 passengers passed by, report the direction of close escalator	2-AFC	10

Open in a separate window

Tasks details:

1. Cups: Two white cups with warm water were placed on a black table in front of the subject, at 53.3 and 78.7 cm distances. The transverse (left-right) distance between the cups was 25.4 cm. The subject was asked to report which cup was closer. The response of left or right was recorded.
2. Bowls: Three or four white bowls with warm water were located at the 3, 6, 9, and 12 o’clock positions on a black square table in front of a seated subject, similar to a meal set for four people; in a randomly chosen 80% of the trials one bowl was removed, with the missing bowl being chosen randomly, balanced across conditions. The subject was asked to detect if any bowl was missing, and if so, report the location of missing bowl. The subject responded with one of five possible choices: “full set”, or 3, 6, 9, or 12 o’clock is missing.
3. Distance discrimination: A person with a white shirt was standing in front of the seated subject at 1, 3, or 6 m and the subject was asked to report the location of the person. The responses of 1, 3, or 6 m was recorded.

   • Direction of motion tests near a pair of escalators (tasks 4 and 5): the subject was seated 536 cm to the side of a pair of escalators moving in opposite directions and observed people moving onto and off the escalators (Fig. 3). The subject could not see the escalators themselves but could use size cues provided by people walking onto and off the escalators to determine in which directions the closer and farther escalators were moving. Due to the real-life nature of the scenario, we had no control over the number of people passing in each direction; however, an off-line review of the video recordings confirmed that approximately 50% of passengers traveled in each direction over the course of the experiment.
     

     Open in a separate window

     Fig. 3.

     Subjects’ view for tasks 4 and 5 is indicated by the white rectangle, which approximates the visual field of an Argus II user.

4. Direction of person walking: if any motion was detected that indicated a person walking onto or off the escalators, the subject was asked to specify the direction of motion (left or right). The subject was not informed if a person was passing by. Each person passing by and detected constituted a trial for this task; if the subject failed to detect the person, the trial was not included in the direction of person walking analysis for this task. A total of 100 persons passed by in either direction, thus the maximum possible number of trials was 100, but the actual (i.e., detected) number of trials was lower, and varied by subject and by camera system.
5. Direction of closer escalator: Each time 10 people had passed by (on either escalator, and whether the subject detected them or not), the subject was asked to report in which direction the closer escalator was moving (left or right) using any available clue, e.g., size of the persons. Since the passing of 10 persons (detected or not) constituted a trial, there were 10 trials in this task. To avoid subjects learning the true escalator direction, the video image was left/right reversed for a randomly chosen 5 out of these 10 trials; subjects also wore ear plugs to prevent the use of auditory cues.

Results

All subjects successfully completed the cups, bowls and distance discrimination tasks in a lab setting with both heat-sensitive and visible-light cameras. The direction of person walking, and direction of closer escalator tasks were performed at a public escalator, and only subjects 2 and 4 completed these tasks with both heat-sensitive and visible-light cameras. Subject 3 completed the tests near the pair of escalators only with visible-light camera and subject 1 did not perform this task, due to limited access.

In tasks 4 and 5 (the tasks near a pair of escalators), using a visible-light camera, out of 100 passengers that passed by, subject 2 detected 9 passengers, subject 3 detected 35 passengers, and subject 4 detected 29 passengers. When the heat-sensitive camera was used, out of 100 passengers that passed by, subject 2 detected 33 passengers and subject 4 detected 69 passengers (no data for subject 3; Table 3). Performance with the thermal camera fell well outside the estimated confidence intervals for performance with the normal Argus II camera (subject 2: 95% CI [4, 16]; subject 4: 95% CI [20, 39]).

Fig. 4 shows estimated item measures and 95% confidence intervals for each task across all subjects and conditions. All estimated item measures were negative and showed the range of difficulty of the tasks in our experiment. More negative d′ item measures represent easier tasks, and d′ = 0 represents chance performance for the average person.

Open in a separate window

Fig. 4.

Estimated item measures in d′ units show the range of difficulty levels for the tasks in our experiment. Error bars specify 95% confidence intervals.

The direction of closer escalator task (task 5) was particularly difficult compared to other tasks because after every 10 passengers passing by the subjects had to infer the direction of the closer escalator based on the movements of the detected passengers (the subject was required to guess if none of the 10 passengers were detected on a given trial). With the visible-light camera, subject 2 detected between 0 and 3 out of 10 passengers per trial, subject 3 detected between 2 and 7 out of 10 passengers per trial, and subject 4 detected between 1 and 5 out of 10 passengers per trial. With the thermal camera, subject 2 detected between 2 and 5 out of 10 passengers per trial, and subject 4 detected between 5 and 9 out of 10 passengers per trial (Table 3). A Wilcoxon–Mann–Whitney two-sample rank-sum test showed that these differences were statistically significant for the two subjects who completed the tasks with both cameras (for both subjects p < 0.001; subject 2 Mann-Whitney U = 150; and subject 4 Mann-Whitney U = 152).

Fig. 5 shows estimated person measures for each subject in the two camera conditions across all tasks, with d′ = 0 representing the average ability of all subjects across all tasks and cameras. The thermal camera improved performance by 0.83, 0.73, 1.00 and 0.73 d^′ units for subjects 1 to 4, respectively. Welch’s test showed that this performance difference was significant across all m-AFC tasks for all subjects (all four p < 0.001; for subject 1, t_(370.7) = −215.5; subject 2, t_(482.2) = −262.8; subject 3, t_(346.9) = −263.1; subject 4, t_(465.6) = −223.7).

Open in a separate window

Fig. 5.

Estimated person measures for each subject across all tasks, relative to the average person measure (at d′ = 0) across all tasks, are plotted for the heat-sensitive camera (red squares) and visible-light camera (blue circles). Higher person measures indicate more capable persons. The error bars show the estimated standard errors.

Fig. 6 shows estimated person measures for each task, plotted relative to the average person ability (d′ = 0) in that task, across all subjects in the two conditions. Note that average person ability in each task varied due to different levels of task difficulty (see Fig. 4), which means that for each task d′ = 0 represents a different level of person ability. Despite the differences between subjects and tasks, the estimated person measures show that all tasks were easier for subjects with the heat-sensitive camera than with the visible-light camera (using Welch’s test all five p < 0.001; for task 1, t_(301.4) = −139.5; task 2, t_(656.9) = −239.4; task 3, t_(470.7) = −533.2; task 4, t_(34.4) = −12.5; task 5, t_(148.7) = −64.1). In order of benefit level, the thermal camera improved performance in the distance discrimination task by 1.43 d′ units, in the cups task by 0.68 d′ units, in the direction of person walking task by 0.64 d′ units, in the bowls task by 0.51 d′ units, and in the direction of closer escalator task by 0.42 d′ units.

Open in a separate window

Fig. 6.

Estimated person measures across subjects, by task, relative to the average person ability (d′ = 0), for the thermal (red squares) and visible-light (blue circles) cameras. Vertical lines separating the tasks are for emphasis that each task has its own performance average. Error bars show the estimated standard errors.

Discussion and conclusion

In this study, to improve the current Argus II system, we investigated the benefit of using a heat-sensitive camera compared to the regular Argus II camera in tests approximating real-life scenarios. As expected, the thermal camera simplified the information provided to Argus II users and improved performance in tasks where temperature differences play a role.

At the end of the test, subjects were asked how useful the thermal camera is and in what tasks they might use it. Subjects indicated that it takes too long to understand the phosphene vision conveyed by the current Argus II system, and the addition of a thermal camera to the Argus II system could improve performance of certain activities. For example, subjects 2 and 3 suggested safety might improve when using a cane with the thermal camera. Subject 4 mentioned the increased ease in locating people with a thermal camera while having a conversation. Subject 1 reported that he is willing to try the new camera even though he can achieve his daily goals without using the Argus II system.

The d′ analysis for the m-AFC tasks showed that the heat-sensitive camera significantly improved performance for the diverse set of tasks in our study. Even for the easier tasks, such as the bowls task where average performance of all subjects with both cameras was much better than chance level, subject accuracy improved when using the thermal camera. The average improvement across all tasks combined was similar for all subjects, regardless of their baseline performance with the normal Argus II camera (Fig. 5). In real-life, this could mean that the chance of knocking over a hot cup or touching a very hot object is much lower with the thermal camera because the user can more readily locate the object. The heat-sensitive camera also offers benefits for distinguishing persons from other objects in a room, as well as determining how far away they are by using size cues (for a known object). In our experiments, subjects were significantly better at distinguishing the distance of a standing person at 1, 3, or 6 m with the thermal camera than with the regular Argus II camera. Our experiments also showed that the thermal camera improved subjects’ abilities to not only detect a walking person, but also identify the direction of the person walking. This could help with orientation and with safely walking toward a target location in a crowded place such as a metro station.

One limitation to the use of a thermal camera is that it only detects warm targets. In the absence of a warm target, a thermal imager filters out all cold objects in the scene and will not provide any useful information. However, in the presence of a warm target, a thermal camera detects the target regardless of the brightness and luminance contrast. Since the normal Argus II camera depends on visible light, it requires high contrast and enough brightness to show a target object against a background, and even then, the object may be hard to distinguish from a cluttered background. Therefore, a thermal camera would be a useful addition to the current Argus II system if added in such a way that the user can switch between using the thermal camera and the normal Argus II camera. For example, consider a situation where a subject enters a room with people sitting on chairs and looks for an empty chair to sit down. The user may perceive both chairs and people similarly with the normal camera, but only people will be visible when using a thermal camera, so by switching back and forth between two cameras the user will be able to distinguish the empty chair from the occupied chairs. If the image frames from thermal and visible-light cameras can be fused, then warm targets can be highlighted within normal settings without need for a switch. A production prototype with these capabilities will be tested in upcoming studies.

One possible alternative to using a thermal camera for detecting and identifying objects is using machine learning algorithms. However, the robustness of these algorithms depends on many factors such as the luminance contrast, brightness, and the quality of the training set data (Dhillon & Verma, 2020; Padilla, Netto, & Da Silva, 2020). For example, it is still a major challenge for face recognition algorithms to identify a person’s face from every angle in a wide range of light conditions (Kumar Dubey & Jain, 2019). Another challenge is how to convey the information to a blind user without adding more confusion. A thermal camera is simple, has a low processing cost, and subjects who participated in our study learned how to use the system within a few minutes.

We have shown that all four subjects benefited from the thermal camera in stationary tasks. However, we have not tested the system in mobility tasks where the user is required to move through the environment. Temperatures may change when moving from one location to another and warm targets may be detected differently. A recently published study (He et al., 2020) reported a benefit for Argus II users of using a thermal camera in a mobility and an orientation task in an indoor setting. However, it is important to also test the efficacy of the system in outdoor settings where buildings and hot pavement in the summer might look brighter than people when using the thermal camera. In future studies, it is important to assess how the heat-sensitive camera would be useful to each person’s unique needs, and whether a blended image from thermal and visible-light sensitive cameras may have greater utility for certain tasks. If such tests use m-AFC scenarios they should employ the analysis used in the present study.

Overall, this study demonstrated the efficacy of the heat-sensitive camera for Argus II users in a set of standardized situations approximating real-life scenarios. The results show that a thermal camera can be helpful to detect hot objects and distinguish people, at different distances. The item measures estimated for the tasks in this study can also be used as a guideline for further training and rehabilitation.

Acknowledgement

Footnotes

Contributor Information

Roksana Sadeghi, Department of Biomedical Engineering, Johns Hopkins School of Medicine, Baltimore, MD, USA.

Arathy Kartha, Department of Ophthalmology, Johns Hopkins School of Medicine, Baltimore, MD, USA.

Michael P. Barry, Second Sight Medical Products, Sylmar, California, United States.

Chris Bradley, Department of Ophthalmology, Johns Hopkins School of Medicine, Baltimore, MD, USA.

Paul Gibson, Advanced Medical Electronics Corporation, Maple Grove, MN, USA.

Avi Caspi, Second Sight Medical Products, Sylmar, California, United States; Jerusalem College of Technology, Jerusalem, Israel.

Arup Roy, Second Sight Medical Products, Sylmar, California, United States.

Gislin Dagnelie, Department of Ophthalmology, Johns Hopkins School of Medicine, Baltimore, MD, USA.

References