Learn more: PMC Disclaimer | PMC Copyright Notice
Detection, eye–hand coordination and virtual mobility performance in simulated vision for a cortical visual prosthesis device
Abstract
In order to assess visual performance using a future cortical prosthesis device, the ability of normally sighted and low vision subjects to adapt to a dotted ‘phosphene’ image was studied. Similar studies have been conduced in the past and adaptation to phosphene maps has been shown but the phosphene maps used have been square or hexagonal in pattern. The phosphene map implemented for this testing is what is expected from a cortical implantation of the arrays of intracortical electrodes, generating multiple phosphenes. The dotted image created depends upon the surgical location of electrodes decided for implantation and the expected cortical response. The subjects under tests were required to perform tasks requiring visual inspection, eye–hand coordination and way finding. The subjects did not have any tactile feedback and the visual information provided was live dotted images captured by a camera on a head-mounted low vision enhancing system and processed through a filter generating images similar to the images we expect the blind persons to perceive. The images were locked to the subject’s gaze by means of video-based pupil tracking. In the detection and visual inspection task, the subject scanned a modified checkerboard and counted the number of square white fields on a square checkerboard, in the eye–hand coordination task, the subject placed black checkers on the white fields of the checkerboard, and in the way-finding task, the subjects maneuvered themselves through a virtual maze using a game controller. The accuracy and the time to complete the task were used as the measured outcome. As per the surgical studies by this research group, it might be possible to implant up to 650 electrodes; hence, 650 dots were used to create images and performance studied under 0% dropout (650 dots), 25% dropout (488 dots) and 50% dropout (325 dots) conditions. It was observed that all the subjects under test were able to learn the given tasks and showed improvement in performance with practice even with a dropout condition of 50% (325 dots). Hence, if a cortical prosthesis is implanted in human subjects, they might be able to perform similar tasks and with practice should be able to adapt to dotted images even with a low resolution of 325 dots of phosphene.
1. Introduction
Experiments in the late sixties and early seventies on humans (Brindley and Lewin 1968, Brindley 1970, 1982, Dobelle and Mladejovsky 1974, Dobelle et al 1974, Dobelle 2000), demonstrated that individual dots of light or a sense of light called ‘phosphene’ could be evoked by stimulating the visual cortex. These studies have also confirmed the visuotopic organization of the visual cortex, and demonstrated that subjects could assimilate information that was delivered to the visual cortex by electrical currents passed via groups of electrodes. This early work provided the path to pursue an approach to the intracortical visual prosthesis in which fine wire metal electrodes are inserted into the visual cortex for selective stimulation (Bak et al 1990, Schmidt et al 1996, Bradley et al 2005, Troyk et al 2002, 2003, 2005). The goal of a cortical visual prosthesis device is to give the blind patient some useful vision with the ability to perform simple tasks: recognize basic shapes, perform simple eye–hand coordination tasks, navigation and reading large printed text. Before a cortical prosthesis device is implanted in a blind volunteer, it is required to estimate as what should be expected from this implantation.
In the field of visual prosthesis, psychophysical simulation studies have been used to understand the requirements of a visual prosthesis device and asses the performance expected from these devices (Cha et al 1992a, 1992b, 1992c, Thompson et al 2003, Humayun et al 2003, Boyle et al 2003, Hayes et al 2003, Dowling et al 2004, Chen et al 2004, 2005, Dagnelie et al 2006, 2007, Hallum et al 2007). Cha et al (1992a, 1992b, 1992c) conducted psychophysical tests to understand the requirement of a cortical prosthesis device for reading and mobility tasks. They found that a 25 × 25 array of pixels with a visual field of 1.7° is sufficient to provide good reading performance, and the same 25 × 25 pixel array with a minified 30° field of view was good enough to provide good mobility performance. Cha et al (1992a, 1992b, 1992c) in their experiments assumed that an evenly placed square electrode array will produce evenly spaced phosphenes in a square with the size of phosphene and distance between the phosphenes being equal. Cortical stimulation studies have shown that the phosphene size and location will depend upon the location of stimulation on the visual cortex (Brindley and Lewin 1968, Brindley 1970, 1982, Dobelle and Mladejovsky 1974, Dobelle 1974a, 1974b, Schmidt et al 1996, Lee et al 2000, Kaido et al 2004). The phosphenes are smaller in size toward the foveal region of cortex and increase in size following the ‘cortical magnification’ as one move away from the foveal region (Brindley and Lewin 1968). This also agrees with the visual field map representation on the cortex (Horton and Hoyt 1991). After considering surgical difficulties, cortical prosthesis researchers are planning to use intracortical electrode arrays on the dorso-lateral surface of the occipital pole within a region of about 3 cm radius, geometrically centered on the lobe generating phosphenes over approximately 25° of eccentricity in the lower quadrant of visual space. The tests conducted were designed to test performance for this implantation. The aim was to judge performance with a realistic phosphene map. A description of surgical difficulties and expected phosphene maps has been published earlier (Srivastava et al 2007). A similar approach to generate phosphene has been suggested by Buffoni et al (2005).
The visual cortex of individuals might vary up to 40% (Stensaas et al 1974, Flores 2002); hence, the area for electrode implantation will vary and in few patients the area available for the electrodes might be less than average. Also, few electrodes might fail during surgical procedure itself and few more during a long-term implantation. To study the effect of fewer phosphenes, dropout effects were included in the studies. Before conducting the studies it was difficult to estimate the limit on dropouts beyond which we might not get any positive results. A 50% dropout was decided based on the condition that the lowest surgical area available on a subject might be 50% of an average subject or if few electrodes fail, 50% failure might be the worst condition. Cha et al (1992a, 1992b, 1992c) had given an array size of 625 phosphenes for reading and mobility performance for a cortical prosthesis device; hence, at an array size of 325 phosphenes (50% dropout), a good performance or even an improvement with practice was not expected for all the tasks.
During cortical stimulation, it has been found that the phosphenes move in space when the eye moves (Brindley and Lewin 1968); hence, gaze stabilization was simulated by locking the image on the retina of the subject under test. The hypothesis was that the subjects will respond by trying to keep the gaze steady and scan by just camera, i.e. head movements; similarly, it is expected that a prosthetic device wearer will respond by suppressing voluntary eye movement and reducing the vestibule-ocular reflex as mentioned by Dagnelie et al (2006). In future, a visual prosthesis can be designed in which a continuous eye tracking is done and the image is adjusted accordingly to achieve free viewing conditions.
In this paper, results of detection, eye–hand coordination and virtual mobility psychophysical tests for the estimated phosphene map for a cortical prosthesis device are presented.
2. Methods
Five test subjects, three men (named L1, N3, N4 for reference in this document) and two women (N1 and N2) aged 26–52 years, participated in the counting and placing experiments and five test subjects, four men (N5, L1, N3, N4) and one woman (N1) aged 26–52 years, participated in the virtual mobility experiments. Four subjects (L1, N2, N3, N4) participated in all the experiments, including a low vision subject (L1). One female subject (N1) was replaced by a male subject (N5) for the virtual mobility experiment due to schedule conflicts. The subjects came in for 1 h test visits during 3–4 months. One-hour sessions were chosen to minimize the effect of fatigue and errors which might result due to it. All subjects had previously participated in prosthetic vision simulation studies but had never done any experiments with the phosphene map used for our cortical prosthesis studies. The phosphene map used for testing for these experiments had phosphenes in lower 25° of one hemisphere. The phosphenes were randomly placed with increasing size with the increase in eccentricity with few of the phosphenes overlapping. This map is shown in the left image of figure 2. Four subjects in each study were normally sighted and one subject was severely visually impaired (best corrected visual acuity of 20/310) with moderate nystagmus in the tested eye. All the subjects read and signed a consent form approved by the IRB of the Johns Hopkins University School of Medicine. Subjects were informed as to the purpose of the experiments and all testing adhered to the tenets of the Declaration of Helsinki.
2.1. Experiment 1
2.1.1. Purpose
The purpose of this experiment was to determine the subject’s ability to perform detection by counting white fields on a checkerboard and eye–hand coordination by placing black checkers on the white fields of the checkerboard. The accuracy of counting and placing along with the time taken to complete the task were used to judge performance. Results were checked for the changes with the increase in the complexity of the task and learning effect with the increase in practice of the experimental task. The effect of dropout was also tested on the performance under 0% dropout (650 dots), 25% dropout (488 dots) and 50% dropout (325 dots).
2.1.2. Equipment
The studies were conducted at the Low Vision Lab at LionsVision Research and Rehabilitation center, Johns Hopkins University. The headset used was a modified Low Vision Enhancement System, head mounted display with a 48 × 36° field of view and a monochrome VGA resolution (640 × 480 pixels) display. The right display of the headset was switched off and the test subjects were allowed to use just the left eye. Further details of this headset can be read in Dagnelie et al (2006). The system used to generate the cortical prosthesis simulation has been developed at the Laboratory of Neural Prosthesis Research at Illinois Institute of Technology, Chicago. A CMOS digital camera (Omnivision’s C3088) was used to capture the images which are processed on a Xilinx Virtex-II XC2VP30 FPGA (Field Programmable Gate Array)-based visual prosthesis proto type device on a Digilent Virtex-II Pro Development test board. The FPGA implements the image-processing filter. The filter incorporates the expected phosphene map and calculates the average threshold pixel value of the pixels in the confidence interval of obtaining the spatial percept of a corresponding electrode. This calculated value is compared to four different preset threshold values to decide the brightness level of the dotted phosphene. The system then generated the appropriate level of brightness out of the four different levels, generating images with four different grayscale levels. The FPGA generated the expected cortical perceptual response as a VGA signal, which was converted to NTSC using a commercial VGA to NTSC converter (Grand Tec Ultimate PC to TV Converter). The NTSC signal was sent to a desktop PC where feedback from the eye tracking was incorporated and the image was adjusted for display on the headset. Since the vertical resolution of the headset display is only 36°, the display will show only partial map of 25° eccentricity if the center of visual field of the goggle and the image are matched. To completely fit the map on the display, the center of the phosphene map was shifted up vertically by 7° and the change was incorporated in the hardware filter.
The video headset also had a built-in IR illumination and a CCD camera imaging the subject’s pupil. Eye tracking software (ViewPoint, Arrington Research, Scottsdale, AZ) running on a separate desktop PC allowed recording the eye movements and calculating the pupil center location. The software presents the X and Y coordinates of the subject’s pupil center to the image adjustment computer via an IEEE 488 connection at a rate of 30 s−1. At the start of each test session, gaze angle calibration data were obtained which were combined with the X and Y coordinates to maintain the position of the image registered with the subject’s gaze angle.
The camera was mounted on a bracket in the center of the front cover of the headset for the counting and placement task for which head scanning by the subjects was required. There was an offset of about 3.2 cm between the axis of the camera view and the subject’s eye position. The optical axis of the camera was downward, perpendicular to the subject’s line of sight. For the counting and the placement tasks, modified checkerboards of 8 × 8 fields with a field size of 2 × 2 cm were created. Twenty-five boards were created with 1–16 white fields, with all the remaining fields black. Few boards had the same number of white fields but different layouts of the fields. Each board was framed by a 2 cm white border. White fields were distributed in random positions: the white fields were never contiguous horizontally or vertically but could assume positions along an edge, diagonally or in a corner. Boards were presented to the subject in all four possible orientations, hence creating 100 (25 × 4) possible combinations, eliminating the possibility that subjects would memorize the layouts of boards with the increase in practice. Game pieces used in the placement task were round black checkers with a diameter of 2.2 cm. This allowed 90% coverage of a white field with a centrally placed black checker. The light source to illuminate the board was from above to simulate real conditions as prosthetic wearer might face. During the placement task when subjects brought their hands over the board they could see the reflection of their hand and fingers and the shadow of the hand, if in line with the optical axis of the camera. Our assumption was that this might confuse the subjects, increasing the complexity of the task or that they would learn to recognize the reflection of their fingers and might even use it as a proprioceptual feedback helping them in their task. During our testing, we observed that initially all subjects were confused by the refection of their own fingers which increased the complexity as expected but with practice they learned to avoid it and at least one subject learnt to use it as a feedback to locate the white fields and place the checkers on the board. Figure 1 shows the experimental setup with the headset, the camera mounted on the headset and the checkerboard kept at a reading distance. Figure 2 shows an image of the expected phosphene map in the visual space on the left. The image in the center is a sample board. A phosphene map has been superimposed on the board image in the center image to show the degree of cover of the board from a reading distance. The image on the right shows the dotted image of the area of the board covered by the map as observed in the headset display. To create the 25% and 50% dropout conditions, the phosphenes were dropped randomly from the 650 dot map.
2.1.3. Experimental procedure
Subjects were seated in a chair with the boards presented in front of them parallel to the floor on a flat horizontal support. The experiments were conducted under the gaze-locked condition in which the eye position was tracked and the images were adjusted on the display as per the eye movement. The board was at a reading distance from the camera mounted on the headset and the subjects had the freedom to increase or decrease the distance between the board and the camera by bringing their heads up or down, in effect producing a scenario in which the subject could change the degree of area of the checkerboard as seen through the dotted filter. If the phosphene map covered the whole visual field, the board would have looked similar to as seen normally without a headset. By moving close to the board, the subjects could see the white fields in detail, e.g. during the placing task to position the checkers accurately; by moving away with a broader view of the board, they could quickly scan the whole board, which could help them in the counting task. The subjects were also able to inspect the board by scanning in parallel with the board or at an angle by tilting their heads. The decision by the subjects for scanning the boards was intuitive for each individual. They were instructed before the testing as how they can scan but never guided during the testing sessions. Spatial and temporal integration was tried to be achieved, by giving the subject the ability to scan the board. The subjects viewed and manipulated the boards and checkers under the test conditions, wearing the headset. The subjects felt the checkers and the surface of the boards with their hands but never saw them in the normal view. They were told to complete the task as accurately and as quickly as possible. While placing the checkers, they were advised not to loose the checkers or drop it on the board since the background of the board was black and once dropped it would be difficult to retrieve the checker from the board under the test conditions. The subjects came for 1 h sessions for the tests. Before starting the tests each subject was given 1 h of practice for the counting and the placing task under the free viewing condition (eye tracking switched off), to become familiar with the task and with the dotted images. Before starting the experiments, the subjects were given a general description of the board. They were given details about the counting and the placing tasks. For the counting task, they were told that the accuracy and time were the measured outcomes. For the placing task, they were advised to cover the whole white field. They were informed that the loss or misplacement of a checker counted as an error and that the time to complete the task, the extent of coverage of the white fields by the checkers and the errors will be the measured outcome. A warning was given to them that the reflection of the hand and the shadow of their hand and fingers might confuse them during the placement task. The test hour was used for conducting both the counting and the placing test experiments, with counting and placing tasks randomized. Several counting tasks were followed by several placing tasks and vice versa where the number of times each task was performed before the other was randomized and different each time. During the experiment, the boards from 1 to 16 fields were presented to the subject in random order with any of the four sides of the board presented toward the subject. Hence, both the board and the side of the board to be presented toward the subject were randomized. A board which was once presented either for a counting task or for a placing task was not presented again during the same testing session, hence eliminating the possibility that the subject might remember the pattern of the white fields from the placing task to the counting task or vice versa. The first session of testing for all the subjects was with a 0% dropout condition. After the first visit, the dropout level selected for each visit for all the subjects was randomizded.
2.1.4. Results for counting task
Counting time and the number of fields reported by the subject were recorded for each trial. To account for the errors, the count time for each trial was modified to give an adjusted time (Tadjust) for the number of white fields missed or over counted, according to the formula
where Traw is the total time recorded for the counting task for a trial, Ncount is the number of white fields reported by the subject, Nerr is the absolute difference between the number of white fields reported by the subject and the real number of white fields on the board, where N is the number of real number of white fields on the board. The average adjusted time was calculated for one board for each visit as Tmeanb and the average adjusted time for one white field for each visit as Tmeanf:
where Ntrials is the number of the boards tested, i.e. number of trials, for the counting task for that visit. To asses the change from session to session, Tmeanb and Tmeanf plots for 0%, 25% and 50% dropouts shown in figure 3 were plotted. It was observed that the maximum percentage of drop (the maximum learning effect) between two consecutive visits for the adjusted time is seen from the first to second visit. This could have affected the statistical analysis; hence, the reading of the first visit was dropped for every subject from any further calculations. The regression line was plotted through the scatter plot of Tadjust as a function of N, for 0%, 25% and 50% dropouts, to observe the effect of the increase in the number of white fields. To test whether an increase in the number of white fields increases the complexity of the task or if it remains a simple task of scanning the board in a limited time, the slope was determined, i.e. the time to count an additional number of white field, and the Y intercept, i.e. the time required to scan a blank board for a white field before any white field is counted, was noted. The plots are shown in figure 4. For each subject, the adjusted time plotted as a function of the white fields averaged across multiple trials. Each plot shows the regression line (dashed) and a solid line connecting the center of mass of all data points and the origin. The solid line represents the expected regression if the detection time for each field was equal for all the boards, i.e. with the adjusted time increasing proportionally with the number of white fields. The change in the performance with increasing number of white fields for different dropouts varies substantially between subjects and also shows the difference between the different dropout levels of the same subject. A flat dotted regression line would be interpreted as the subject counting the white fields while scanning the board and the time taken as the time required for scanning the board with the increase in the number of white fields not having a significant effect on the performance. A dotted line falling close to the mean solid line shows the task becoming more time intensive with an increasing number of white fields. Most of the plots show the intermediate dependence of the counting time on the number of white fields with N1 showing a flat regression line for 0%, N3 showing a flat regression line for 25% and N4 showing almost flat regression lines for 25% and subject 1 showing the regression line close to the mean slope line for 25% dropout and L1 showing the same for 0% dropout. These plots do not reflect the effect of the dropout sequence in consecutive sessions. Since the dropouts were randomized with different dropout levels for different visits for each subject, the sequence of the dropout combined with the learning effect might have affected the slopes and the intercepts of the plots; hence, we see a large variability between and within the subjects for the different dropout plots. To see the effect of each factor and the interaction of the factors Subject, Visit (or practice), Dropout and Board (number of white fields), we performed an analysis of variance (PROC GLM with backward regression analysis) in SAS on the counting task data for all the subjects together (for adjusted time value and errors recorded) and also on data for the individual subjects. All possible interaction terms were considered other than those with ‘Subject’. The subjects under test show a variation of more than 50% for the adjusted time in the plots for tadjust against N, between and within the subjects and for the different dropout levels, also changes within the subject was of more interest hence for statistical analysis interaction of the Subject with other factorial terms of Visit, Dropouts and Board was not considered.
Plot of the mean count time in seconds per board (left) and plot of the mean time in seconds per field (right) as a function of Visit for all the subjects. The different data series in each plot shows different dropout levels. An improvement of task timing with increased practice for all the three different dropout levels in all the subjects is observed.
Average adjusted count time as a function of the number of white fields for 0%, 25% and 50% dropout for the five subjects. The dotted line is the regression line and the solid line shows the expected regression if the mean time to count the fields of each board was the same.
For the adjusted time value on the overall data, the significant factors were Subject (F = 25.29, p < 0.0001), Visit (F = 23.58, p < 0.001), Dropout (F = 8.41, p = 0.0040) and Board (F = 50.75, p < 0.001) (the value of R-square was 0.444 086). The analysis shows an increase in Tadjust time with an increase in the dropout and an increase in the number of the fields and shows a decline with more practice (Visit). The dropout has a weak significance compared to the other factors. The error analysis shows a decrease in the error with an increase in the practice (number of visits) (F = 14.83, p = 0.0001) with a weak R-square value of 0.041 673 (errors for the counting task were rare and sporadic). Analysis of Tadjust (PROC GLM with backward regression using SAS) with all possible interactions of Visit, Dropout and Board for individual subjects is shown in table 1. Factors and interactions which are not in the table were not statistically significant.
Table 1
Statistical analysis for the counting task.
| Significant parameters | |||||||
|---|---|---|---|---|---|---|---|
| Subject | R-square | Observed values | Visit | Dropout | Visit * Dropout | Board | Visit * Board |
| F | 15.25 | 23.28 | 26.64 | 15.68 | 8.93 | ||
| N1 | 0.655 658 | Pr > F | 0.0005 | <0.0001 | <0.0001 | 0.0005 | 0.0058 |
| Estimate | −80.809 | −13.216 | 3.117 | 35.982 | −6.0513 | ||
| F | 10.96 | 14.01 | |||||
| L1 | 0.492 138 | Pr > F | 0.0024 | 0.0007 | |||
| Estimate | 2.601 | 13.273 | |||||
| F | 83.76 | 15.07 | |||||
| N2 | 0.636 191 | Pr > F | <0.0001 | 0.0003 | |||
| Estimate | −25.346 | 4.734 | |||||
| F | 13.95 | 9.24 | |||||
| N3 | 0.154 715 | Pr > F | 0.0003 | 0.003 | |||
| Effect | −3.339 | 2.042 | |||||
| F | 31.02 | 19.4 | |||||
| N4 | 0.322 744 | Pr > F | <0.0001 | <0.0001 | |||
| Estimate | −4.794 | 2.67 | |||||
N1 shows a decrease in time with an increase in dropouts, which is contrary to what is expected. In figure 3, it can be observed that the maximum dropout sessions were conducted in the last two visits for subject N1; also the visit and the dropout interaction shows a high significance. Hence, the effect of practice (Visit) might have dominated over the effect of dropout. Subjects N2, N3 and N4 show no significance for dropouts, showing that a dropout of 50% also did not affect the performance and the improvement in time is due to practice (Visit). For the counting task, the normally sighted subjects needed to differentiate the contrast level of white fields from the black background. An increase in the dropouts might have affected the resolution of the white fields with an increase in the difficulty to see the corners and the edges of the white field squares but even up to a dropout of 50% would not have significantly affected the ability of the subjects to differentiate between the contrast level of the white fields and the black background, required for counting. Subject L1 is a low vision subject and shows no significant improvement in counting by practice and shows an increase in time with the dropouts. This might be possible for a low vision subject who cannot trust his own vision and always takes a certain minimum time for completely scanning the board for confidently counting the white fields with accuracy. In figure 3, an improvement in the counting time can be observed for consecutive visits for the same dropout level but this change is not statistically significant. Subject L1 also shows an increase in task time with an increase in dropout showing that the subject takes more time to confirm the presence of a white field confidently with higher dropouts. The low vision subject saw the images on the headset’s display. Even with the maximum resolution of dotted images, the image perceived by the low vision subject would be poor as compared to normally sighted persons; hence, an increase in the dropout would have affected the visual perception of the low vision subject the most. All the subjects show an increase in time for an increase in the number of white fields (board) which we also observed in the Tadjust against N plots.
2.1.5. Results for placing task
Placing time and the number of errors were recorded for each placing trial. For error counting, a scoring method was developed in which the checkers covering more than 50% of the field earned 1 point, and the checkers with coverage less than 50%, but covering some part of the white field, earned 0.5 points. The addition of points for all board fields is the number of fields covered (Nplace) for the board under test. The numbers of checkers misplaced (not touching any of the white fields) were counted as errors (Nmisplace). For accounting the errors, the place time for each trial was adjusted for the placing errors, according to the formula
where Traw is the total time recorded for the placing task of a trial, and N is number of real number of white fields on the board. The average adjusted time for each white field was calculated as Tmeanf:
where Ntrials is the number of boards tested for the counting task for that visit. To assess the change from session to session, Tmeanf against the session number was plotted for 0%, 25% and 50% dropouts. The plots are shown in figure 5. Similar to the counting task, the maximum percentage of drop between two consecutive visits for the adjusted time is seen from the first to second visit; hence, the readings of the first visit for every subject were dropped for any further calculations. The regression line was plotted through the scatter plot of Tadjust as a function of N, for 0%, 25% and 50% dropouts similar to the counting task. The plot conventions used are similar to the plots for the counting task. The plots shown in figure 6 illustrate that the placing task becomes more time intensive with increasing number of white fields. The plots show some variation for the dropouts and show up to 50% variation between the subjects for the adjusted time. The nature of the plots observed is different from the nature of the counting plots. The slopes of the regression lines for the placing time, i.e. the mean time per field, are higher than the slopes for the counting time. For the same visit and dropout level, the task of placing becomes more time consuming as the number of white fields increases as compared to the counting task, showing that the placing task becomes more time intensive. This can be easily seen in the plots of N3 and N4. To further investigate the effects of different factors and interactions, analysis of variance (PROC GLM with backward regression analysis) was performed in SAS on the placing data of all the subjects together for the adjusted time and the calculated errors and also on the individual data sets of the subjects, since the subjects show a variation of more than 50% for the adjusted time in the plots for Tadjust against N. Subject, Visit (practice), Dropout and Board were considered as factors affecting the result with all the possible interaction terms other than those with ‘Subject’.
Plot of the mean time per field as a function of Visit. The different data series in each plot show different dropout levels. The placing time to place a checker on a board reduces with increase in practice.
Plot of the mean time per field as a function of Visit. The different data series in each plot show different dropout levels. Placing time to place a checker on a board reduces with increase in practice even with a higher dropout rate.
For the adjusted time value on the overall data, the significant factors for an R-square value of 0.481 336 were Subject (F = 14.02, p < 0.0001), Visit (F = 30.12, p < 0.001), Dropout (F = 9.03, p = 0.0040) and Board (F = 109.42, p < 0.001). The analysis shows an increase in Tadjust with dropout and number of fields and decline with more practice (Visit). The dropout has a weak significance compared to the other factors. The error analysis shows a decrease in errors with an increase in practice (Visit) (F = 14.83, p = 0.0001), and an increase with an increase in dropout (F = 13.20, p = 0.003) and white fields (Board) (F = 5.87, p = 0.0160), and an interaction of Visit and Dropout (F = 8.31, p = 0.0042) with an effect of decrease in the task time. The R-square value for the error analysis is 0.427 951. Analysis of Tadjust with the factors Visit, Dropout and Board (number of white fields) and all their possible interactions for each individual subject is shown in table 2. Factors and interaction values which are not in the table were not statistically significant.
Table 2
Statistical analysis of placing task.
| Significant parameters | ||||||
|---|---|---|---|---|---|---|
| Subject | R-square | Observed values | Visit | Dropout | Visit * Dropout | Board |
| F | 23.82 | 8.4 | 11.24 | 20.3 | ||
| N1 | 0.678 305 | Pr > F | <0.0001 | 0.0075 | 0.0025 | 0.0001 |
| Estimate | −202.828 | −16.297 | 4.108 | 23.365 | ||
| F | 4.41 | 5.23 | 5.68 | |||
| L1 | 0.285 218 | Pr > F | 0.0457 | 0.0306 | 0.0248 | |
| Estimate | −47.457 | 4.077 | 13.827 | |||
| F | 17.79 | 11.97 | ||||
| N2 | 0.360 131 | Pr > F | 0.0001 | 0.0012 | ||
| Estimate | −33.191 | 11.509 | ||||
| F | 30.26 | 5.21 | 56.41 | |||
| N3 | 0.4643 | Pr > F | <0.0001 | 0.0246 | <0.0001 | |
| Effect | −8.595 | 0.597 | 7.866 | |||
| F | 30.9 | 4.51 | 76.86 | |||
| N4 | 0.524 347 | Pr > F | <0.0001 | 0.036 | <0.0001 | |
| Estimate | −9.859 | 0.597 | 9.919 | |||
All the subjects show a significant reduction in the task time with increasing practice. N1 shows a decrease in the time with an increase in the dropouts, just as was observed for the counting task. Subjects L1, N3 and N4 show an increase in the time with an increase in the dropout level but the significance levels are modest. In the plot of Tmeanf against N for subject N2, it can be observed that the dropouts have some effect but this was not statistically significant. This shows that the practice effect dominates over the dropout effect and a decline in the performance levels due to dropouts can be overcome by practice. The effect of ‘Board’ (increase in white fields) is an increase in the task time with an increase in the number of white fields, as expected from figure 6. Subjects L1 and N2 show low values for R-square, showing that other factors such as test–retest variability for boards with the same number of fields and interactions which are statistically not significant also affect the results.
2.2. Experiment 2
2.2.1. Purpose
The purpose of this experiment was to determine the subject’s ability to recognize a pathway and orient oneself to follow it without memorizing or recall. The time taken to complete the task along with the way-finding errors was used to judge the performance. The learning effect on performance was checked with the increase in practice of the experimental task. The effect of dropout on mobility performance was also tested for 0% dropout (650 dots), 25% dropout (488 dots) and 50% dropout (325 dots).
2.2.2. Equipment
The headset and the image processing hardware were the same as for the placing and the counting tasks. For the mobility test ten 11-room mazes were created in World craft free ware which could be navigated in the Half Life gaming engine using a game controller. All the mazes had different floor plan. The mazes had white walls with black alphabetical labels on one of the walls of each room in ascending order to help the subjects recognize the room. Room 1 had label ‘A’, room 2 had label ‘B’ and so forth, and the 11th room had label ‘END’ for the subjects recognize that they have reached the final room. The label of each room was on different walls for each room. The label location was different for each room in different mazes. The roof and the floor of the virtual maze were black and the sides of the doorways and the corners had a different grayscale level to help the subjects recognize the doors and the corners. The virtual maze had one to two obstacles in each room with a black and white checkered pattern on it, which made the task more difficult for the subjects. The CMOS camera was mounted on a stand in front of a LCD monitor for the virtual maze experiments. The subjects moved in the virtual maze using a game controller and no head scanning was required. The camera captured the virtual maze displayed on the LCD screen and sent the real-time video to the headset through the processing hardware. The changes in the maze by the controller were reflected on the LCD display and captured by the camera. The zoom of the camera was adjusted in a way as to fit the complete maze on the designed image processing filter with 25° of field view, as shown in figure 7. The picture on the left shows the image of a virtual maze. The center image shows the phosphene map superimposed on the image of a virtual maze. The image in the center shows that the complete virtual maze was made to fit the phosphene field by adjusting the zoom of the camera. The image on the right shows the dotted image of the virtual maze as seen in the headset display through the hardware filter implementing the phosphene map. In a real maze situation where such a system has been implanted on a blind subject, the image will be adjusted by the subject himself by adjusting the zoom of the camera or moving toward or away from the object under visual observation and by using a head scanning instead of the game controller. The processed dotted images were displayed on the headset with the eye tracking switched on. For 25% and 50% dropout conditions, the dropped phosphenes were random.
2.2.3. Experimental procedure
Subjects were seated in a chair wearing the headset and could see the virtual maze on the headset display through the dotted filter with the eye tracking switched on. The images were also adjusted according to the eye movements (gaze-locked condition). The subjects could move through the maze with forward and backward movements, left and right movements and up and down movements using the game controller. The subjects came for 1 h sessions for the virtual mobility tests. Before starting the tests, each subject was given 1 h practice with the mobility task under the gaze-locked condition (eye tracking on and the images adjusted according to eye movements) on practice mazes which were similar to the test mazes but instead of alphabets had numerical labels on the walls from 1 to 10 and did not have any obstacles in the rooms. All the test trials were done under the gaze-locked condition. Before starting the experiments, the subjects were given general description of the mazes. The subjects were instructed to go from room ‘A’ (room 1) to room ‘J’ (room 10) and then to the final room with label ‘END’ (room 11), in order. Before the test began, they were instructed to read the label of the current room, locate the door to another room and move from the current room to the other and keep moving till the time they reach the room with label ‘END’. They were instructed that there were obstacles in the room which might confuse them and advised them to always be aware of their orientation in the maze; else they might move in backward order rather than forward. A way-finding error was counted if the subjects reentered a previous room and did not realize it. If they realized their mistake in few seconds without reading the room label, it was not counted as a way-finding error since the subjects were penalized with an increase in the task completion time. The subjects were informed that the way-finding errors and the total time to travel a maze were the measured outcome. During the test hour, the ten virtual mazes were presented to the subjects in a random order. The first session of testing for all the subjects was with a 0% dropout condition, after which the dropout conditions for every visit for all the subjects were randomized.
2.2.4. Result
The time to complete each maze and the number of way-finding errors were recorded. The mean time (Tmean) to complete one maze and average errors (Eavg) per maze were calculated for each session. Plots of Tmean and Eavg against visits were plotted for all the subjects for 0%, 25% and 50% dropout conditions. The Tmean and Eavg against Visit plots are shown in figure 8. The error bars represent the standard deviation. The plots show an improvement in Tmean with practice and show some deterioration in performance with higher dropouts. Tmean varies more than 50% between the subjects. We performed statistical analysis (PROC GLM with backward regression) using SAS for Subject, Dropout and Visit factors and all possible interactions excluding interactions with Subject to find their effect on average time Tmean and average error Eavg. The factors affecting Tmean were Subject (F = 34.21, p < 0.0001), Visit (F = 90.04, p < 0.0001) and Dropout (F = 6.06, p = 0.0189), with R-square = 0.868 954. The factors showing a significant effect on Eavg were Subject (F = 12.11, p < 0.0001) and Visit (F = 17.95, p = 0.0002) with R-square = 0.629 828. For Tmean analysis, the Dropout has a low significance similar to the counting and the placing experiments and a separate analysis checking the effects within individual’s yielded practice (Visit) as the only significant effect, as shown in table 3. Other factors and interactions had no statistical significance. Hence, for mobility even with the low number of dots (325 for 50% dropout), the subjects could easily complete the task with an improvement with practice. This result is in agreement with the results of the mobility task for a retinal prosthesis device simulation study by Dagnelie et al (2006), where 60 dots of phosphenes were enough to complete the mobility task.
Plot of the mean time to complete a maze (left) and the number of errors (right) as a function of Visit. The different data series in each plot show different dropout levels.
Table 3
Statistical analysis for the mobility task.
| Subject | R-square | Observed values | Significant parameters Visit |
|---|---|---|---|
| F | 26.96 | ||
| N5 | 0.793 896 | Pr > F | 0.0013 |
| Estimate | −49.754 | ||
| F | 7.46 | ||
| L1 | 0.515 997 | Pr > F | 0.0293 |
| Estimate | −54.714 | ||
| F | 12.43 | ||
| N2 | 0.756 47 | Pr > F | 0.0243 |
| Estimate | −93.386 | ||
| F | 75.9 | ||
| N3 | 0.915 564 | Pr > F | <0.0001 |
| Estimate | −57.333 | ||
| F | 17 | ||
| N4 | 0.708 393 | Pr > F | 0.0044 |
| Estimate | −39.91 |
3. Discussion
A cortical prosthesis holds the promise to restore limited vision to blind individuals even with completely damaged retina or optic nerve by directly targeting the visual cortex for generating the phosphenes. The simulation experiments presented were designed to estimate the efficacy of a cortical prosthesis device by testing visual inspection (counting), eye–hand coordination (placing of checkers) and way-finding (mobility) ability under simulated conditions. The experiments were designed and conducted under realistic conditions expected from a cortical prosthesis device. Historically, researchers have assumed either a hexagonal or a square pattern of dots, which does not contain the peculiarities of the cortical response. The dotted image generated was designed considering the biological response of the visual cortex with cortical magnification and the surgical placing of the electrodes, where few phosphenes were fused and few were overlapping. The conditions of electrode failure and placement of different number of electrodes for different individuals according to the surgical area available were tested by including the dropout conditions. Eye tracking was used to generate the condition of phosphenes moving in space with eye movements. By using head scanning for the counting and placing task and scanning by a game controller for mobility, we simulated a condition in which a person will be required to create a continuous mental image of the scene from a limited field of view.
These results demonstrate that even with a limited number of phosphenes in one visual hemi-field with limited eccentricity in the visual space, it is possible to attain a level of proficiency in which prosthesis wearers can perform simple tasks with practice. All subjects learned and adapted to the dotted images in the tests. Individual subjects have shown variability in the results, which can be expected if a cortical device in the blind individuals is implanted. All subjects have shown improvement in the task time and error reduction with practice except the low vision subject for whom the improvement was not statistically significant for the counting task. The longer time required by the subject can be attributed to nystagmus (Dagnelie et al 2006). The low vision subject did show improvement in task performance with practice for the placing and mobility task. Counting is a simple task; there is less to be learned and the subject was already experienced in doing such a task with a different phosphene map. The low significance of dropouts and high significance of practice show that the effect of dropouts to a large extent can be negated with an increase in the practice for 325 dots of phosphenes. This result can be significant for the individuals with low areas of visual cortex available for electrode implantation demonstrating that the persons with a low exposed visual cortex area could also be subjects for a prosthesis implantation. This result also helps the researchers who worry about failure of electrodes during surgical implantation and during the lifetime of a prosthesis device and the effect this may have on the performance. If the dropout rate of phosphenes is increased, then we will reach a limit beyond which we will not see any positive results. A future experiment can be designed to find this lower limit. For the counting and the placing tasks, the subjects were allowed to observe the checkerboards with normal settings of the camera, but for the virtual mobility experiment, the zoom of camera (acceptance angle of the camera) had to be adjusted, to fit the whole maze on the phosphene map. This was done since during the training, few subjects reported that they could perform the task better with zoom adjusted so that the phosphene map covers the whole maze. This shows that depending upon various scenes, the acceptance angle of the camera should be adjusted by the subject to get an optimum view. This can also be achieved by either mapping a part of the captured image or presenting the whole image on the phosphene map. The results from our experiments also provide clues to rehabilitation instructors helping the prosthesis wearers as to how to train different individuals after the implantation of a cortical prosthesis.
The phosphene map used for experiments had a maximum of 650 dots of phosphenes in 25° eccentricity in lower half of one hemifield corresponding to the resolution expected from an intracortical implantation. If we try to generate a large number of phosphenes in a limited visual area by implanting a large number of microelectrodes, the result might not be a huge gain in performance since many of the phosphenes might fuse. A better performance might be achieved if surgical techniques are developed to obtain phosphenes in higher degrees of eccentricities covering more visual space.
These experiments were designed to test the efficacy of the prosthesis device based on dotted phosphene images. Phosphenes are simple spatial percepts, seemingly similar to pixels, but the biological visual system does not work on the principle of pixels. The neural network of the visual cortex derives the spatial and temporal information and does motion selectivity, direction selectivity, orientation selectivity and ocular dominance to generate the overall visual perception. How can these characteristics be induced using a visual prosthesis device is not known. As understanding of visual neuroscience increases and methods are developed to manipulate the biological features of vision, using artificial stimulation, visual prosthesis devices will evolve which will mimic the processing of the natural visual pathways.
4. Conclusion
The simple tasks of detection, eye–hand coordination and mobility were learned by the subjects under conditions of low resolution, low visual eccentricity in one half of the visual field and gaze-locked condition. The crude and limited information provided by the cortical visual prosthesis device under development, if successful, will help blind subjects get some assistance for conducting simple tasks in daily life. The intracortical visual prosthesis device stimulates the cortex electrically to generate phosphenes. The ability of the human visual cortex to assimilate multiple dots of light into a meaningful visual perception has not been experimentally demonstrated.
The remaining question is whether the visual cortex will indeed be able to adapt to phosphene-like images. If the visual cortex is found to be plastic enough to adapt to the scanned phosphene perceptions, then individuals with blindness might have a means to have limited vision in the near future. Current research has advanced to a point where it is feasible to implant a prototype intracortical visual prosthesis and test it on human volunteers. Results from these cortical simulation studies are encouraging for experiments in which electrodes are implanted in a blind human volunteer and images are captured and processed in real time and the corresponding electrodes stimulated to generate dotted images (phosphenes) to judge the performance in real stimulation conditions.
References
- Bak M, Girvin JP, Hambrecht FT, Kufta CV, Loeb GE, Schmidt EM. Visual sensations produced by intracortical microstimulation of the human occipital cortex. Med. Biol. Eng. Comput. 1990;28:257–259. [PubMed] [Google Scholar]
- Boyle J, Maeder A, Boles W. Scene specific imaging for bionic vision implants. Proc. of 3rd International Symposium in Image and Signal Processing and Analysis. 2003;3:423–427. [Google Scholar]
- Bradley DC, et al. Visuotopic mapping through a multichannel stimulating implant in primate V1. J.Neurophysiol. 2005;93:1659–1670. [PubMed] [Google Scholar]
- Brindley GS. Sensations produced by electrical stimulation of the occipital poles of cerebral hemispheres and their use in constructing visual prosthesis. Ann. R. Coll. Surg. 1970;47:106–108. [PMC free article] [PubMed] [Google Scholar]
- Brindley GS. Effects of electrical stimulation of the visual cortex. Hum. Neurobiol. 1982;1:281–283. [PubMed] [Google Scholar]
- Brindley GS, Lewin WS. The sensations produced by electrical stimulation of the visual cortex. J. Physiol.(London) 1968;196:479–493. [PMC free article] [PubMed] [Google Scholar]
- Buffoni L-X, Coulombe J, Sawan M. Image processing strategies dedicated to visual cortical stimulators: a survey. Artif. Organs. 2005;29:658–664. [PubMed] [Google Scholar]
- Cha K, Horch KW, Normann RA. Simulation of a phosphene based visual field: visual acuity in a pixelized vision system. Ann. Biomed. Eng. 1992a;20:439–449. [PubMed] [Google Scholar]
- Cha K, Horch KW, Normann RA. Mobility performance with a pixelized vision system. Vis. Res. 1992b;32:1667–1672. [PubMed] [Google Scholar]
- Cha K, Horch KW, Normann RA, Boman DK. Reading speed with a pixelized vision system. J. Opt. Soc. Am. 1992c;9:673–677. [PubMed] [Google Scholar]
- Chen SC, Hallum LE, Lovell NH, Suaning J. Visual acuity measurement of prosthetic vision: a virtual-reality simulation study. J. Neural. Eng. 2005;2:S135–S145. [PubMed] [Google Scholar]
- Chen SC, Lovell NH, Suaning GJ. Effect on prosthetic vision visual acuity by filtering schemes filter cut-off frequency and phosphene matrix: a virtual reality simulation. Conf. Proc. of the 26th Annual International Conf. 2004;6:4201–4204. IEEE EMBC. [PubMed] [Google Scholar]
- Dagnelie G, Keane P, Narla V, Yang L, Weiland J, Humayun M. Real and virtual mobility performance in simulated prosthetic vision. J. Neural. Eng. 2007;4:S92–S101. [PubMed] [Google Scholar]
- Dagnelie G, Walter M, Yang L. Playing checkers: detection and eye hand coordination in simulated prosthetic vision. J. Mod. Opt. 2006;53:1325–1342. [Google Scholar]
- Dobelle WH. Artificial vision for the blind by connecting a television camera to the visual cortex. ASAIO. J. 2000;46:3–9. [PubMed] [Google Scholar]
- Dobelle WH, Mladejovsky MG. Phosphenes produced by electrical stimulation of human occipital cortex and their application to the development of a prosthesis for the blind. J. Physiol.(London) 1974;243:553–576. [PMC free article] [PubMed] [Google Scholar]
- Dobelle WH, Mladejovsky MG, Girvin JP. Artificial vision for the blind: electrical stimulation of visual cortex offers hope for a functional prosthesis. Science. 1974;183:440–444. [PubMed] [Google Scholar]
- Dowling J, Maeder A, Boles W. Mobility enhancement and assessment for a visual prostheses. Proc. of Physiology Function and Structure from Medical Images—Medical Imaging. 2004;5369:780–791. [Google Scholar]
- Flores LP. Links occipital lobe morphological anatomy: anatomical and surgical aspects. Arq. Neuropsiquiatr. 2002;60:566–571. [PubMed] [Google Scholar]
- Hallum LE, Dagnelie G, Suaning GJ, Lovell NH. Simulating auditory and visual sensorineural prostheses: a comparative review. J. Neural. Eng. 2007;4:S58–S71. [PubMed] [Google Scholar]
- Hayes JS, Yin VT, Piyathaisere D, Weiland JD, Humayun MS, Dagnelie G. Visually guided performance of simple tasks using simulated prosthetic vision. Artif. Organs. 2003;27:1016–1028. [PubMed] [Google Scholar]
- Horton JC, Hoyt WF. The representation of the visual field in human striate cortex: a revision of the classic Holmes map. Arch. Ophthalmol. 1991;109:816–824. [PubMed] [Google Scholar]
- Humayun MS, et al. Visual perception in a blind subject with a chronic microelectronic retinal prosthesis. Vis. Res. 2003;43:2573–2581. [PubMed] [Google Scholar]
- Kaido T, Hoshida T, Taoka T, Sakaki T. Retinotopy with coordinates of lateral occipital cortex in humans. J. Neurosurg. 2004;101:114–118. [PubMed] [Google Scholar]
- Lee HW, Hong SB, Seo DW, Tae WS, Hong SC. Mapping of functional organization in human visual cortex: electrical cortical stimulation. Neurology. 2000;54:785–786. [PubMed] [Google Scholar]
- Schmidt EM, Bak MJ, Hambrecht FT, Kufta CV, O’Rourke DK, Vallabhanath P. Feasibility of a visual prosthesis for the blind based on intracortical microstimulation of the visual cortex. Brain. 1996;119:507–522. [PubMed] [Google Scholar]
- Srivastava NR, Troyk PR, Towle VL, Curry D, Schmidt E, Kufta C, Dagnelie G. Estimating phosphene maps for psychophysical experiments used in testing a cortical visual prosthesis device. Proc. of IEEE EMBS Conf. on Neural Engineering. 2007:130–133. [Google Scholar]
- Stensaas SS, Eddington DK, Dobelle WH. The topography and variability of the primary visual cortex in man. J. Neurosurg. 1974;40:747–755. [PubMed] [Google Scholar]
- Thompson RW, Barnett GD, Humayun MS, Dagnelie G. Facial recognition using simulated prosthetic pixelized vision. Invest. Ophthalmol. Vis. Sci. 2003;44:5035–5042. [PubMed] [Google Scholar]
- Troyk PR, Bak M, Berg J, Bradley D, Cogan S, Erickson R, Kufta C, McCreery D, Schmidt E, Towle V. A model for intracortical visual prosthesis research. Artif. Organs. 2003;27:1005–1015. [PubMed] [Google Scholar]
- Troyk PR, et al. Multichannel cortical stimulation for restoration of vision. EMBS/BMES Conf. 2002;3:2045–2046. [Google Scholar]
- Troyk PR, et al. Intracortical visual prosthesis research—approach and progress. IEEE-EMBS Conf. 2005;27:7376–7379. [PubMed] [Google Scholar]