Pulse trains to percepts: the challenge of creating a perceptually intelligible world with sight recovery technologies

Abstract

1. Introduction

More than 200 different gene mutations result in irreversible photoreceptor diseases, which collectively have the potential to affect over 20 million individuals worldwide (see RetNet, the Retinal Information Network, at http://www.sph.uth.tmc.edu/RetNet/; [1]). As a consequence, there is considerable interest in developing technologies to restore visual function that do not require targeting each genetic defect independently.

This review focuses on sight recovery methodologies, such as optogenetics, small molecule photoswitches and electrical prosthetics that use artificial stimulation of the retina to elicit percepts. Optogenetic proteins create novel light-sensitive ion channels or pumps that make cells responsive to light [2,3]. If inserted into a subset of remaining retinal cells, they have the potential to make these cells light sensitive [4]. Small molecule photoswitch compounds create novel light responses via small molecules that directly modulate the activity of ion channels by reversibly activating and deactivating the targeted channel with exposure to particular wavelengths of light [5–7]. Retinal and cortical prostheses directly elicit neural responses using electrical stimulation, analogous to a cochlear implant (e.g. [8–11]). In theory, optogenetics and photoswitch compounds might be developed to function using the natural illumination of the retina. However in practice, all currently developed prosthetic methods require a camera to dynamically capture the visual scene and an encoding method to translate the camera output into either a light (for optogenetics and photoswitches) or electrical stimulation protocol.

All three approaches are capable of producing very low vision in either animal models or humans (optogenetics: [4,7], prostheses: [8,12–15], small molecule photoswitches: [16–18]). However, these are early days: no approach to date has convincingly proved capable of reliably eliciting behavioural performance levels equivalent to that of human patients with, for example, finger counting levels of vision.

For all of these technologies, the interplay between the stimulating technology and the underlying neurophysiology is likely to result in distortions of the perceptual experience. Here, we describe and discuss three classes of distortions: (i) those caused by the diversity of cells in the retina, (ii) spatial distortions caused by stimulation of axon fibres in electrical prostheses, and (iii) temporal distortions caused by the sluggish kinetics of optogenetics and small molecule photoswitches. Finally, we discuss how methods for encoding the stimulus might be modified to minimize the effects of these distortions, and the ability of cortex to compensate for these various distortions in the context of what is known about plasticity in the early visual pathways.

2. The encoding problem

As was the case for first generation cochlear implants, early sight recovery devices have relied on straightforward signal processing and encoding schemes that assume that the transformation from stimulation to percept is virtually linear: these models presume that stimulating a grid of positions on the retina leads to the corresponding percept of a grid of luminous dots. However, a variety of psychophysical [19–24] and retinal [25] data, as well as the potential distortions described in this review, suggest that this ‘linear scoreboard’ model is inadequate as an encoding model. To take the development of cochlear implant technology as a guide, a combination of psychophysical and neurophysiological research has led to dramatic improvements in cochlear implant coding schemes over the years. Indeed, for cochlear implants it could be argued that improvements in encoding have been as important as improvements in physical technology in improving perceptual outcomes [26,27].

Two main ways of developing models of retinal stimulation have been proposed: bottom-up algorithms based on mimicking the neural code of individual ganglion cells and ‘top-down’ algorithms based on measuring the perceptual output of implanted individuals.

In vitro models have been developed that are capable of replicating the neural code of individual ganglion cells with impressive accuracy (e.g. [25,28–30]), thereby providing proof in principle that stimulation by a prosthetic device might eventually be able to mimic normal vision. However, any practical implementation of a prosthetic coding scheme based on a model of retinal responses will need to deal with certain formidable difficulties.

Ganglion cell responses, even when measured from a very similar location in the retina, are strikingly diverse. As well as having separate ON-centre and OFF-centre pathways, ganglion cells differ in the size of their receptive fields, the transience of their responses (how rapidly the response to a preferred stimulus declines over time) and their chromatic tuning (for reviews, see [31,32]). Linear–nonlinear cascade models successfully predict the effects of this diversity of ganglion cell responses to complex stimuli [28] and natural scenes [25]. However, these models require an estimate, for each cell, of both the linear filter representing the receptive field and the nonlinear function converting filter output to instantaneous spike rate [29]. For obvious reasons, eliciting and measuring retinal spikes in individual ganglion cells in patients would be a somewhat heroic task that is not possible with current technologies. An alternative is to create a generic code for specific cell subtypes, which could then be specifically targeted using, for example, optogenetics [25]. However, it remains to be seen how well generic models for specific cell subtypes can replicate the natural retinal code across relatively wide regions of the retina.

The alternative is to base stimulation protocols on psychophysical models that are constrained by perceptual experience. Any psychophysical model needs to be fairly basic, given that psychophysical data not only lack richness in terms of their spatio-temporal resolution, but are also slow and difficult to collect, especially in elderly patients. Indeed, as the Second Sight's Argus 60 is implanted in more patients, it has become clear that simply finding the current that determines the threshold level for detecting a percept for a mere 60 electrode array is not trivial (Second Sight Medical Products, Inc. 2015, personal communication). While there are a variety of studies modelling the percepts elicited by electrical stimulation based on behavioural performance [19–24], and it has been shown that these models show similarities to linear–nonlinear models of the retina [20], this work has not yet been integrated into a single model capable of predicting the optimal stimulation sequence needed to replicate a desired visual percept.

Although the encoding problem is a significant and important problem in its own right, in the remainder of this review we assume it can be solved. Here, we focus on three classes of perceptual distortions that are not due to errors in the encoding model, but rather are consequent on the interface between the sight recovery technology and the underlying physiology.

3. Distortions due to retinal cell diversity

Signals from mammalian cones flow to a variety of morphologically distinct (at least three cell types [33]) ‘OFF-centre’ and ‘ON-centre’ bipolar cells. These pathways pass in turn to ON- and OFF-centre ganglion cell pathways that each contain 10 or more cell types [31,32]. This diversity within ganglion cells is in striking contrast to the auditory nerve fibres stimulated by cochlear implants, whose responses show far less diversity in functional response (e.g. [34,35]). Each type of bipolar [33] and ganglion cell has strikingly different functional properties and carries distinct information about the visual input [32]. In the case of ganglion cells, these cell types form orderly mosaics [36–39], such that a given location in the visual field is represented by a wide range of cell types whose responses to a given stimulus vary dramatically (indeed, stimuli that induce firing in ON-centre pathways tend to suppress OFF-centre pathways, and vice versa).

Although the functional properties and cortical projections of many (though not all) of these cell types have been described, there is still much to be learned about how the information from these different retinal representations are combined at later stages of processing (see [40], for a review). One reason for this is that despite rapid progress [41], it remains extremely difficult to selectively stimulate or block particular cell types or ON- or OFF-centre cells, especially in primates. It is clear that these representations are not independent [42,43] and are often (likely due to the rectified nature of neural responses) complementary. This diversity of representation within bipolar and ganglion cells has significant implications for sight recovery technologies. Although optogenetics and small molecule photoswitches can be targeted with varying degrees of cell specificity (e.g. to cones [44], ganglion cells [4,5,7,12–14], bipolar ON cells [7,13,15], and even ganglion cell dendrites and soma [45]), none of the technologies currently under development has the ability to match the normal pattern of responses across the full mosaic of cell types. To achieve this would require multiple compounds, each capable of being independently stimulated (e.g. each narrowly tuned to a different wavelength of light) and each capable of selectively targeting a particular cell type (or a small subset of cell types with similar response properties). What then is the likely perceptual consequence of targeting a subset of the full cell mosaic?

(a) The effect of stimulating either the ON- or the OFF-pathway

First, we consider the potential perceptual consequences of targeting ON- or OFF-centre pathways in isolation. Figure 1a shows two example greyscale images, and figure 1b,c show simple approximations of the representations thought to be carried by ON- and OFF-pathways. While perceptually interpretable, neither looks much like the original image, and both carry impoverished information content compared to the band-pass image shown in figure 1d.

Open in a separate window

Figure 1.

A simulation of the perceptual information carried by ON- and OFF-centre pathways. (a) Two example images: a white-on-black eye-chart and a frame of a natural scene movie of a child scooting. Both subtend 24° of visual angle. (b) These two images were convolved by a bank of difference of Gaussian filters followed by rectification to provide a rough approximation of the information carried by ON-centre pathways. (c) Images filtered by an inverted version of the same bank of Gaussian filters followed by rectification provide a rough approximation of the information carried by OFF-centre pathways. (d) The difference of the images (b,c) restore a band-pass version of the original image. (e) The sum of the images (b,c) represents an approximation of one potential perceptual outcome of simultaneously stimulating both ON-centre and OFF-centre pathways. Movie versions are shown for (a–e) in the electronic supplementary material.

However, this simulation underestimates the complexity of stimulating ON- but not OFF-pathways (or vice versa), because it does not represent the fact that these pathways are complementary to each other. An absence of response in a given pathway is not at all the same thing as an absence of information from that pathway. To take a simple example, if one somehow blocked transmission in all leftward-selective motion-sensitive cells, then it would not be the case that one simply would not be able to see leftward motion (the analogy to the simulation above)—rather, due to the nature of motion opponency, it is likely that everything would be perceived as moving rightward. Interestingly, while it is possible to visualize or simulate the perceptual results of blocking transmission of leftward-tuned cells, it is impossible to visualize or stimulate the perceptual result of stimulating only ON-centre or OFF-centre pathways because there is no visual stimulus that elicits such a retinal response—responses to visual stimuli in ON-cells are always accompanied by suppression in OFF-centre cells with receptive fields in nearby locations. Indeed, one possibility is that stimulating a subset of pathways will create a ‘rivalrous’ stimulus (as when different stimuli are presented to each eye [46]) in which one pathway signals a blank field and the other signals some sort of visual stimulus, as discussed more fully below.

4. Spatial distortion due to axon fibres

Retinal prosthetics can either be implanted subretinally next to the choroid, in the space of the missing or ailing photoreceptors, or epiretinally, between the ganglion cells and the vitreous humour. Every ganglion cell has an axon that traverses the retinal surface en route to the optic nerve, between the ganglion cell bodies and the vitreous humour, so in the case of epiretinal implantation, the axon fibres lie between the electrodes and cell bodies. Stimulating a ganglion cell axon is likely to lead to the percept of a stimulus at the location associated with that axon's cell body, which could be several degrees away. Thus, an electrode that stimulates underlying axonal fibres would be expected to produce the percept of ‘comets’ whose heads lie at the intended location of stimulation and whose tails lie along the axonal fibre pathway in the direction leading away from the optic disc, as illustrated in figure 2a. As a consequence, if an electrode stimulates axons of ganglion cells with distant cell bodies, then the percepts elicited by electrical stimulation of those axons will be elongated in shape and poorly localized. Both animal models [47], modelling [48] and human data [49] suggest that epiretinal devices produce significant axonal stimulation.

Open in a separate window

Figure 2.

An example of how a model of retinal axonal pathways can predict patient percepts. (a) Schematic of why axonal stimulation results in axon comets. Each image of the retina is flipped so that the upper region of the retina represents the upper visual field. The modelled axon trajectories (green lines) are based on a computational model of axon fibre trajectories developed using traced nerve fibre bundle trajectories extracted from fundus photographs of 55 human subjects [50]. The red circles represent ganglion cell bodies whose axon fibres pass underneath the electrode on their way to the optic nerve, yellow shading represents a perceptual ‘axon-comet’. The dotted red box outlines the retinal regions shown in (b–d). (b,c) A subject implanted with the Argus 1 prosthesis (this array contained interleaved 250 and 500 μm electrodes) was simultaneously stimulated on two different pairs of electrodes. Each pair of stimulated electrodes are shown outlined in red. Predicted percepts generated using the model described above are shown in white. (d,e) Subject drawings (averaged over five trials) of the percepts induced by these stimulation patterns. (Online version in colour.)

As described previously [19], Argus I and II (epiretinal arrays) patients typically report that phosphenes appear light grey, white or yellow, with the shape varying between round to highly elongated ellipses. As the stimulation amplitude or frequency increases, subjects tend to report phosphenes as brighter with sharper contours. Suggestive of axonal stimulation, nearly all phosphenes appear as elongated ellipses (approx. 93%), with their minor axis length being less than 50% of the major axis length.

To simulate the perceptual effects of axonal stimulation, we used a previously validated model of retinal axonal pathways based on that of Nanduri et al. [49]. One Argus I and three Argus II retinal prosthesis subjects were asked to trace the shape of induced phosphenes on a touch screen. These perceptual data were fit using a model that assumed that the activation sensitivity of a passing axon fibre decays exponentially with the distance (x) between the axon's initial segment and the electrode, such that s = e^−x/λ. A constant of λ = ∞ implies that the entire axon fibre is equally sensitive to electrical stimulation, λ = 0 means only the initial segment is sensitive to stimulation. Nanduri et al. found that patient percepts were best predicted using values of λ ranging between 1.0° and 3.4° of visual angle, indicating that while electrical stimulation is not confined to the axon initial segment, it did fall off rapidly as a function of distance.

As shown in figure 2, this model was able to successfully predict the length and orientation of phosphenes based on the orientation and length of the underlying axon fibres. In addition, the model could successfully predict whether stimulation of two electrodes resulted in a single or double percept, based on whether or not the two electrodes fell along shared fibre pathways.

While the increased distance of the stimulating electrodes from the axonal fibres means that axonal stimulation may be less of a concern for subretinal devices [51,52], there are indications that in a subset of patients a certain amount of axonal stimulation occurs, especially at suprathreshold levels of stimulation. While most patients implanted with subretinal electrodes report percepts as round spots of light with a yellowish appearance, some patients report percepts that include arcs, short lines or semicircles [51,52].

In figure 3, we use this model to show examples of the perceptual distortions that might be produced by axonal stimulation with more naturalistic images, using a 101 × 101 square electrode array subtending 12° of visual angle. Here, we show simulations based on values of λ = 0.5, 1° and 2° of visual angle. Figure 3a,b show the images projected onto the retina, overlaid on the simulated axon fibre pathways. Figure 3c,d show the effect of such axonal ‘comets’. As distortions follow the path of the axon fibre pathways, the ‘comet-trails’ predicted by axonal stimulation vary in angle and length across the retina. In the simulation of a contrast-reversed (white letters on a black background) eye-chart using λ = 2 (based on [49]), only the top (e) (which subtends just less than 3° of visual angle) is clearly identifiable, consistent with Snellen acuity of 20/600, similar to the best acuity reported to date (just under 20/600) for human prosthetic users [10,17].

Open in a separate window

Figure 3.

Simulations of perceptual distortions as a result of axonal stimulation. (a,b) Two images (the same as in figure 1) are overlaid on the retinal surface. The position of the electrode array (subtending 12°) is shown by a red dotted box. (c,d) The predicted effect of axonal stimulation on the two images of figure 1b,d for λ = 0.5, 1 and 2. Movie versions are shown in the electronic supplementary material. (Online version in colour.)

The simulation of the natural image is still more troublesome: for λ = 2, even the main figure in the image cannot clearly be delineated. The critical difference seems to be that in the contrast-reversed letter chart only a small subset of the electrodes are stimulated, and the electrodes representing the black background remain entirely unstimulated. By contrast, most electrodes are partially activated in the natural scene, resulting in ‘comet-trails’ smearing the image much more severely. This is a particular concern given that the vast majority of the paradigms used to evaluate both acuity and ‘real world performance’ in retinal implant users have relied on discrete white objects on a black background (e.g. [8,17,53]) (figure 4).

Open in a separate window

Figure 4.

Example of one test within a standardized ‘Activities of daily living’ test proposed by Stingl et al. [53] (reprinted with their permission). The subject is asked to identify, describe and localize objects while sitting at a table. (Online version in colour.)

One way of reducing the perceptual consequences of comet trails might be to carry out initial image processing to enhance contrast and edges (e.g. ‘zero-crossing’ algorithms), to minimize the number of stimulated electrodes. Alternatively, it is possible that these perceptual effects can be reduced by incorporating them into the encoding model, such that the visual region encoded by an electrode is represented by its ‘comet’ rather than by the region of visual space corresponding to position of the electrode on the retinal surface. However, even if the phosphenes elicited by each electrode were accurately represented, creating small discrete phosphenes (such as spots or lines) would remain impossible, even with a very high-resolution array. For example, in figure 2c,e, it is highly ambiguous which and how many electrodes along the fibre pathway are being stimulated.

A third possibility is developing more complex encoding models that include anodic (suppressive) stimulation on electrodes on the same axon pathway but distal to the optic nerve. Thus, to create a punctate phosphene under the electrode of figure 2a, one would anodically stimulate electrodes lying above the ganglion cells represented by red circles. This technique differs from ‘current shaping’, in which a combination of anodic and cathodic stimulation on electrodes with overlapping current fields are used to ‘shape’ the current field [54–57]. The electrodes used to suppress axonal fibre stimulation need not have overlapping current fields with the stimulating electrode, but rather must lie along the same axon pathway.

5. Temporal distortion due to slow kinetics

Currently, both optogenetics and small molecule photoswitches tend to have relatively slow kinetics, and it is not clear whether these kinetics can be improved without loss of sensitivity.

As far as optogenetics are concerned, wild-type and most known variants of Channelrhodopsin-2 either have slow kinetics or low sensitivity, or both. Healthy photoreceptors (particularly rods) have a robust signal-transduction cascade that greatly amplifies the signals emanating from a small number of photons. Small molecule photoswitches and optogenetic approaches to sight restoration lack this cascade. As a consequence, the neural signal elicited by photoswitches or optogenetic proteins is weaker than that produced by properly functioning photoreceptors. This is one of the reasons why these methods currently tend to use an artificial light source that can provide greater illumination (and with wavelengths matched to the spectrally tuned photoswitch or optogenetic channel). Another method for increasing the sensitivity of the artificial photoreceptor cell is to use slow dynamics to increase the amount of time for light to integrate [58]). Although more sensitive optogenetic proteins with fast dynamics are being developed [59], those currently being used for sight recovery in animal models still have relatively slow dynamics [4] compared with normal light responses. Similarly, while small molecule photoswitches have a response to the onset of light that is faster than that of normal photoreceptors (likely due to the lack of phototransduction), the return to ground state for current remains relatively slow even in the most recent more rapid photoswitches [7] and, as described above, any increase in kinetics is likely to come at a cost to sensitivity.

Slow dynamics do not simply reduce the ability to process rapidly moving objects. Rather, they have the potential to lead to motion streaks (similar to the blur seen in a moving cursor using a sluggish monitor) and can also reduce the contrast of moving objects [4]. This streaking is induced by all forms of retinal motion, including those induced by eye (or camera) movements. Figure 5 shows a simulation of the expected perceptual effects of sluggish photokinetics, using the dynamics of the small molecule photoswitch LiGluR as an example [7]. The image of figure 5b (identical to figure 1a) was in fact taken from a movie. In figure 5c, we show that same image frame after filtering the movie using the temporal dynamics of LiGluR-MAG0₄₆₀. Stationary objects in the scene are almost unaffected. However, the sluggish temporal dynamics causes the scooting child to, rather dramatically, almost completely disappear. (As can be seen in the original image, the tipped-over scooter on the doorway is actually a different scooter from the one the child is riding.) Original and filtered movies are included in the electronic supplementary material.

Open in a separate window

Figure 5.

The perceptual effects of sluggish response kinetics. (a) Response kinetics for LiGluR-MAG0₄₆₀ and normal phototransduction. LiGluR-MAG0₄₆₀ cells are activated by visible light (time-course shown as a thick horizontal outlined bar; yellow online) and relax spontaneously in the dark. Circles (blue online) show human embryonic kidney cell recordings in voltage-clamp configuration at −75 mV (replotted with permission from [7]). Our simulated approximation (thin solid line; blue online) used an exponential onset of time-constant of 20 ms and an offset time-constant of 200 ms. For comparison, a crude simulation of primate phototransduction dynamics (using an exponential time-constant of 20 ms for both onset and offset) is also plotted (dashed line; red online). For our purposes, a crude approximation of time-courses was adequate; more accurate modelling of these time-courses can be found in [7] and [60], respectively. (b) The image of a child scooting (same as figure 1a). (c) The image of figure 1a was in fact taken from a movie sequence. Here, we show the image frame of figure 1a, with the movie filtered using the temporal dynamics of LiGluR-MAG0₄₆₀. Rather dramatically, the sluggish temporal dynamics causes the scooting child to almost completely disappear. The movie version of (c) is shown in the electronic supplementary material. (Online version in colour.)

In contrast to the ‘axon-comets’ described above, which have a fixed location on the retina, the direction and extent of motion streaks due to sluggish dynamics depend on the speed and direction of motion in the scene. Thus, while it is possible that motion streaks induced by sluggish temporal dynamics could be reduced by including them within the stimulation model, encoding such compensations would likely be relatively complex and, to make it even more challenging, would have to be computed dynamically.

6. Perceptual plasticity

While the consequences of the perceptual distortions described above may appear daunting, it is worth noting that to date no sensory or motor prosthetic has successfully recreated the missing sense or effector organ. Rather, current prosthetics have provided an approximation of the sensory input to the missing sense (e.g. cochlear implant) or effector organ (e.g. prosthetic limbs). Encouragingly, the plasticity of the auditory and motor system has proved up to the challenge of making perceptual sense of highly distorted input. Here, we discuss what is currently known about cortical plasticity and how such plasticity might serve to minimize the perceptual effects of the distortions described above.

7. Conclusion

As a final example, we show three simulations. All are once again based on a 101 × 101 array of electrodes subtending 12°. Figure 6a represents a basic scoreboard model in which the brightness of the apparent percept is linearly related to the strength of current on the retina (based on [85]). Figure 6b,c represent ‘neuro-perceptual’ simulations of the effects of electrical and small molecule photoswitch stimulation, respectively. Each simulation is strikingly different, and it is immediately apparent that array resolution may be less critical than the ability to create perceptually comprehensible percepts.

Open in a separate window

Figure 6.

Three example models of the potential perceptual experience of sight recovery. All images subtend 12° of visual angle. (a) Scoreboard model. The luminance of the apparent percept is linearly related to the strength of current on the retina. (b) Simulation of electrical stimulation. This particular simulation is based on the model of simultaneously stimulating ON- and OFF-pathways as described in figure 1e, followed by the model of axonal stimulation as described in figure 3. (c) Simulation of small molecule photoswitch stimulation. This simulation is based on the model of simulating ON-centre pathways in isolation as described in figure 1b, followed by the effects of sluggish temporal dynamics as described in figure 5. Movie versions are shown in the electronic supplementary material.

Thus, the next decade is likely to see fascinating interdisciplinary research examining the interplay between sight recovery technologies, the underlying neurophysiology and the perceptual capacities of individuals. As second-generation sight recovery technologies begin to develop encoding schemes to maximize the perceptual intelligibility of the world, we believe ‘neuro-perceptual’ models of the effects of artificial stimulation will play an increasingly important role. The simulations of this paper should not be considered as having the status of models, nor of providing genuine predictions of what might be expected of any given sight recovery technology. Rather they are an attempt to demonstrate the useful information that more fully developed and validated models might provide.

Supplementary Material

Supplementary Material

Supplementary Material

Supplementary Material

Acknowledgements

References