End-to-end optimization of prosthetic vision

Training procedure

The end-to-end model was implemented in PyTorch version 1.3.1, using a NVIDIA GeForce GTX 1080 TI graphics processing unit (GPU) with CUDA driver version 10.2. The trainable parameters of our model were updated using the Adam optimizer (Kingma & Ba, 2015). The end-to-end model was treated as a single system (i.e. all components of the model were trained simultaneously). To account for potential convergence of the model parameters toward local optima (i.e. to reduce the likelihood that a suitable parameter configuration of the network is missed due to a combination of a specific weights initialization and learning rate), a “random restarts” approach was used. That is, each model was trained five times, each time randomly starting with a different weight initialization. The results only show the best performing one out of these five models (i.e. the one with the lowest loss on the validation dataset), unless stated otherwise.

Experiment 1: SPV-based reconstruction of visual stimuli

The objective of the first experiment was to test the basic ability of the proposed end-to-end model to encode a stimulus into regular, binary phosphene representations, and decode these into accurate reconstructions of the input image. For this purpose, we trained the model on a self-generated dataset containing basic black and white images with a randomly positioned lowercase alphabetic character. Each image in the dataset is 128×128 pixels, and contains a randomly selected character, displayed in one of 47 fonts (38 fonts for the training dataset and 9 fonts for the validation dataset). The model was trained to minimize the (pixel-wise) MSE.

between the intensities of the input image x⁽ⁿ⁾ and the output reconstruction x^(n) over all training examples 1 ≤ n ≤ N.

The results of Experiment 1 for the best performing model out of the five random restarts are displayed in Figure 3. As can be observed, the reconstruction loss is successfully minimized until convergence after 39 epochs. The performance metrics on the validation dataset indicate that the model is capable of adequately reconstructing the input image from the generated SPV representation (MSE = 0.018, SSIM = 0.139, and PSNR = 17.59). Notably, the model adopted an encoding strategy where the presence of the alphabetic character (white pixels) is encoded with the absence of phosphenes and vice versa, resulting in an average electrode activity (percentage of active electrodes) of 95.65%. Such an “inverted” encoding strategy was found for two out of the five random restart initializations and observed for all 234 images in the validation dataset.

Open in a separate window

Figure 3.

Results of Experiment 1. The model was trained to minimize mean squared error loss. (a) The training curves indicating the loss on the training dataset and validation dataset during the training procedure. (b) Visualization of the network input (left) the simulated prosthetic vision (middle) and the reconstruction (right).

Experiment 2: Shaping phosphene vision via constrained optimization

In the second experiment, we assess whether our model allows the inclusion of additional constraints. To exemplify this advantage of our proposed approach, we chose to evaluate the effects of adding a sparsity loss term. Considering the potentially adverse effects of electrical stimulation (Lewis et al., 2015; McCreery, Agnew, Yuen, & Bullara, 1988), one might want to introduce such a sparsity requirement that constrains the stimulation protocol, limiting the neural degradation by enforcing minimal energy transfer to neural tissue. Let s = (s₁,…,s_M) denote a stimulation vector representing the stimulation pattern for M electrodes. We define sparsity as the L1 norm on the stimulation protocol:

where s⁽ⁿ⁾ is the stimulation vector for the n-th training example. The objective is to minimize the total loss:

where L_I is the pixel-wise reconstruction loss and the parameter κ can be adjusted to choose the relative weight between the reconstruction loss and sparsity loss. We evaluated 13 values of κ, again each time using the random restarts approach, testing five different weight initializations. We performed a regression analysis on the overall percentage of active electrodes and the reconstruction performance (MSE) to evaluate the effectiveness and the decrease in performance, respectively.

The results for three of the κ-parameters are displayed in Figure 4. As can be observed, adding additional sparsity loss by increasing the κ parameter resulted in an overall lower percentage of active phosphenes. For larger values of κ, the reconstruction quality dropped. In contrast to the results of Experiment 1, addition of a sparsity loss led to phosphene patterns that more naturally encode the presence of pixels by the presence of phosphenes (instead of vice versa).

Open in a separate window

Figure 4.

Results of Experiment 2. The model was trained on a combination of mean squared error loss and sparsity loss. The 13 different values for sparsity weight κ were tested. (a) Visualization of the results for three out of the 13 values for κ. Each row displays the performance metrics for the best-performing model out of five random restarts, and one input image from, the validation dataset (left), with the corresponding simulated phosphene representation (middle) and reconstruction (right). (b) Regression plot displaying the sparsity of electrode activation and the reconstruction error in relation to the sparsity weight κ. The red circles indicate the best-performing model for the corresponding sparsity condition, as visualized in panel a.

Experiment 3: End-to-end phosphene vision for naturalistic tasks

In the third experiment, the model was trained on a more complex and naturalistic image dataset. To this end, we made use of the ADE20k semantic segmentation dataset (Zhou et al., 2017; Zhou et al., 2019). Compared with the synthetic character dataset, which we used for the aforementioned experiments, one of the key challenges of such naturalistic stimuli is that instead of merely foreground objects on a plain dark background, the images contain abundant information. Here, not all features may be considered relevant, and therefore the task at hand (implemented by a loss function) should control which information needs to be preserved in the phosphene representations. Note that the proposed end-to-end framework allows for optimization to virtually any type of task that can be formalized as a loss function. However, with the current experiments, we merely aimed to demonstrate the basic principle by exploring the encoding strategies for three different types of image reconstruction tasks. We compared the pixel-based MSE reconstruction task that was used in the first experiment with two other types of reconstruction tasks: first, an unsupervised perceptual reconstruction task (see related work by, Johnson, Alahi, and Fei-Fei (2016), Ledig et al. (2017)) which, in contrast to the pixel-wise MSE-based reconstruction task, aims to only preserve high-level perceptual features. Second, a supervised semantic boundary reconstruction task to evaluate the additional value of using labeled supervision to specify which information needs to remain preserved in the reconstructions.

The perceptual reconstruction task was formulated with the aim of minimizing higher-level perceptual differences between the reconstruction and the input image. These more abstract perceptual differences are defined in feature-space, as opposed to the more explicit per-pixel differences which were used in the previous experiments. The feature loss is defined as

where N is the number of training examples, φd(x(n)) and φd(x^(n))are the d-th layer feature maps extracted from the input image and the reconstruction image using the VGG16 model pre-trained on the ImageNet dataset (Simonyan & Zisserman, 2015) and K is the number of feature maps of that layer. For lower values of d, the perceptual loss reveals more explicit differences in low-level features such as intensities and edges, whereas for higher values of d the perceptual loss focuses on more abstract textures or conceptual differences between the input and reconstruction (Zeiler & Fergus, 2014). We chose d equal to 3 as an optimal depth for the feature loss, based on a comparison between different values (see Figure 5).

Open in a separate window

Figure 5.

Comparison between different values of d for the perceptual reconstruction task that was used in Experiment 3, where d indicates the layer depth for the VGG-based feature loss.

In the supervised semantic boundary reconstruction task, the objective was not to minimize the differences between reconstruction and input image. Instead, we aimed to provide labeled supervision that guides the model toward preserving information of semantically defined boundaries. Here, the objective was to minimize the differences between the output prediction of the decoder and processed semantic segmentation target labels of the ADE20K dataset. The semantic segmentation labels that are provided with the dataset were converted to a binary segmentation map representing the boundaries between semantic categories (i.e. boundary pixels have a value of 1 and non-boundary pixels contain the value 0). The reconstruction loss was formalized as the weighted binary cross entropy (BCE), defined as:

where zj(n) is the ground truth boundary segmentation label for pixel j of example n and w is a constant that is introduced for counterbalancing the unequal distribution of non-boundary compared to boundary pixels. In our experiments, w is set equal to 0.925, matching the inverse ratio of boundary pixels to non-boundary pixels. Both for the perceptual loss and the BCE loss we included a sparsity loss as in Experiment 2.

In addition to the three training conditions for our proposed end-to-end model, we included separate conditions where the decoder was trained on SPV representations that were generated using conventional image processing to facilitate a quantitative comparison. These reference SPV images were generated using Canny edge detection (Canny, 1986) and a deep learning-based contour detection method, called holistically nested edge detection (Xie & Tu, 2017). Differences in reconstruction performance were tested for significance using a paired t-test on the minibatches of the validation dataset. The significance level α was set to 0.05 and adjusted with a Bonferroni correction to correct for multiple comparisons.

The results of Experiment 3 are displayed in Figure 6, Figure 7, and Table 3. Both for the perceptual reconstruction tasks and the supervised semantic boundary reconstruction task, the model seems to have adopted a different phosphene encoding strategy compared to the intensity-based reconstruction task. The average MSE is significantly lower in the intensity-based reconstruction task, and the average FSIM is significantly higher in the perceptual reconstruction task. In the supervised semantic boundary condition, 69.7% of the boundary pixels were classified correctly.

Open in a separate window

Figure 6.

Results of Experiment 3. The model was trained on naturalistic stimuli, comparing three reconstruction tasks. (a) Original image. (b) Pixel intensity-based reconstruction task with MSE loss (see Equations 1–3). (c) Perceptual reconstruction task, using VGG feature loss (see Equation 4; d is set equal to 3). (d) Semantic boundary reconstruction task, using weighted BCE loss (see Equation 5) between the reconstruction and the ground truth semantic boundary label (i.e. a binary, boundary-based, version of the ground truth label from the dataset). (e) Simulated prosthetic percept after conventional image preprocessing with (left) Canny edge detection (Canny, 1986) and (right) holistically nested edge detection (Xie & Tu, 2017).

Open in a separate window

Figure 7.

Receiver-operator curves for the semantic boundary prediction task in Experiment 3. Our proposed end-to-end method is compared against existing approaches: Canny edge detection (Canny, 1986) and holistically nested edge detection (Xie & Tu, 2017). The specificity (1 - False Positive Rate), sensitivity and area under the curve (AUC) of the thresholded predictions are also provided in Table 3.

Compared to the reconstructions of the SPV-encodings using existing approaches, our end-to-end model achieved adequate reconstruction performance: our model scored the highest performance for the intensity-based reconstruction (MSE, SSIM, and FSIM) and the perceptual reconstruction (SSIM and FSIM). In the semantic boundary reconstruction task, our model scored average for the accuracy, sensitivity, specificity, and precision, and had the largest area under the receiver-operator curve (AUC) compared to the other models.

Automated optimization

Our end-to-end model is based on an autoencoder architecture, and aims to make use of their well-described ability to efficiently encode information into a low-dimensional latent representation (Bengio et al., 2013). Instead of optimizing image preprocessing as an isolated operation, our approach is designed to automatically optimize the entire process of phosphene generation for a given task. The results from Experiment 1 demonstrate that the model successfully converges to an optimal encoding strategy for a latent representation that consisted of a 32×32 binary simulated phosphene pattern. The model achieved adequate reconstruction performance, as indicated by the low MSE of 0.018 on the validation dataset. Note, that in the unconstrained setting, the model merely maximizes information transfer and has no knowledge about practical requirements, such as sparsity. Therefore, the possible phosphene encoding strategies that may be found by the model are not limited to ecologically useful solutions. This is exemplified by the “inverted” phosphene encoding strategy with a large average electrode activity of 95.65%. Here, finding an undesirable phosphene encoding strategy can be considered an expected consequence of an imprecise learning objective. To guide the model toward useful latent SPV representations, in experiments 2 and 3, we implement additional regularizing constraints (sparsity), and explore different optimization tasks, as discussed below.

An important requirement for automated optimization with deep learning, is that all components of the artificial neural network make use of differentiable operations. In this paper, we contribute a basic implementation of a fully differentiable phosphene simulation module, including a straight-through estimator for quantized phosphene activation. Further studies could adapt or extend this implementation to test automated optimization for different phosphene simulations, for instance, varying the number of electrodes, and the positions of the phosphenes.

Tailored optimization to sparsity constraints

Implementing additional constraints in the optimization procedure may provide a general solution to account for practical, medical, or biophysical limitations of the prosthetic device. For instance, the inflammatory response of brain tissue is a major concern that limits the long-term viability of cortical electrode implants (Fernández, Alfaro, & González-López, 2020; Polikov, Trresco, & Reichert, 2005) and these adverse effects may partly be avoided by limiting the chronic electrical stimulation itself (Lewis et al., 2015; McCreery et al., 1988). The results of Experiment 2 demonstrate that implementation of such a sparsity constraint may help to regularize the electrode activity. Comparing the results with Experiment 1, we can observe that even with a low value for the sparsity parameter κ, the new objective function causes the model to find a more ecologically useful encoding strategy. Notably, a larger sparsity weight κ results in fewer active electrodes, but also in impaired reconstruction performance. Choosing a balanced value for κ, depending on the needs of the patient, can be seen as a part of a tailored optimization approach of image preprocessing in prosthetic vision.

Importantly, the proposed method enables the implementation of virtually any type of additional constraint that can be incorporated in the optimization procedure. Other examples of biophysical limitations for prosthetic vision, besides sparse electrode activation, could include minimal distance for simultaneously activated electrodes, maximal spread of electrode use, or minimal temporal separation. Future research focusing on such biophysical limitations could extend the proposed method to include such or other constraints in the optimization procedure.

Task-specific optimization for naturalistic settings

Due to the relative complexity, and the presence of non-relevant information, the encoding of naturalistic scenes into phosphenes remains a challenge and it requires task-dependent processing. This challenge is explicitly addressed by the proposed end-to-end approach.

In Experiment 3, we tested three different reconstruction tasks. The results indicate that for different tasks the model converges to a different optimal encoding strategy, which may indicate task-specific optimization. The higher FSIM and the lower MSE in the perceptual reconstruction task, compared to the intensity-based reconstruction task, indicate that information about the higher-level perceptual features is favored over pixel-intensity information. Similarly, when the model was trained with BCE-loss to reconstruct the processed target labels from the ADE20k dataset, only semantic boundary information was preserved.

The phosphene encodings that were found by the model in this condition are comparable to those found with traditional preprocessing approaches (see Figure 6e), and yield similar reconstruction quality (see Table 3) compared to edge detection (Canny, 1986) or holistic contour detection (Xie & Tu, 2017). Note that a variation on these approaches is investigated in Sanchez-Garcia et al. (2020), who demonstrated that preprocessing with semantic segmentation may successfully improve object recognition performance in simulated prosthetic vision (compared to preprocessing with conventional edge detection techniques). Different from the aforementioned traditional strategies, the proposed end-to-end architecture, merely requires supervision to the output reconstructions and the labels do not directly control the phosphene representations themselves. The proposed end-to-end method takes the advantages of existing deep learning approaches (such as supervision with large precisely labeled data sets) to achieve comparable results. In addition to that, our proposed method provides a generalized approach that opens the possibility for task-specific and tailored optimization.

The VGG feature loss and BCE loss that were implemented in this paper were not chosen only because of their well-established application in optimization problems (Asgari Taghanaki, Abhishek, Cohen, Cohen-Adad, & Hamarneh, 2021; Zhang, Isola, Efros, Shechtman, & Wang, 2018) but also because they represent basic functions that are normally performed in the brain. The feature representations found in deep neural networks illustrate a similar processing hierarchy to that of the visual cortex (Güçlü & van Gerven, 2015; Yamins, Hong, Cadieu, Solomon, Seibert, & DiCarlo, 2014) and boundary detection is one of these processing steps needed for segregation of objects from background (Roelfsema, Lamme, Spekreijse, & Bosch, 2002). Although many details about the downstream information processing of direct stimulation in V1 are yet to be discovered, we know that conscious awareness of a stimulated percept requires coordinated activity across a whole network of brain areas (Bosking, Beauchamp, & Yoshor, 2017). By acting as a digital twin, a well-chosen reconstruction task may mimic the downstream visual processing hierarchy, enabling direct optimization of visual prosthetics to the biological system. Still, fully optimizing the interaction between prosthetic stimulation and the downstream visual processing, requires a deep understanding of the biological networks involved. The proposed end-to-end approach is designed in a modular way and future research can extend the concept with virtually any reconstruction model and task.

Tailored optimization to realistic phosphene mappings

The precise characteristics of the artificial percept that can be generated with visual prosthetics will depend on many factors, including the electrode placement and the visual cortex of the patient. By including a more realistic simulation module with customized phosphene mapping, we explored the potential of our end-to-end method to optimize for specific phosphene configurations.

The results of Experiment 4 indicate that our end-to-end approach can successfully optimize phosphene encoding for arbitrary configurations. A reduction of the number of available phosphenes from 650 to 488 or 325 phosphenes was associated with reduced reconstruction performance. The overall reconstruction performance with the customized phosphene mapping was lower compared to the regular phosphene mapping that was used in Experiment 3. This reduction may partly be explained by the reduced number of phosphenes. However, this was not formally tested. Possibly, the extension of our model to arbitrary phosphene mappings forms an inherently more challenging task.

Knowledge about the perceived phosphene coverage, which is unique for every patient, can be informative for finding a suitable encoding strategy (Buffoni, Coulombe, & Sawan, 2005; Kiral-Kornek, Savage, O'Sullivan-Greene, Burkitt, & Grayden, 2013). By including a customizable phosphene simulation module in our end-to-end architecture, we aim to provide a tool that can be used for tailored optimization to implant- or patient-specific characteristics.

Supported by a grant (NESTOR) of the Dutch Organization for Scientific Research (NWO).

Commercial relationships: none.

Corresponding author: Jaap de Ruyter van Steveninck.

Email: jaap.deruyter@donders.ru.nl.

Address: Department of Artificial Intelligence, Donders Institute for Brain, Cognition and Behaviour, Radboud University, Heyendaalseweg 135, 6525 AJ Nijmegen, The Netherlands.