Towards biologically plausible phosphene simulation for the differentiable optimization of visual cortical prostheses
- PMID: 38386406
- PMCID: PMC10883675
- DOI: 10.7554/eLife.85812
Towards biologically plausible phosphene simulation for the differentiable optimization of visual cortical prostheses
Abstract
Blindness affects millions of people around the world. A promising solution to restoring a form of vision for some individuals are cortical visual prostheses, which bypass part of the impaired visual pathway by converting camera input to electrical stimulation of the visual system. The artificially induced visual percept (a pattern of localized light flashes, or 'phosphenes') has limited resolution, and a great portion of the field's research is devoted to optimizing the efficacy, efficiency, and practical usefulness of the encoding of visual information. A commonly exploited method is non-invasive functional evaluation in sighted subjects or with computational models by using simulated prosthetic vision (SPV) pipelines. An important challenge in this approach is to balance enhanced perceptual realism, biologically plausibility, and real-time performance in the simulation of cortical prosthetic vision. We present a biologically plausible, PyTorch-based phosphene simulator that can run in real-time and uses differentiable operations to allow for gradient-based computational optimization of phosphene encoding models. The simulator integrates a wide range of clinical results with neurophysiological evidence in humans and non-human primates. The pipeline includes a model of the retinotopic organization and cortical magnification of the visual cortex. Moreover, the quantitative effects of stimulation parameters and temporal dynamics on phosphene characteristics are incorporated. Our results demonstrate the simulator's suitability for both computational applications such as end-to-end deep learning-based prosthetic vision optimization as well as behavioral experiments. The modular and open-source software provides a flexible simulation framework for computational, clinical, and behavioral neuroscientists working on visual neuroprosthetics.
Keywords: bionic vision; blindness; cortical stimulation; deep learning; human; neural implants; neuroscience; neurotechnology; simulated prosthetic vision.
© 2024, van der Grinten, de Ruyter van Steveninck, Lozano et al.
Conflict of interest statement
Mv, Jd, AL, LP, BR, PR, Mv, Rv, UG, YG No competing interests declared
Figures
![Figure 1.](https://www.ncbi.nlm.nih.gov/pmc/articles/instance/10883675/bin/elife-85812-fig1.gif)
![Figure 1—figure supplement 1.](https://www.ncbi.nlm.nih.gov/pmc/articles/instance/10883675/bin/elife-85812-fig1-figsupp1.gif)
![Figure 1—figure supplement 2.](https://www.ncbi.nlm.nih.gov/pmc/articles/instance/10883675/bin/elife-85812-fig1-figsupp2.gif)
![Figure 1—figure supplement 3.](https://www.ncbi.nlm.nih.gov/pmc/articles/instance/10883675/bin/elife-85812-fig1-figsupp3.gif)
![Figure 2.](https://www.ncbi.nlm.nih.gov/pmc/articles/instance/10883675/bin/elife-85812-fig2.gif)
![Figure 3.](https://www.ncbi.nlm.nih.gov/pmc/articles/instance/10883675/bin/elife-85812-fig3.gif)
![Figure 3—figure supplement 1.](https://www.ncbi.nlm.nih.gov/pmc/articles/instance/10883675/bin/elife-85812-fig3-figsupp1.gif)
![Figure 3—figure supplement 2.](https://www.ncbi.nlm.nih.gov/pmc/articles/instance/10883675/bin/elife-85812-fig3-figsupp2.gif)
![Figure 4.](https://www.ncbi.nlm.nih.gov/pmc/articles/instance/10883675/bin/elife-85812-fig4.gif)
![Figure 5.](https://www.ncbi.nlm.nih.gov/pmc/articles/instance/10883675/bin/elife-85812-fig5.gif)
![Figure 6.](https://www.ncbi.nlm.nih.gov/pmc/articles/instance/10883675/bin/elife-85812-fig6.gif)
![Figure 7.](https://www.ncbi.nlm.nih.gov/pmc/articles/instance/10883675/bin/elife-85812-fig7.gif)
![Figure 8.](https://www.ncbi.nlm.nih.gov/pmc/articles/instance/10883675/bin/elife-85812-fig8.gif)
![Figure 8—figure supplement 1.](https://www.ncbi.nlm.nih.gov/pmc/articles/instance/10883675/bin/elife-85812-fig8-figsupp1.gif)
![Figure 8—figure supplement 2.](https://www.ncbi.nlm.nih.gov/pmc/articles/instance/10883675/bin/elife-85812-fig8-figsupp2.gif)
![Figure 8—figure supplement 3.](https://www.ncbi.nlm.nih.gov/pmc/articles/instance/10883675/bin/elife-85812-fig8-figsupp3.gif)
![Figure 8—figure supplement 4.](https://www.ncbi.nlm.nih.gov/pmc/articles/instance/10883675/bin/elife-85812-fig8-figsupp4.gif)
![Author response image 1.](https://www.ncbi.nlm.nih.gov/pmc/articles/instance/10883675/bin/elife-85812-sa2-fig1.gif)
![Author response image 2.](https://www.ncbi.nlm.nih.gov/pmc/articles/instance/10883675/bin/elife-85812-sa2-fig2.gif)
Update of
- doi: 10.1101/2022.12.23.521749
Similar articles
-
Utilization of brain scans to create realistic phosphene maps for cortical visual prosthesis simulation studies.Annu Int Conf IEEE Eng Med Biol Soc. 2023 Jul;2023:1-4. doi: 10.1109/EMBC40787.2023.10341189. Annu Int Conf IEEE Eng Med Biol Soc. 2023. PMID: 38083444
-
The Influence of Phosphene Synchrony in Driving Object Binding in a Simulation of Artificial Vision.Invest Ophthalmol Vis Sci. 2023 Dec 1;64(15):5. doi: 10.1167/iovs.64.15.5. Invest Ophthalmol Vis Sci. 2023. PMID: 38051263 Free PMC article.
-
End-to-end optimization of prosthetic vision.J Vis. 2022 Feb 1;22(2):20. doi: 10.1167/jov.22.2.20. J Vis. 2022. PMID: 35703408 Free PMC article.
-
Electrical Stimulation of Visual Cortex: Relevance for the Development of Visual Cortical Prosthetics.Annu Rev Vis Sci. 2017 Sep 15;3:141-166. doi: 10.1146/annurev-vision-111815-114525. Epub 2017 Jul 28. Annu Rev Vis Sci. 2017. PMID: 28753382 Free PMC article. Review.
-
Restoration of vision in blind individuals using bionic devices: a review with a focus on cortical visual prostheses.Brain Res. 2015 Jan 21;1595:51-73. doi: 10.1016/j.brainres.2014.11.020. Epub 2014 Nov 15. Brain Res. 2015. PMID: 25446438 Review.
References
MeSH terms
Grants and funding
- P15-42 'NESTOR'/Nederlandse Organisatie voor Wetenschappelijk Onderzoek
- 17619 'INTENSE'/Nederlandse Organisatie voor Wetenschappelijk Onderzoek
- 899287 'NeuraViper'/HORIZON EUROPE Excellent Science
- 650003/Horizon 2020 Framework Programme
- 024.005.022 'DBI2'/Nederlandse Organisatie voor Wetenschappelijk Onderzoek
LinkOut - more resources
Full Text Sources