Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2018 Oct 29;7(5):29.
doi: 10.1167/tvst.7.5.29. eCollection 2018 Sep.

Motion Parallax Improves Object Recognition in the Presence of Clutter in Simulated Prosthetic Vision

Cheng Qiu  1   2 , Kassandra R Lee  1 , Jae-Hyun Jung  1 , Robert Goldstein  1 , Eli Peli  1
Affiliations

Motion Parallax Improves Object Recognition in the Presence of Clutter in Simulated Prosthetic Vision

Cheng Qiu et al. Transl Vis Sci Technol. .

Abstract

Purpose: Efficacy of current visual prostheses in object recognition is limited. Among various limitations to be addressed, such as low resolution and low dynamic range, here we focus on reducing the impact of background clutter on object recognition. We have proposed the use of motion parallax via head-mounted camera lateral scanning and computationally stabilizing the object of interest (OI) to support neural background decluttering. Simulations in head-mounted displays (HMD), mimicking the proposed effect, were used to test object recognition in normally sighted subjects.

Methods: Images (24° field of view) were captured from multiple viewpoints and presented at a low resolution (20 × 20). All viewpoints were centered on the OI. Experimental conditions (2 × 3) included clutter (with or without) × head scanning (single viewpoint, 9 coherent viewpoints corresponding to subjects' head positions, and 9 randomly associated viewpoints). Subjects used lateral head movements to view OIs in the HMD. Each object was displayed only once for each subject.

Results: The median recognition rate without clutter was 40% for all head scanning conditions. Performance with synthetic background clutter dropped to 10% in the static condition, but it was improved to 20% with the coherent and random head scanning (corrected P = 0.005 and P = 0.049, respectively).

Conclusions: Background decluttering using motion parallax cues but not the coherent multiple views of the OI improved object recognition in low-resolution images. The improvement did not fully eliminate the impact of background.

Translational relevance: Motion parallax is an effective but incomplete decluttering solution for object recognition with visual prostheses.

Keywords: motion parallax; object recognition; sensory substitution device; visual prosthesis.

PubMed Disclaimer

Figures

Figure 1
Figure 1
The capture of simulated prosthetic images. (a) Top view of the physical setup showing the nine camera positions on the left, which are 3 cm apart. For an object set 50 cm from the camera, each 3-cm lateral shift requires approximately 3° rotation to re-center the OI. For the purpose of this figure, the dead leaves background is cropped to cover the FoV of camera position rays 1, 3, and 5. (b) Captured image examples from nine lateral viewpoints from the left to the right.
Figure 2
Figure 2
Natural image examples with various ranges of edge densities and the corresponding examples of the dead leaves backgrounds in high and low resolutions. The low-resolution images of dead leaves (20 × 20) are low-pass filtered, as in the BrainPort simulation, to reduce pixel edge artifacts.
Figure 3
Figure 3
Example images in Clutter Complexity experiment. Images with the running man sculpture as presented in the following seven experimental conditions: from left to right, high-resolution control without background clutter and low-resolution images with 0%, 5%, 10%, 15%, 20%, and 25% background clutter complexity.
Figure 4
Figure 4
Median recognition rates with different resolutions and background clutter complexities. Background clutter reduces the recognition rate in low-resolution images, though the effect was smaller than the effect from the resolution change (from 480 × 480 to 20 × 20 both without background clutter). The P values were calculated using the Wilcoxon signed-rank test with paired samples and were Bonferroni corrected for multiple comparisons. Error bars represent the interquartile range.
Figure 5
Figure 5
Recognition rate of the same images collected in the Clutter Complexity (x-axis) and the static condition of the Motion Parallax (y-axis) experiments. The blue triangles are from the no clutter and static condition, and the red circles show the recognition rates with the cluttered background. Each point represents the responses to one object under the corresponding condition (70 points in total). The points may overlap indicating the same recognition rates among the objects. The darker the icon is, the more points are overlapping. For example, the majority recognition rates from the cluttered condition are either 0% or 10%. The recognition rates in the two experiments are significantly correlated.
Figure 6
Figure 6
Median recognition rate and average response time for each condition. (a) Median recognition rates across the 35 objects for the six experimental conditions. Background clutter significantly reduces the recognition rates. Motion parallax (both coherent and random scanning) significantly improved the object recognition rate. Error bars represent the interquartile range. (b) Average response times for the six conditions. The conditions with clutter show a significantly longer response time than the conditions without clutter. Subjects also tended to spend more time in the conditions with multiple views when compared with the static conditions. The P values were Bonferroni corrected. Error bars are standard error of the mean (SEM).
Figure 7
Figure 7
The difference in object recognition between with and without background clutter (Recognoclutter − Recogclutter) for binary edge images as a function of resolution, calculated from the data presented in figure 8 of Jung et al.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

References

    1. Leonard R, Gordon AR. Statistics on Vision Impairment: A Resource Manual. 5th ed. New York, NY: Lighthouse International; 2002.
    1. da Cruz L, Coley BF, Dorn J, et al. The Argus II epiretinal prosthesis system allows letter and word reading and long-term function in patients with profound vision loss. Br J Ophthalmol. 2013;97:632–636. - PMC - PubMed
    1. Stingl K, Bartz-Schmidt KU, Besch D, et al. Subretinal visual implant alpha IMS-clinical trial interim report. Vision Res. 2015;111(Pt B):149–160. - PubMed
    1. Wicab I. Middleton, WI: 2017. BrainPort V100. Available from: https://www.wicab.com/brainport-v100. Accessed 26 September, 2017.
    1. Nau A, Bach M, Fisher C. Clinical tests of ultra-low vision used to evaluate rudimentary visual perceptions enabled by the BrainPort Vision Device. Transl Vis Sci Technol. 2013;2(3):1. - PMC - PubMed
-