Portable Multi-focal Visual Evoked Potential Diagnostics for Multiple Sclerosis/Optic Neuritis patients

Hardware

We have developed a wireless headset based on mobile virtual reality technology, where a smartphone with an OLED screen displays stimuli while an array of eight neuroelectric sensors attached to the rear of the headset capture the subject’s responses from the visual cortex. We are using an updated prototype similar to the one that was developed in our lab for the diagnosis of age-related macular degeneration (AMD) [^{10, 11}]; the modifications include: a more ergonomic fit, decreased weight, and better ventilation for the headset; the incorporation of a commercial eye-tracking device (Pupil Labs VR/AR add-on [¹²]); and a new generation of our custom hydrogel electrodes.

Headmount

The headset is primarily made of parts that were 3D printed using a Formlabs Form 3 stereolithographic system. The design mounts a smartphone to the front using a switchable adapter plate and positions sensors on the back with an adjustable tension mechanism to allow consistent electrode contacts at precise scalp areas. A combination of rigid and flexible materials have been used to ensure a comfortable and light-tight fit on the head. The interior temperature and the headset’s overall weight were two other crucial factors that received particular attention. The headset also features a phototransistor sensor for accurate timing of stimulus onset and reversal and eye-tracker cameras to ensure the subjects’ compliance to test protocols.

Neuroelectric Sensing

The NeuroVEP system makes use of custom reusable hydrogel electrodes that are both comfortable and convenient to set up. The tip of the soft yet mechanically robust hydrogel electrode (see Figure 1.B.) is textured in order to hold a small layer of liquid electrolyte, which penetrates through scalp hair, providing a stable and low impedance contact. The tip diameter is between 6–7mm, smaller than standard 10mm EEG cup electrodes. We have previously established [¹³] that contact impedances (at 30Hz) below 150 kOhms give adequate low noise levels, where values of 50–100 kOhm after touch-ups with additional electrolyte are typical.

Open in a separate window

Figure 1.

Portable wireless NeuroVEP system. A. NeuroVEP device prototype integrating NeuroVEP sensor with visual stimulus headset, B. Closeup of NeuroVEP EFEG sensor arrays, C. Location of electrodes on the scalp over the visual cortex, D. Dichoptic stimulus and typical neuroelectric response, Visual pathway.

Image in part D: Derived from (https://commons.wikimedia.org/wiki/File:Human_visual_pathway.svg)

Visual Stimuli

There are two sections to the test: a short 2.5-minute ffVEP test is followed by a longer 12.5-minute mfVEP test. The two tests are separated by a 30-second break period, and this makes the total length of the test 15.5 minutes. For the ffVEP test, a total of 90 trials are recorded for each eye, with intervals of 0.5 s plus a random 0 to 0.1 s between stimulations; the trials are broken into 6 segments of 30 repetitions, switching eyes every new segment where there is a discarded onset/offset period of 2 seconds. The mfVEP test has a total of 16384 overlapping trials at a rate of 60 frames per second; the sequence is divided into 16 segments of 1024 trials, switching eyes every new segment with a similar onset/offset period at the start.

The Google Pixel 2 XL smartphone’s OLED screen combined with optics that resemble the Google VR Daydream View (first version, 2016) was used for displaying the stimulus. Given the light-tightness of our headset and minimal reflected light in the viewer, the OLED screen on the phone offers an effectively infinite contrast ratio. The stimuli patterns were generated as high-resolution pixel maps (2048×2048) using custom Python programs, then were loaded into a custom app using the Google VR Android SDK with OpenGLES2 texture rendering.

The mfVEP stimulus is similar to a dartboard that extends to 22.25° eccentricity and is broken into 36 visual field subregions [¹⁴] (Figure 2.B.). The sector sizes are scaled based on cortical magnification factors to produce relatively similar cortical stimulation. Sectors are reversed (or not) simultaneously according to 36 pseudorandom uncorrelated sequences at 60 frames per second; individual responses are extracted using a correlation between each sector sequence’s time markers and the continuous EEG recording, known as the m-sequence technique [¹⁵]. The ffVEP stimulus is the same geometry dartboard, but the whole pattern reverses at the same time (Figure 2.A.).

Open in a separate window

Figure 2.

A. ffVEP dartboard pattern-reversal visual Stimulus, B. mfVEP 36 sectors visual stimulus.

Data Analysis

The EEG recording (at 1000 samples per second) and paradigm event data are streamed wirelessly to a computer as it is acquired and are cached into a file based on the Advanced Scientific Data Format (ASDF) [¹⁶], along with subject metadata and electrode impedance measurements. On the computer, an automated data analysis framework does the work of post-processing, analysis, and classification separately on the ffVEP and mfVEP signals.

Before subsequent data processing, each channel is prefiltered using a Stationary Wavelet Transform (SWT) baseline removal with an effective cutoff frequency of 0.5 Hz; this step reduces the effect of impulse response artifacts in subsequent filtering steps [¹⁰]. For this study, where the responses will be primarily analyzed using machine learning algorithms that may be sensitive to excess noise, we made the choice to reduce the frequency content to the band of 3–13Hz (which includes P1 deflections as well as intrinsic alpha wav’es), using a Bessel high-pass IIR and an SWT low-pass filter. It is important to recognize that bandwidth reduction will inevitably create some distortion of time domain features, such as the latency of the P1 peaks; however, many other experimental aspects, such as the details of the stimuli presentation will also affect the distributions of response parameters – this is precisely why each laboratory or device system must have its own set of normative data to compare against [¹⁷].

Full-Field Neuro-responses Signal Processing

The steps involved in the processing of the full-field responses are shown in Figure 3. The raw data initially undergoes the bandwidth reduction filters described above. Then, an unsupervised machine learning outlier rejection algorithm (Scikit-Learn’s “Isolation Forest” model [¹⁸]) excludes trials with outlier variances which are mainly caused by movement artifacts. Next, an alpha wave sensitive outlier rejection algorithm does a Fast Fourier Transform (FFT) on individual trials and excludes those with power predominantly within the alpha frequency band (9~12 Hz).

Open in a separate window

Figure 3.

Full-Field Stimulus Data Analysis Steps.

It has been shown that alpha waves are amplified during resting, sleeping, eye closures and in our experience, when the subject is unable to see the stimulus; conversely, alpha waves are attenuated by visual attention [^19–21]. Then, the selected trials recorded from each electrode are averaged across time to form 8 individual response channels. Using these 8 signals we then calculate responses from the three scalp locations: the left hemisphere is the average of electrodes 1 and 2 (O1), the right hemisphere is the average of electrodes 7 and 8 (O2) and the central signal is calculated by averaging electrodes 3, 4, 5 and 6 (Oz) (Figure 4).

Open in a separate window

Figure 4.

Arrangement of the electrodes and calculation of left, center and right array responses.

The first major positive component of the full-field VEP response, P1, has been shown to be delayed or absent in patients with MS or ON [^{3, 22, 23}]. We measure the latency and amplitude of P1 components of individual eyes at three different scalp locations (O1, Oz, and O2). We also calculate the interocular and interhemispheric latency difference and amplitude ratios. Waveform and amplitude abnormalities also have been considered biomarkers for MS and ON [²⁴].

Full-Field Classification and Visualization

Figure 5 shows an example of a full field report. The data are presented separately for each eye and for three separate scalp locations. Data from the left hemisphere is the average of electrodes 1 and 2 (O1), the central location is the average of electrodes 3, 4, 5, and 6 (Oz), and the data from the right hemisphere is the average of electrodes 7 and 8.

Open in a separate window

Figure 5.

Full-field test reports of a representative normal subject from NeuroVEP Device.

6 distinct parameters have been investigated for the ffVEP signals: latency, amplitude, interocular latency difference, interocular amplitude ratio, interhemispheric latency difference, and interhemispheric amplitude ratio. Each of these parameters may indicate dysfunctions in the prechiasmatic, chiasmatic, or postchiasmatic regions of the visual field pathway.

Latency-based biomarkers are less sensitive to retinal and ocular diseases and more reliable in examining visual pathways compared to amplitude-based ones. However, in the absence of other retinal diseases, latency abnormalities generally signify demyelination whereas amplitude abnormalities are a sign of axonal loss. [¹⁹].

A monocular delayed latency suggests a dysfunction in the optic nerve on one side. Conversely, if there is a bilateral abnormality in latency, it indicates a dysfunction in the visual pathway on both sides, but determining whether it is located in pre or post -chiasmatic regions would require additional evaluation of amplitude and topographic (visual field mapping) features. [²⁵]

Amplitude-based metrics of P1 are much more sensitive to ocular and retinal disease. Patient factors such as poor fixation, loss of focus, tearing, inattention, or drowsiness can all cause a decrease in mid-occipital amplitude. However, after ruling out these factors, a monocular amplitude abnormality suggests a dysfunction in the visual pathway of one eye before the optic chiasm. A bilateral abnormality indicates bilateral disease, but more detailed analysis of topographic features or responses to partial field stimulation is necessary to localize the specific site affected beyond the post-chiasmal region. Testing one or both eyes may reveal lateral occipital amplitude asymmetries (interhemispheric amplitude ratio abnormalities). In the absence of P1 latency or interocular amplitude abnormalities, such asymmetries are not necessarily of clinical importance. However, in the presence of the mentioned abnormalities, they may be the sole indication of dysfunction in the chiasmal or post-chiasmal visual pathway. Typically, such abnormalities are present in both eyes. If the asymmetry is unilateral, it may suggest a partial dysfunction before the optic chiasm.

To evaluate all parameters, each laboratory or device must establish its own unique normative values [¹⁷]. We tested a control group of 13 normal subjects (26 eyes) with a mean age of 33±7 years to establish classification criteria for all parameters. Based on the following criteria, each parameter is classified as normal (green), borderline (yellow), or abnormal (red), where std refers to its standard deviation: for all latency-based metrics, including interocular and interhemispheric latency differences, a value below “mean + 2 std” was marked as normal, between 2 to 3 std from the mean was marked as borderline, and above 3 std away from the mean was marked as abnormal. For all amplitude values, a natural logarithm of one plus the distribution (ln(amp_i+1)) was first calculated to get a more normal distribution, then similarly, any nonexistent (or negative amplitude) P1 or values below “mean − 3 std” were marked as abnormal, between “mean − 3 std” and “mean − 2 std” were marked as borderline and above “mean − 2std” were marked as normal. This procedure has been recommended by American Clinical Neurophysiology Society Guidelines on Visual Evoked Potentials [²⁵]. The main ffVEP statistics for the control subject pool are summarized in Table 1.

For all amplitude ratios, based on the distributions, the values were classified based on thresholds. In all cases, the ratios were calculated as the larger value to the smaller value, and then ratios falling above 2.5 were marked as abnormal, between 2 to 2.5 were marked as borderline, and less than 2 were classified as normal.

Signal Processing of Multi-focal Neuro-visual responses

Multifocal data analysis is more complicated. The steps involved in the data processing of mfVEP data are shown in Figure 6. Similarly to ffVEP, the mfVEP data is filtered for bandwidth reduction (3–13Hz) and the same outlier rejection algorithm mark trials for exclusion from response averages. Next, individual sector responses are extracted using the m-sequence technique [²⁶].

Open in a separate window

Figure 6.

Multi-focal Stimulus Data Analysis Steps.

We take advantage of recording from 8 individual sensors to increase the signal quality. Given the variations in the folding of the brain’s visual cortex among individuals, it has been shown that recording from multiple locations and then combining the responses can significantly improve the SNR [^27–29]. Figure 7.A. shows how derived signals D1, D2, D3, and D4 are computed from the original 8 channels. Based on our experience and recommendations from previous studies, these four combinations provided the highest SNR signals [^{27, 30}]. Figure 7.B. Shows the method of calculating the SNR values. Our selection of response window and noise window is similar to what was done in the previous studies [³¹]. However, our calculation of the SNR and noise window criteria is slightly different. The main difference is that instead of using the average of noise windows of all sectors for each subject as the noise, we use the noise region of each individual sector of each subject for the calculation of the SNR.

Open in a separate window

Figure 7.

A. Multi-focal test electrode arrangement and calculation of derived signals. B. Calculation of SNR for mfVEP responses: Signal window [45:150] ms and Noise window [325:430] ms.

Finally, we compute the optimized response for each sector in the following manner: the derived signal responses are polarity matched to D2 and then are averaged together using their SNR as a weighting factor.

Multi-Focal Data Classification

We have developed a machine learning framework for classifying and quantifying the responses. We have also created unique color maps to visualize mfVEP results.

Machine Learning Data Set Curation

For training the supervised Machine Learning Algorithm (MLA), we started by manually selecting and labeling the most apparent sector responses from 7 different experiments (14 eyes) as “normal” and “abnormal” - a binary classification task. The selected experiments included 6 cases of healthy subjects with artificial defects introduced to the stimulus plus 1 case of MS subjects.

The artificial defects were designed by partially or fully masking individual mfVEP sectors with black patches directly using the paradigm presentation software; we hypothesize that this masking would adequately simulate the effects of loss of vision in those regions. Responses from the masked regions or abnormal responses from the MS subject were labeled as “abnormal” and the healthy responses were labeled as the “normal” category. This initial dataset included 382 samples; however, we established that we could increase the accuracy of the machine learning algorithm by using data augmentation techniques to greatly increase the number of training samples. The training data was augmented to 65,536 (2¹⁶) samples using a randomized nearest-neighbor mixing and rescaling approach, where a random signal is selected and linearly interpolated with its nearest neighbor using a Gaussian random variable mixing factor centered around 0.5 with a standard deviation of 0.25, clipped to range [0,¹]. We employed the KDtree algorithm [³²] from the SciPy’s package for Python [⁹] for efficient nearest-neighbor identification.

Despite sophisticated signal processing methods that do very well in filtering artifacts and alpha waves in some individuals, it has been observed that algorithms fail to avoid false positive or false negative classifications. This issue is not unique to mfVEP – the Humphrey Visual Field test can suffer a similar problem from subject participation errors. Other researchers have tried various methods to deal with these issues for monocular mfVEP test evaluation [^{33, 34}]. Because of the risk of over-fitting, where MLA accuracy during training is much higher than when applied to new tests, the final evaluation of the system must not use data encountered during training. To evaluate the performance of both the MLA and the mfVEP test conducted by the NeuroVEP device, we built a separate dataset based on custom mfVEP stimuli with artificial defects (shown in Figure 8) containing 576 samples from both eyes of 2 healthy volunteers (4 tests each, 16 tests total). In this case, the samples were not selected manually; instead, they were labeled based on whether the sector was masked or not.

Open in a separate window

Figure 8.

Schematic of the stimuli used for creating the “artificial defect” evaluation dataset. In each case, only sectors on one ring were unmasked (stimulating).

Feature Extraction

Feature extraction is a crucial step in developing a high-performance machine learning model. We started by extracting a large database of custom features which could be generally categorized into two types, Waveform-based and Frequency-based features. In the visual field regions where the subjects are unable to see (or just barely perceive) the stimulus, they usually produce responses within the frequency range of alpha waves (8~13 Hz); this effect can be seen in the waveforms – for “abnormal” responses (which are often indistinguishable from noise), the signal and noise windows that are used for SNR calculation look very similar to each other and contain higher amplitude alpha band content, than the same windows of “normal” responses.

Figure 9 shows 2D histograms of the two classes of responses, where the overlaid curves are binned to produce a color-scaled counts dimension.

Open in a separate window

Figure 9.

Overlay of training data A. Normal class vs. B. Abnormal class.

Based on these considerations we kept 12 custom features which could effectively discriminate between the two classes of our machine learning problem. Later, we found that we could further improve the accuracy and performance of our model by introducing 7 more features from the well-known Catch-22 feature set [³⁵]. These features belong to the following categories: Linear autocorrelation (CO_f1ecac and CO_FirstMin_ac), successive differences (MD_hrv_classic_pnn40, SB_BinaryStats_diff_longstretch0, and SB_MotifThree_quantile_hh) and simple temporal statistics (DN_OutlierInclude_p_001_mdrmd and DN_OutlierInclude_n_001_mdrmd). Therefore, for our final machine learning model, we used 19 features, listed in Table 2.

This work was partially supported by the National Institutes of Health CTSI grant UL1TR002544. We would like to thank Dr. Srinivasan Radhakrishnan for fruitful discussions on the data analysis of mfVEP.