A standardized immune phenotyping and automated data analysis platform for multicenter biomarker studies

Cryopreservation and cell handling affect reproducible phenotyping of selected cell populations.

In contrast to the excellent center-to-center reproducibility for the 4 panels described above, there was considerable variability (CVs >20% and ICCs <0.75) in several T cell subsets quantified in the T cell activation (T-ACT) and memory/regulatory T cell (T-MEM-REG) panels (Figure 2). Closer examination revealed that populations with high CVs and low ICCs were primarily limited to those defined by CD45RA, with particularly high CVs and low ICCs for subsets defined by both CD45RA and CD62L.

Open in a separate window

Figure 2

Inter-center comparison of populations quantified in cryopreserved PBMCs from healthy subjects with the T-ACT and T-MEM-REG panels.

Replicate aliquots of cryopreserved PBMCs from 5 subjects were analyzed at 5 different sites using the indicated DuraClone panels. Raw data were analyzed centrally, and the reproducibility of population proportions was determined by statistical analysis. All calculations are based on the cell proportion relative to the parent gate; the identity of parent gates for all variables is listed in Supplemental Table 1. Shown are CV and ICC values for each population. CV and ICC values indicative of poor reproducibly (i.e., CV >20% and/or ICC <0.75) are shaded in gray. 45RA, CD45RA.

To further explore the source of variation in these T cell populations, we compared data obtained from cryopreserved PBMCs with those obtained from analysis of unfractionated blood obtained from the same blood draw. We first compared the results between blood and PBMCs for populations with good reproducibility within replicate PBMC samples (CV <20%) and found that they were highly comparable, with no significant differences, except in the case of CD4⁺CD25⁺⁺CD127^lo Tregs (Figure 3A), which were significantly reduced in all PBMC samples (P = 0.003). However, as this reduction was highly consistent, it did not lead to decreased reproducibility in the inter-center comparison.

Open in a separate window

Figure 3

Effects of cryopreservation on standardization.

Peripheral blood from healthy subjects was either analyzed immediately or processed into PBMCs and cryopreserved for later analysis with the same DuraClone panels. Raw data were analyzed manually at one center. Data are from 3 subjects analyzed at 3 different sites (n = 9); unfractionated blood was only analyzed at the collection site. The identity of the parent gates is shown in Supplemental Table 1. Representative populations as measured in blood versus PBMCs with (A) CV <20% or (B) CV >20% as shown in Figures 1 and ​and2.2. *P < 0.05, **P < 0.01, multiple t test with FDR adjustment according to Benjamini, Hochberg, and Yekutiel. (C) Representative data from 1 individual for naive/memory CD4⁺ T cell proportions detected in blood or in replicate samples of cryopreserved PBMCs analyzed at 3 different centers (sites 1–3). (D) The proportion of CD45RA⁺ Tregs (of total Tregs) in 3 different individuals was measured in unfractionated blood or PBMCs. **P < 0.01, 2-way repeated-measures ANOVA with Šidák’s multiple comparison test. (E) Representative data from Tregs quantified in blood or replicate samples of cryopreserved PBMCs analyzed at 3 different centers. CS mem B, class-switched memory B cells; mono, monocytes.

We then compared populations with poor reproducibility (i.e., CV >20%), focusing on those that were scarce (CD4⁺γδ T cells, BDCA3⁺ mDCs, CD4⁺CD28^– T cells), defined by poorly resolved gates (monocyte and NK cell subtypes) or by cryopreservation-sensitive markers (CD45RA, CD62L) (14, 21–23) (Figure 3B). Of all of the populations with CVs >20%, only those defined by the cryopreservation-sensitive markers CD45RA and/or CD62L were significantly different when data were obtained from blood versus PBMCs. Cryopreservation did not significantly impact the levels of monocytes, NK cells, CD4⁺ γδ T cells, or BDCA3⁺ mDCs.

To further explore the poor reproducibility of populations expressing CD45RA and/or CD62L, we carried out a more detailed examination of cryopreserved PBMC data from the T-MEM-REG panel. Figure 3C shows a comparison of data obtained from blood versus PBMCs for a single individual’s CD4⁺ T cells defined by expression of CD45RA and CD62L, or CCR7 and CD62L. While PBMCs analyzed at centers 1 and 3 retained the expected level of CD45RA expression, this signal was completely lost at center 2. A similar result was found for CD62L. The effect is quantified in Figure 3D using the example of CD45RA⁺ Tregs.

The poor reproducibility in detection of CD45RA- and/or CD62L-expressing cells could be due to variable loss of protein expression and/or loss of the cell type after cryopreservation. As the loss of the total Treg proportions (defined as CD25^hiCD127^lo cells per CD4⁺ T cells) was not center-dependent (Figure 3E), variation in detection of cryopreservation-sensitive markers may be driven by center-specific cell handling that results in variable loss of these markers.

Optimized timing for analysis of unfractionated blood.

With the observation that certain populations (e.g., Tregs) and markers (e.g., CD45RA and CD62L) are strongly affected by cryopreservation, we next carried out experiments to test the range of time over which blood can be reproducibly analyzed. Previous studies determined that analysis within 4–6 hours of blood collection was optimal (12, 24), but this may not always be feasible for multi-center studies electing to ship samples to a central site. We compared data obtained from blood that was stained and acquired within 4 hours with data obtained from blood that was immediately stained but then acquired after 24 hours, or blood that was left unmanipulated for 24 hours and then stained and acquired. The results showed that, for most panels (Basic, TCR, T-ACT, T-MEM-REG), leaving the blood unmanipulated for 24 hours and then staining and acquiring was preferable to immediate staining followed by acquisition 24 hours later, as this protocol resulted in data that were most similar to those from samples stained and acquired within 4 hours. (Supplemental Figure 2A). For the B cell and DC panels, there was no advantage to holding blood for 24 hours before staining versus immediate staining and later analysis. In fact, plasmablasts were undetectable in samples incubated 24 hours before staining, whereas samples that were stained prior to 24-hour incubation and acquisition retained levels similar to those found in their freshly analyzed counterparts (Supplemental Figure 2, B and C).

Application of standardized flow cytometry to post-HSCT patient samples.

Studies of standardized flow cytometry often use samples from healthy individuals (12, 14, 19, 24), yet the utility of this method is with clinical samples, often coming from patients with abnormal leukocyte counts and/or proportions. To test the applicability of our standardized protocols on clinical samples, we carried out inter-center comparisons using cryopreserved PBMCs from patients 100 days after HSCT. Figure 4 shows CV and ICC values derived from replicate analysis of PBMCs from 5 patients at 3 different sites. While the CVs tended to be higher (average increase, 4.2%), the overall data were largely comparable to those from healthy control samples (Figures 1 and ​and2),2), with the same problematic populations (e.g., monocyte subtypes, rare cells, and/or those defined by cryosensitive markers) having CVs >20%. A notable exception was the B cell panel, which showed poor reproducibility for all subpopulations due to the expected low B cell reconstitution at this time after transplant (25) and thus the low absolute numbers of B cells. Specifically, populations with CVs >20% were those with less than 0.4% of total PBMCs for the healthy subjects and less than 2.5% of total PBMCs for the post-HSCT subjects (Figure 5). A notable exception to this rule was observed with some monocyte subsets in healthy samples: although they were ~1% of PBMCs, their CVs were >20%. The same monocyte subsets had CVs >50% in post-HSCT samples.

Open in a separate window

Figure 4

Inter-center comparison of cryopreserved PBMC data from post-HSCT subjects.

Replicate aliquots of cryopreserved PBMCs from 5 subjects were analyzed at 3 different sites. Raw data were analyzed centrally, and the reproducibility of population proportions was determined by statistical analysis. All calculations were based on the cell proportion relative to the input gate; the identity of parent gates for all variables is listed in Supplemental Table 1. Shown are CV and ICC values for each population. CV and ICC values indicative of poor reproducibility (i.e., CV >20% and/or ICC <0.75) are shaded in gray. 45RA, CD45RA.

Open in a separate window

Figure 5

The effect of population size on CVs.

CVs were plotted against the log of median population sizes from all inter-site comparisons. All population sizes are expressed as percent of CD45⁺ PBMCs. Solid and dotted lines indicate CVs of 10% and 20%, respectively. Gray box indicates populations that are less than 2.5% of PBMCs. Red symbols indicate monocyte subtypes. Populations shown to be affected by cryopreservation (defined by CD45RA or CD62L) were excluded.

We next compared cell population frequencies in healthy versus post-HSCT subjects. Focusing on blood (Figure 6A), significant differences in population proportions were found in the Basic, B cell, and DC panels. Specifically, in comparison to healthy controls, HSCT subjects had lower proportions of lymphocytes, B cells, and BDCA3⁺ mDCs, but higher proportions of monocytes and CD56⁺⁺ NK cells. B cell numbers were often too low to accurately define B cell subpopulations; only 5 of 11 samples met the criterion of >1,000 events in the CD19⁺ gate. Despite the small sample size, we found that HSCT subjects had significantly increased proportions of transitional B cells and plasmablasts, but decreased IgM memory and marginal zone B cells, consistent with the well-documented immature B cell phenotype in this patient population (26). Although T cell populations in blood from healthy controls and 3-months-post-HSCT patients are known to differ in terms of absolute cell counts (25), none of the T cell population proportions examined showed significant differences.

Open in a separate window

Figure 6

Detecting differences between 2 cohorts using unmanipulated blood versus cryopreserved PBMCs.

Populations from all panels that significantly differed between healthy controls and patients 100 ± 20 days after HSCT in either (A) whole blood or (B) cryopreserved PBMCs. Note: Post-HSCT, but not healthy control, PBMCs were incubated 24 hours before cryopreservation. Proportions of parent gates for each population are shown as box-and-whisker plots; midline is the median, box is the interquartile range, and whiskers show minimum and maximum values. Means are indicated by a thick black band. See Supplemental Table 1 for a list of parent gates. Only samples that were evaluable in both unmanipulated blood and PBMCs were compared; n = 9 healthy and n = 8 post-HSCT subjects (except n = 4 for B cell–derived populations). *P < 0.05, **P < 0.01, ***P < 0.001; unpaired t test with Holm-Šidák corrections.

When findings in blood were compared with data obtained from cryopreserved PBMCs, we found that the changes in lymphocytes, CD56⁺⁺ NK cells, and most B cell populations remained consistent (Figure 6B). Exceptions included plasmablasts, which, as described above, did not survive a 24-hour delay in isolation of PBMCs, and a low-abundance BDCA3⁺ DC population. On the other hand, monocyte subsets such as CD14⁺CD16⁺ cells showed significant differences in the PBMCs that were not present in whole blood.

Development of automated pipelines to analyze standardized data.

Previous attempts to develop standardized flow cytometry methods found that the most significant source of variation is introduced during manual analysis (14, 19). To minimize this effect, data are often analyzed by a single operator, but this is time-consuming and difficult to implement across multiple independent studies, and results remain subjective. To streamline the workflow, we developed an automated analysis workflow based on flowCore (27) and flowDensity (5). flowDensity was previously shown to outperform unsupervised algorithms in a FlowCAP (Flow Cytometry: Critical Assessment of Population Identification Methods) consortium study, with its performance matched by only one other supervised approach (28). We developed flowDensity automated pipelines customized for the 2 panels currently available as off-the-shelf products (Basic and B cell). Automated analysis aimed to replicate manual gate placement, with pipelines developed using data from the unfractionated blood samples of healthy adults and then further refined on unfractionated blood from the post-HSCT subjects. The proportions of key cell populations were determined as before (Supplemental Figure 3 and Supplemental Table 1).

The performance of automated pipelines was assessed by their ability to match on a per-event basis values obtained by an expert manual analyzer (i.e., the reference manual), currently considered the “gold standard” approach. In addition, we compared the reference manual values with those obtained by 2 additional manual analyzers who followed an identical gating strategy. Data were compared using Spearman’s correlation as well as the F₁ score to measure the performance of automated analysis. F₁ is the harmonic mean of precision and recall, with a score of 1 indicating that all individual events were placed in the same series of gates by both analysis methods. The Basic panel was highly amenable to automated gating, as most cell populations were defined by clear boundaries. Accordingly, manual and automated analyses produced results that were highly correlated with those obtained in the reference manual analysis (Spearman’s rank correlation coefficient [ρ, r_s] >0.8) (Figure 7A). Most median F₁ scores were >0.9 (Figure 7B), with an overall F₁ average of 0.93.

Open in a separate window

Figure 7

Generation of automated data analysis pipelines for the DuraClone IM Basic panel.

The proportions of cell populations indicated in Supplemental Table 1 for the Basic panel were determined in unfractionated blood from healthy (n = 9) and HSCT (n = 11) subjects using automated gating, 2 different manual analyzers (Manual 1, Manual 2), or a reference manual analyzer. Data from Manual 1 or 2, or automated gating were compared with those from the reference manual to determine (A) Spearman’s correlation coefficients (r_s) or (B) F₁ scores (harmonization of precision and recall, maximum value 1.0) ordered by population size (highest on the left). Box-and-whisker plot: midline is the median, box is the IQR, and the whiskers extend to 1.5 times the IQR. (C) Representative automated or manually gated data from 1 healthy subject and correlation graphs for (C) CD14⁺ total monocytes, (D) CD14⁺CD16⁺ monocytes, and (E) CD56⁺⁺ NK cells. White circles indicate Manual 1, and gray squares automated, both plotted against the reference manual data.

As an example, comparison of automated with corresponding manual reference values for total monocytes revealed a high correlation (r_s >0.99) (Figure 7C). For a manual versus manual comparison, an alternative manual (Manual 1) plotted against the reference manual (r_s >0.99) is shown in the same graph. The precision and recall of automated versus manual placement of events within the same gates was also excellent for this population, as shown by the F₁ score of 0.99.

We next examined correlations between automated and manual gating for populations with lower F₁ scores. Compared with the other population, nonclassical (CD14⁺CD16⁺) monocytes (29, 30) had particularly low r_s values and F₁ scores: both 0.83, driven by indistinct boundaries between CD14⁺ and CD14⁺⁺ (particularly in the CD16⁺ population), as well as between CD16^– and CD16⁺ (Figure 7D). These unclear boundaries led to variability in manual gating (Manual 1 and Manual 2 vs. reference manual, both r_s = 0.83), as well in automated versus reference manual (r_s = 0.83). For simplicity, only Manual 1 versus reference manual is shown. The effect of unclear boundaries was most apparent for low-abundance monocyte subtypes; the larger CD14⁺⁺CD16^– population had higher correlation values and F₁ scores, of 0.90 and 0.97, respectively.

In contrast to monocytes, CD56⁺⁺ NK cells, which are also rare and defined by a poorly resolved marker (CD56⁺⁺), were reproducibly detected by both automated and manual gating (r_s = 0.98, F₁ = 0.88) (Figure 7E). In this case a second marker (coincidentally also CD16) helped to clearly define the (largely CD16^–) CD56⁺⁺ population.

We also developed an automated pipeline for the B cell panel, which quantifies several low-abundance populations defined by markers with indistinct boundaries such as CD24, CD27, and CD38 (see the gating strategy in Supplemental Figure 1E). Although the correlation values were typically lower than for the Basic panel, all but one (IgD^–CD27^–) had an r_s value >0.8 (Figure 8A), and most mean F₁ scores were >0.8 (Figure 8B), with an overall average of 0.79. Plasmablasts, which are typically <1% of total B cells and are defined by high expression of multiple markers (CD27, CD24, and CD38), showed one of the lowest correlation values (r_s = 0.81) and an exceedingly low F₁ score (0.37). Although automated analysis accurately identified the single sample with abnormally high plasmablast proportions (Figure 8C, red box), subtler differences may be lost. Moreover, there was good correlation between automated and manual gating for class-switched memory cells (r_s >0.85), which are defined by the same markers and gates as are the plasmablasts but represent a greater proportion of the population. Similarly, despite the low r_s value for IgD^–CD27^– cells (r_s = 0.65), the one sample with high proportions was identified by both manual and automated analysis (Figure 8D, red boxes). The typically more abundant naive B cells (IgD⁺CD27^–), defined by the same gates, had r_s >0.9. Interestingly, the F₁ score for the IgD^–CD27^– population was fairly high (0.81), despite the low correlation score. As Spearman’s correlation assesses similarity by rank rather than absolute value, samples that have many similar values (such as the IgD^–CD27^- proportions) may have lower r_s. Nevertheless, the high F₁ score shows that the majority of the cells were being ordered into the same gates in manual and automated analyses.

Open in a separate window

Figure 8

Generation of automated data analysis pipelines for the DuraClone B cell panel.

The proportions of cell populations indicated in Supplemental Table 1 for the B cell panel were determined in unfractionated blood from healthy (n = 9) and HSCT (n = 5) subjects using automated gating, 2 different manual analyzers (Manual 1, Manual 2), or a reference manual analyzer. Data with event counts under threshold values for key gates were not included (see Methods, Flow cytometry data analyses). Data from Manual 1 or 2, or automated gating were compared with those from the reference manual to determine (A) Spearman’s correlation coefficients (R_S) or (B) F₁ scores. Box-and-whisker plot: midline is the median, box is the IQR, and the whiskers extend to 1.5 times the IQR. (C and D) Correlation graphs and representative manual and automated plots and gates for (C) plasmablasts and class-switched (CS) memory B cells, both derived from the CD27 versus CD38 plot, pre-gated on IgM^–IgD^– B cells; or (D) naive and IgD^–CD27^– B cells. Red boxes indicate data points and manual or automated gates from which they were derived for the samples with outlying proportions of plasmablasts or IgD^–CD27^– cells.

Evaluation of data from healthy versus HSCT subjects using automated pipelines.

Flow cytometry is often used to find biomarkers that discriminate between disease and control groups. We thus next compared the ability of automated versus manual analyses to identify significant differences between healthy and HSCT subjects (Table 2). Automated and manual analyses identified the same populations as being significantly different between the 2 groups with one exception: the marginal zone–enriched B cell population (P = 0.06). Half of the P values were less significant after automated than after manual gating — especially in the case of B cells, for which the sample numbers were very low (n = 5). Conversely, automated but not manual analysis found that CD14⁺⁺CD16⁺ and CD64⁺⁺CD16⁺ monocytes were significantly increased in HSCT patients. However, for CD14⁺⁺CD16⁺ monocytes, both data sets showed significant differences when tested alone (Supplemental Figure 4; manual P = 0.0042, automated P = 0.0012).

Table 2

Cell population differences in healthy versus HSCT subjects as determined by manual or automated gating

Open in a separate window

Validation of automated pipelines using an independent data set.

Standardized flow cytometry should facilitate direct comparisons of data collected at different centers and/or from different cohorts. To assess whether this was the case for our approach, we obtained an independent data set from the ONE Study, which used the same antibody panel and fluorescence intensity settings (12), and reanalyzed these data manually and with our automated pipelines. Remarkably, we found that the correlations between manual and automated results were similar to those from the data sets used for development of the automated pipelines. When all populations analyzed were combined for the ONE Study or CNTRP data, correlation values between automated and manual gating were all >0.9 (Figure 9; Supplemental Table 2 shows the individual r_s values). Thus, automated gating pipelines developed with one set of data can readily be used to accurately analyze independent data if they are collected using the same standardized methodology.

Open in a separate window

Figure 9

Validation of automated gating pipelines in an independent dataset.

Blood from healthy subjects enrolled in the ONE (n = 9) or CNTRP (n = 20 for Basic, n = 10–14 for B cells) study was analyzed with the Basic or B cell panels, and data were analyzed manually or using automated pipelines developed using the CNTRP data. All population proportions listed in Supplemental Table 1 for the Basic and B cell panels were measured, and results from automated versus manual analyses were compared. Shown are correlation graphs merging all data from either the CNTRP (training set) or ONE (validation set) study quantified by automated versus manual gating in relation to input gate. Each population proportion is a different color, and each data point represents an individual. Spearman’s correlation coefficient is shown at the upper left. See Supplemental Table 2 for individual r_s values.

Design of inter-center experiments.

For cross-center comparisons using blood from healthy subjects, blood was collected, processed into PBMCs, and cryopreserved in aliquots of 10⁶ cells at the 3 research centers listed above (3 blood draws per site, 9 samples in total). Unfractionated blood from the same collection was stained and acquired within 4 hours to enable direct blood versus cryopreserved PBMC comparisons. Two additional centers stained PBMCs only (Toronto General Research Institute and CancerCare Manitoba). Frozen aliquots of PBMCs were shipped on dry ice between all participating centers, so that each had at least 1 aliquot from the same 5 blood draws.

For the cross-center comparisons using blood from post-HSCT patients, 1 ml unfractionated blood was stained at one site (Vancouver) within 4 hours of collection. The remainder (~45 ml) was incubated for 24 hours at room temperature (RT) to emulate real-world conditions in which samples are shipped to a central site for processing. PBMCs were isolated 1 day after collection, and aliquots of 10⁷ per vial were frozen as described above. Replicate aliquots from 5 different patients were shipped on dry ice from Vancouver to Edmonton and Montreal to enable a 3-center comparison of post-HSCT PBMCs.

Staining of unfractionated blood using DuraClone dry antibody cocktail tubes.

All flow cytometry reagents were obtained from Beckman Coulter unless otherwise stated. Six ONE Study DuraClone panels were used; see Table 1 for a list of markers in each panel and Supplemental Table 3 for a list of clones. 100 μl anticoagulated blood was added into each of the following tubes: Basic, TCR, T-ACT, T-MEM-REG, and DC tubes. For the B cell tube, 300 μl blood was first washed twice with PBS to remove plasma (and the soluble IgM therein); after resuspending the washed cells in a total volume of 300 μl PBS, 100 μl was added to the B cell DuraClone tube. Liquid antibodies were added to the DC tubes as drop-ins: BDCA3-FITC and BDCA2-APC (Miltenyi; catalog 130-090-513 and 130-090-905, respectively). Tubes were vortexed for 10 seconds to resuspend dried antibodies with the cell sample and incubated for 15 minutes at RT in the dark. Two milliliters VersaLyse with 2.5% IOTest 3 Fixative was added to each tube, immediately vortexed for 10 seconds, and incubated for a further 15 minutes in the dark at RT. Cells were washed in cold IFN buffer (IsoFlow + 2% heat-inactivated FBS [NorthBio Inc.] + 0.1% NaN₃) and centrifuged for 5 minutes at 300 g and 4°C. This wash step was repeated, and cells were resuspended in 300 μl cold IFN buffer, transferred to 1.2 ml FACs tube inserts (VWR International), and stored at 4°C until acquired (within 12 hours).

Isolation, cryopreservation, and thawing of PBMCs.

See the supplemental material for detailed SOPs. Briefly, all reagents and centrifuges were used at RT until the final cryopreservation step. Blood was diluted with 1 volume of PBS and mixed by inversion, and 25 ml was layered over 15 ml Lymphoprep in a SepMate tube (STEMCELL Technologies) following the manufacturer’s instructions. PBMCs were counted by trypan blue staining or with an automated counter, Cellometer Auto 2000 (Nexcelom Bioscience), after staining with acridine orange/propidium iodide (AO/PI) (catalog CS2-0106) according to the manufacturer’s instructions using the setting “Immune Cells, low RBCs.”

After counting, cell pellets were resuspended in freezing medium (10% DMSO in FBS) at final concentration of 10 × 10⁶ cells/ml at RT. Frozen aliquots of PBMCs were shipped to other centers on dry ice. Cryopreserved PBMCs were thawed at 37°C for 1 minute and transferred to 10 ml thawing solution (1.5 ml FBS, 250 μl 1 M HEPES, 50 μl 7.5 % sodium bicarbonate in 10 ml RPMI; all reagents were from Life Technologies) and centrifuged for 10 minutes at RT at 453 g. After resuspension in thawing solution, cells were counted and resuspended in PBS + 2% FBS to a final concentration of 10⁷ cells/ml. The average viability of thawed PBMCs from healthy and post-HSCT subjects was 94% (range 85%–100%) and 80% (range 52%–94%), respectively.

Staining of PBMCs using DuraClone dry antibody cocktail tubes.

See supplemental material for detailed SOPs. Briefly, 10⁶ PBMCs in 100μl of PBS + 2%FBS were added to each DuraClone tube; the drop-in antibodies for BDCA-2 and -3 were added as described above for unfractionated blood staining. Tubes were vortexed for 10 seconds and incubated for 15 minutes in the dark. To remove unbound antibodies prior to fixing, 3 ml FP (2% FBS in PBS) buffer was added to each tube, cells were spun down at 453 g for 5 minutes, and the supernatant was discarded. Pellets were resuspended in 1 ml of 1× IOTest 3 Fixative solution for 15 minutes in the dark. Cells were then washed with 3 ml cold IFN buffer and centrifuged, and pellets were resuspended in 300 μl cold IFN buffer until acquisition.

Instrument standardization (Navios).

All flow cytometry data for the multi-site studies were acquired on 10-color/3-laser Navios flow cytometers (Beckman Coulter). Flow-Check Pro beads were run for daily quality control, and the manufacturer’s criteria were followed to assess whether the instrument was in good working order. To produce consistent fluorescence outputs between the CNTRP Navios instruments and those used in the ONE Study, Flow-Set Pro beads were used to establish the voltages required for each detector on each Navios. For daily quality control, Flow-Set Pro beads were run daily at these voltages, and the instrument was considered to yield stable fluorescence outputs if the MFIs of the beads were ±10% of the initially established MFIs. Voltage recalibration to lot-specific FSP standard intensities was either done using the Navios-instrinsic Autosetup algorithm or manually (for phase I, healthy control samples, Autosetup was used; for phase II, post-HSCT patient samples, the manual method was used). Both procedures are described in detail in the supplemental material (SOP sections 9 and 11). Autosetup was also used to calculate a 9-color compensation matrix using VersaComp antibody capture beads stained with single CD4 antibodies conjugated to each of the 9 fluorochromes. The same initial compensation matrix was used for each of the 6 panels due to the standardized nature of the fluorochromes and instrument settings. Technical staff at each center were trained to perform daily hardware quality control, sample preparation of cells with DuraClone tubes, and data acquisition, based on a written SOP and onsite training (see supplemental material). Effectiveness of training was evaluated by acquisition of a normal blood sample and comparison of data with the central site.

Comparative analyses on the Navios and the LSR-Fortessa.

Some second-generation DuraClone panels incorporated a 10th parameter (FL4/PECy5.5), so this study optimized fluorescence intensities for 10 colors. Due to differences in dynamic range and emission filters, the 2 cytometers could not be calibrated by absolute fluorescence intensities, so stain index–based calibration was used. Briefly, PBMCs stained with single anti-CD4 mAbs conjugated to one of 10 fluorochromes were acquired on the Navios, and stain indices were determined (47, 48). The same samples were then run at 30-V increments on the Fortessa from 230 to 800 V. Stain indices for each fluorochrome at each voltage were determined, and the voltage resulting in the stain index most similar to that on Navios was selected. Blood from 5 healthy volunteers was stained using the DuraClone Basic and B cell tubes (Beckman Coulter, catalog {"type":"entrez-nucleotide","attrs":{"text":"B53309","term_id":"2607643","term_text":"B53309"}}B53309 and {"type":"entrez-nucleotide","attrs":{"text":"B53318","term_id":"2607652","term_text":"B53318"}}B53318, respectively), divided into 2 and run on the same day on the Fortessa and Navios.

Flow cytometry data analyses.

For inter-center comparisons, raw data (LMD files) were sent to a central site for analysis. Compensation for each panel was created from the center-specific acquisition matrix and adjusted for individual panels at the time of central analysis. Samples were gated as previously described (12); an example of gating for each panel is shown in Supplemental Figure 1. All values are provided as a proportion of parent gate, and a list of parent gates for each variable is shown in Supplemental Table 1. Flow cytometry data were manually analyzed using with FlowJo software v. 10.2. or Kaluza Analysis Software v.1.5a (Beckman Coulter).

Threshold event numbers for flow cytometry analysis: For B cells, samples with >1000 events in the CD19⁺ gate were evaluated for subpopulation proportions; samples with >100 events in the IgD^–IgM^– gate were further subdivided into class-switched memory B cells and plasmablasts. Only one sample, just over the cutoff for total B cells by manual gating but under the cutoff for automated gating, was removed to have an equal number of samples for comparison. In accordance with MIFlowCyt (49), all LMD and FCS3.0 files were uploaded to FlowRepository, experiment ID FR-FCM-ZYQT.

Development and evaluation of automated gating pipelines.

The flowDensity algorithm uses a supervised, sequential bivariate clustering approach to generate a set of predefined cell populations using customized cut-offs defined by density distributions for each marker. FlowDensity is freely available from Bioconductor (https://www.bioconductor.org). The customized code used in this study can be downloaded from GitHub (https://github.com/mehrnoushmalek/DuraClone-gating; commit a8f440) for academic use; see license file for full details). Parameters for flowDensity were set globally, customized at every step of the gating hierarchy to replicate the manual approach. Three manual operators independently analyzed all data using the gating strategies outlined in Supplemental Figure 1; one operator was designated as the “gold standard” reference manual and used for the comparison to automated gating. Data similarity was assessed by correlation or F₁ scores (see Statistics).

For the analysis of ONE Study data with the automated pipelines, raw data from 10 healthy subjects and manual analyses thereof were provided by the ONE Study group. Two CNTRP operators reanalyzed the raw data. Automated versus manual as well as manual versus manual correlations were done as described above. One sample (HC43) was excluded due to a high level of unexplained background staining.

For analyses of data acquired on the LSR-Fortessa, the pre-processing component of the pipeline was adjusted due to differences in the amount of forward scatter–low event collection, and the singlet gating was changed from FS-A versus FS-W (Navios method) to FS-A versus FS-H (Fortessa method). For the Basic panel, which expresses some populations as a percentage of total PBMCs, exported FCS files containing only the CD45⁺ subpopulations were used for the automated analysis.

Statistics.

For inter-center variation, CV estimates were obtained by taking the median of the subject-specific CVs. ICCs were obtained by first fitting a variance components model with random effects for site, donor, and residual error. The ICC was calculated by the ratio of biological variability (individual effect) to the sum of the 2 technical variability components (center, residual error). Paired blood and PBMC data were compared using a multiple t test with FDR adjustment by the method of Benjamini, Hochberg, and Yekutiel. Comparisons of staining methods were based on the absolute magnitude of differences (log scale) between the 24-hour delay methods and the optimal method (immediate staining and acquisition) by applying the Wilcoxon’s rank-sum test to cell population values for each panel as the unit of analysis.

Data from healthy or 100-days-post-HSCT subjects were used to compare the results obtained with unfractionated blood versus cryopreserved PBMCs using multiple t tests with Holm-Šidák correction for multiplicity. A P value less than 0.05 was considered significant. To compare data generated by manual or automated analyses, Spearman’s ρ (degree of linear correlation with the reference manual) and F₁ scores were calculated. The F₁ score (harmonic mean of precision and recall or sensitivity) provides a value in the range of 0 to 1 for each population, with 1 indicating a perfect reproduction of a manually gated population by automated gating. Analyses were performed using GraphPad Prism for MacOSX version 7.0 b and R version 3.3.3 (50). For all tests, *P < 0.05, **P < 0.01, and ***P < 0.001.

We gratefully acknowledge the valuable contribution of Salima Janmohamed-Anastasakis (employee of Beckman Coulter), whose technical expertise and help with study design, site training, and manual data analyses were integral to the success of this project. We thank Kaija Strautins for support in figure formatting. This work was supported by the Canadian Institutes of Health Research (CIHR) through the Canadian National Transplant Research Program (TFU 127880 to MKL, DAW, JSD, and LJW), the National Institute of Allergy and Infectious Diseases of the NIH (NIH 1R01GM118417-01A1), and the Natural Sciences and Engineering Research Council (NSERC) (MM, RRB). MKL receives a Scientist Salary Award from the BC Children’s Hospital Research Institute. LJW holds a Tier 1 Canadian Research Chair and is supported by the Gordon English family through the Stollery Children’s Hospital Foundation and Women and Children’s Health Research. JSD is supported by a career award from the Fonds de recherche du Québec-Santé (FRQS).