Learn more: PMC Disclaimer | PMC Copyright Notice
Development and Validation of a Clinically Based Risk Calculator for the Transdiagnostic Prediction of Psychosis
1
,
2
,
3
Grazia Rutigliano, MD,
1
,
4
Daniel Stahl, PhD,
3
,
5
Cathy Davies, MSc,
1
Ilaria Bonoldi, MD, PhD,
1
,
2
Thomas Reilly, MD,
1
and Philip McGuire, MD, PhD
6
Associated Data
Key Points
Question
Can we improve the detection of individuals at risk of developing psychosis among patients accessing secondary mental health care?
Finding
This clinical register-based cohort study of 91 199 patients receiving a first index diagnosis of nonorganic and nonpsychotic mental disorder developed and externally validated an online individualized risk calculator tool, based on simple predictor variables that are easily and routinely collected in clinical settings, for the transdiagnostic prediction of psychosis in secondary mental health care.
Meaning
This individualized risk calculator tool can be used by clinicians and researchers, facilitating the prediction of psychosis and the subsequent implementation of preventive focused interventions.
Abstract
Importance
The overall effect of At Risk Mental State (ARMS) services for the detection of individuals who will develop psychosis in secondary mental health care is undetermined.
Objective
To measure the proportion of individuals with a first episode of psychosis detected by ARMS services in secondary mental health services, and to develop and externally validate a practical web-based individualized risk calculator tool for the transdiagnostic prediction of psychosis in secondary mental health care.
Design, Setting, and Participants
Clinical register-based cohort study. Patients were drawn from electronic, real-world, real-time clinical records relating to 2008 to 2015 routine secondary mental health care in the South London and the Maudsley National Health Service Foundation Trust. The study included all patients receiving a first index diagnosis of nonorganic and nonpsychotic mental disorder within the South London and the Maudsley National Health Service Foundation Trust in the period between January 1, 2008, and December 31, 2015. Data analysis began on September 1, 2016.
Main Outcomes and Measures
Risk of development of nonorganic International Statistical Classification of Diseases and Related Health Problems, Tenth Revision psychotic disorders.
Results
A total of 91 199 patients receiving a first index diagnosis of nonorganic and nonpsychotic mental disorder within South London and the Maudsley National Health Service Foundation Trust were included in the derivation (n = 33 820) or external validation (n = 54 716) data sets. The mean age was 32.97 years, 50.88% were men, and 61.05% were white race/ethnicity. The mean follow-up was 1588 days. The overall 6-year risk of psychosis in secondary mental health care was 3.02 (95% CI, 2.88-3.15), which is higher than the 6-year risk in the local general population (0.62). Compared with the ARMS designation, all of the International Statistical Classification of Diseases and Related Health Problems, Tenth Revision diagnoses showed a lower risk of psychosis, with the exception of bipolar mood disorders (similar risk) and brief psychotic episodes (higher risk). The ARMS designation accounted only for a small proportion of transitions to psychosis (n = 52 of 1001; 5.19% in the derivation data set), indicating the need for transdiagnostic prediction of psychosis in secondary mental health care. A prognostic risk stratification model based on preselected variables, including index diagnosis, age, sex, age by sex, and race/ethnicity, was developed and externally validated, showing good performance and potential clinical usefulness.
Conclusions and Relevance
This online individualized risk calculator can be of clinical usefulness for the transdiagnostic prediction of psychosis in secondary mental health care. The risk calculator can help to identify those patients at risk of developing psychosis who require an ARMS assessment and specialized care. The use of this calculator may eventually facilitate the implementation of an individualized provision of preventive focused interventions and improve outcomes of first episode psychosis.
This UK register-based cohort study measures the proportion of individuals with a first episode of psychosis detected by At Risk Mental State services in secondary mental health services.
Introduction
Existing treatments for psychotic disorders have little effect on the course of illness within standard care.1,2 Prevention and early intervention may be the only available clinical possibility to alter the course of psychosis.3 Prevention of psychosis has been feasible since the introduction of the At Risk Mental State (ARMS) construct 2 decades ago.4 The ARMS construct has been validated internationally,5,6,7 and it can reliably identify young individuals at specific enhanced risk for the development of psychotic disorders,8 mostly schizophrenia spectrum disorders,9 but not of nonpsychotic disorders,10,11 during the 2 to 3 years after initial assessment.12 Randomized clinical trials have shown that focused interventions, if offered to ARMS individuals, can effectively reduce the risk of future illness.13,14 Owing to these unprecedented potentials, the ARMS construct has gained traction to the point that specialized assessment and treatment is recognized as a key component of secondary mental health services by National Institute for Health and Care Excellence guidelines.15
However, the overall clinical impact of the ARMS on psychosis prevention in secondary mental health care and the value of using the ARMS designation compared with standard mental diagnoses (eg, those defined by the International Statistical Classification of Diseases and Related Health Problems, Tenth Revision [ICD-10] ) is not completely clear. For example, most individuals meeting ARMS criteria would also meet criteria for a secondary diagnosis of comorbid mental disorder, mostly depression or anxiety.16 As a result, some authors have claimed that the ARMS construct is not strictly necessary17 and that psychosis could be predicted (and prevented) within the existing diagnostic categories of common mental disorders.18 Whether we can pragmatically predict psychosis outside the ARMS designation or not remains unclear because, to our knowledge, no studies have ever addressed this issue. Such a gap of knowledge may have clinical implications for the provision of preventive intervention services and policy makers. In fact, some authors suggest that it would be better to detect and treat psychosis as it emerges from common mental disorders rather than promoting new ARMS services.18 To further compound the issue, the overall burden of psychosis risk in secondary mental health care is mostly undetermined, and it is not clear whether the ARMS designation is sufficient to pragmatically detect all individuals who will later develop a first episode of psychosis. At Risk Mental State services usually receive referrals on suspicion of psychosis risk, and their referral depends on the subjective judgement of clinicians. Consequently, it is possible that not all individuals in secondary mental health care who will later develop a first episode of psychosis would eventually be detected by ARMS services.
This study investigates for the first time, to our knowledge, the proportion of first-episode individuals detected by ARMS services in secondary mental health services as well as the transdiagnostic risk of developing psychotic disorders across any ICD-10–defined mental disorder. The primary aim was to develop and validate a clinically based, practical, individualized risk calculator tool to facilitate the transdiagnostic prediction of psychosis in secondary mental health care and increase the proportion of individuals at risk for psychosis detected by ARMS services to improve outcomes of first-episode psychosis.
Methods
Data Source
A clinical register-based cohort was selected through a Clinical Record Interactive Search tool19 (eMethods 1 in the Supplement). The study was registered through http://www.researchregistry.com, study number 1487.
Study Population
All individuals accessing South London and the Maudsley (SLaM) National Health Service Foundation Trust services in the period between January 1, 2008, and December 31, 2015, and who received a first index primary diagnosis of any nonorganic and nonpsychotic mental disorder were initially considered eligible. We then excluded those who developed psychosis in the 3 months immediately following the first index diagnosis. Approval for the study was granted by the Oxfordshire Research Ethics Committee C. Because the data set comprised deidentified data, informed consent was not required.19
Study Measures
The outcome (risk of developing any psychotic disorder), predictors (index diagnosis, age, sex, race/ethnicity, and age by sex interaction), and time to event were automatically extracted using Clinical Record Interactive Search.19 Predictors were preselected on the basis of previous meta-analytical clinical knowledge, as recommended20 (eMethods 2 and eTable 1 in the Supplement for full details).
Statistical Analysis
This clinical register-based cohort study was conducted according to the Reporting of Studies Conducted Using Observational Routinely-Collected Health Data Statement21 (eTable 2 Supplement).
Baseline clinical and sociodemographic characteristics of the sample (including missing data) were described by means and standard deviations for continuous variables, and absolute and relative frequencies for categorical variables. The overall cumulative risk of psychosis onset in SLaM patients was described with the Kaplan-Meier failure function (1-survival)22 and Greenwood 95% confidence intervals23 and was qualitatively compared with the risk of psychosis in the local general population (mean predicted cases across SLaM boroughs, estimated with PsyMaptic [http://www.psymaptic.org/]).24
Two-sided P values less than .05 were considered significant. Model development and validation followed the guidelines of Royston and Altman,25 Steyerberg et al,26 and the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis.27
Model Development
We used Cox proportional hazards multivariable complete-case analyses to evaluate the effects of the preselected predictors (index diagnosis, age, sex, race/ethnicity, and age by sex interaction) on the development of nonorganic ICD-10 psychotic disorders and time to development of psychosis, after checking the proportional hazards assumption.28 Model development was not based on stepwise methods, which are not recommended,25 but on a priori selection of predictors based on previous knowledge, as detailed in the eMethods 2 in the Supplement. Continuous variables were not dichotomized.25 Because of significant sociodemographic differences between the SLaM boroughs,29 we used nonrandom split-sample development and external validation,27 with the Lambeth and Southwark cases in the derivation sample and all other cases in the validation sample. The model with all preselected predictors was first fitted to the derivation data to estimate the optimal regression coefficients. Performance diagnostics of individual predictor variables in the derivation data set were explored with Harrell C index,25 which is similar to the area under the receiver operating characteristic curve. We then generated individual prognostic scores, allowing a prognostic index for risk of psychosis onset to be developed in the derivation data set.30 As a supplementary analysis, we fitted the model after excluding the acute and transient psychotic disorder cases.
External Model Validation
The regression coefficients as estimated in the derivation data set were then applied to each case in the external validation data set to generate the prognostic index in the validation data set. Overall model performance (the distance between the predicted outcome and actual outcome26) was assessed with the Brier score (the average mean squared difference between predicted probabilities and actual outcomes, which also captures calibration and discrimination aspects).26 A lower score indicates higher precision and less bias, but interpretation depends on the incidence of the outcome.26 Overall performance was further investigated with the Royston modification of the Nagelkerke R2 (indexing the proportion of variation explained by the model).31 Calibration (the agreement between observed outcomes and predictions26) was assessed with the regression slope of prognostic index26 (which also captures discrimination and model fit),25 with the regression intercept (calibration in the large,26 estimated as previously detailed32) and with the calibration plot (resampling model calibration with hare function).33 Discrimination (accurate predictions discriminate between those with and those without the outcome26) was addressed with Harrell C index25 and with the discrimination slope (difference in mean of predictions between outcomes).26 Studies indicate that unbiased and precise estimation of performance measures can be achieved with a minimum of 100 events in the external validation data set.34
Potential Clinical Usefulness for Psychosis Prevention
We additionally explored the potential clinical usefulness of the risk calculator as recommended by Vickers et al.35 Performance measures do not tell us whether the risk calculator would do more good than harm if used in clinical practice.35 Net benefit analyses35,36 tackle such limitations by including an exchange rate, a clinical judgment of the relative value of benefits (such as preventing psychosis in secondary mental health care) and harms (such as unnecessary treatment) associated with the predictive model (details on the exchange rate in eMethods 3 in the Supplement). Because definition of the exchange rates is subjective, we additionally plotted net benefit for a range of reasonable exchange rates in a decision curve analysis, as recommended.35 All analyses were conducted in STATA, version 13 (StataCorp) and R, version 3.3.0 (R Programming).
Results
Sociodemographic and Clinical Characteristics of the Sample
Of 92 227 patients receiving a first index diagnosis of nonorganic and nonpsychotic mental disorder within SLaM in the period between 2008 and 2015, 91 199 fulfilled the study inclusion criteria and were included in the derivation or validation data sets, as indicated in Figure 1.
ARMS indicates At Risk Mental State; SLaM, South London and the Maudsley.
Table 1 shows the baseline characteristics of the study population, as well as the derivation and validation data sets. As expected, there were significant sociodemographic differences across the derivation and validation data sets (Table 1), particularly with respect to race/ethnicity and index diagnosis. The mean follow-up was 1588 days (95% CI, 1582-1595), with no differences between the derivation and validation data sets (derivation: mean, 1589; 95% CI, 1579-1599; validation: mean, 1588; 95% CI, 1580-1596). The overall 6-year risk of developing a psychotic disorder was 3.02 (95% CI, 2.88-3.15) and higher than the 6-year risk of psychosis of 0.62 in the local general population (eResults 1 and eFigure 1 in the Supplement). The baseline hazard function is reported in eFigure 2 in the Supplement.
Table 1.
| Variable | No. (%) | Validation vs Derivation | ||||
|---|---|---|---|---|---|---|
| Study Population (n = 91 199)a | Derivation Data Set (n = 33 820) | Validation Data Set (n = 54 716) | Test | P Value | ||
| Age, mean (SD), ya | 32.97 (18.63) | 34.4 (18.92) | 31.98 (18.54) | t = 18.73 | <.001 | |
| Sex | ||||||
| Male | 46 404 (50.88) | 17 303 (51.16) | 27 302 (49.90) | χ2 = 13.37 | <.001 | |
| Female | 44 761 (49.08) | 16 507 (48.81) | 27 398 (50.07) | |||
| Missing | 34 (0.04) | 10 (0.03) | 16 (0.03) | |||
| Race/ethnicity | ||||||
| Black | 14 327 (15.71) | 6879 (20.34) | 7023 (12.84) | χ2 = 50.21 | <.001 | |
| White | 55 679 (61.05) | 18 627 (55.08) | 35 392 (64.68) | |||
| Asian | 3830 (4.20) | 1129 (3.34) | 2608 (4.77) | |||
| Mixed | 3319 (3.64) | 1306 (3.86) | 1957 (3.58) | |||
| Other | 5700 (6.25) | 3466 (10.25) | 2084 (3.81) | |||
| Missing | 8344 (9.15) | 2413 (7.13) | 5652 (10.33) | |||
| Index diagnosis | ||||||
| ARMS | 368 (0.40) | 314 (0.93) | 50 (0.09) | χ2 = 48.20 | <.001 | |
| Acute and transient psychotic disorders | 1370 (1.50) | 553 (1.64) | 725 (1.33) | |||
| Substance use disorders | 14 689 (16.11) | 7149 (21.14) | 6507 (11.89) | |||
| Bipolar mood disorders | 2558 (2.80) | 950 (2.81) | 1526 (2.79) | |||
| Nonbipolar mood disorders | 15 496 (16.99) | 6302 (18.63) | 8841 (16.16) | |||
| Anxiety disorders | 24 770 (27.16) | 8235 (24.35) | 15 960 (29.17) | |||
| Personality disorders | 3562 (3.91) | 1286 (3.80) | 2116 (3.87) | |||
| Developmental disorders | 5192 (5.69) | 1412 (4.18) | 3706 (6.77) | |||
| Childhood/adolescence onset disorders | 13 984 (15.33) | 4200 (12.42) | 9629 (17.60) | |||
| Physiological syndromes | 7053 (7.73) | 2555 (7.55) | 4424 (8.09) | |||
| Mental retardation | 2157 (2.37) | 864 (2.55) | 1232 (2.25) | |||
Abbreviation: ARMS, At Risk Mental State.
Model Development
In the derivation data set, there were 1001 transitions to psychosis (52 of which were observed in the ARMS construct [5.19%], eTable 3 in the Supplement). The multivariable model significantly predicted psychosis onset (likelihood ratio χ2 test, 1767.59; P < .001). Age and male sex were significantly associated with an increased risk of psychosis (Table 2). Across men, risk of psychosis decreased with increasing age (Table 2). Relative to white race/ethnicity, black, Asian, mixed, and other races/ethnicities were associated with an increased risk of developing psychosis (Table 2). Compared with the reference ARMS designation, all of the other ICD-10 mental disorders were associated with a lower risk of developing psychosis, with 2 exceptions (Table 2). Bipolar mood disorders and acute and transient psychotic disorders showed a comparable and higher risk of psychosis than the ARMS construct, respectively (Table 2). Post hoc analyses showed that relative to the ARMS construct, bipolar mood disorders and acute and transient psychotic disorders had a higher risk of developing affective spectrum psychoses (HR, 4.63; 95% CI, 1.66-12.94) and schizophrenia spectrum psychoses (HR, 5.46; 95% CI, 2.74-10.89), respectively. Supplementary analyses using the attenuated psychotic symptoms subgroup of the ARMS as a reference group confirmed the model, showing that the brief limited intermittent psychotic symptoms subgroup was at higher risk of developing psychosis than the attenuated psychotic symptoms subgroup. Model diagnostics using the C index are detailed in Table 2. The model showed very good overall apparent performance (good discrimination, C index, 0.80) and explained approximately 75% of the observed variation (Table 3). The model remained significant after removing the acute and transient psychotic disorder cases (eTable 4 in the Supplement).
Table 2.
| Predictor | Multivariable Model | Harrell Ca | ||
|---|---|---|---|---|
| HR (95% CI) | P Value | Decrement if Removed | Increase if Addedb | |
| Age, y | 1.011 (1.001-1.017) | <.001 | <0.001 | NA |
| Sex | ||||
| Male | 1.764 (1.298-2.399) | <.001 | <0.001 | 0.004 |
| Female | 1 [Reference] | NA | ||
| Age by sex (male) | 0.988 (0.981-0.995) | .001 | 0.001 | <0.001 |
| Race/ethnicity | ||||
| White | 1 [Reference] | NA | 0.03 | 0.11 |
| Black | 2.823 (2.438-3.268) | <.001 | ||
| Asian | 1.671 (1.215-2.298) | .002 | ||
| Mixed | 1.839 (1.276-2.626) | .001 | ||
| Other | 1.504 (1.210-1.869) | <.001 | ||
| Index diagnosis | ||||
| ARMSc | 1 [Reference] | NA | 0.13 | 0.20 |
| Acute and transient psychotic disorders | 2.682 (1.981-3.631) | <.001 | ||
| Substance use disorders | 0.146 (0.105-0.202) | <.001 | ||
| Bipolar mood disorders | 0.839 (0.598-1.178) | .31 | ||
| Nonbipolar mood disorders | 0.152 (0.109-0.210) | <.001 | ||
| Anxiety disorders | 0.107 (0.077-0.148) | <.001 | ||
| Personality disorders | 0.213 (0.141-0.321) | <.001 | ||
| Developmental disorders | 0.031 (0.015-0.064) | <.001 | ||
| Childhood/adolescence onset disorders | 0.039 (0.025-0.061) | <.001 | ||
| Physiological syndromes | 0.085 (0.052-0.137) | <.001 | ||
| Mental retardation | 0.086 (0.049-0.151) | <.001 | ||
Abbreviations: ARMS, At Risk Mental State; HR, hazard ratio; NA, not applicable.
Table 3.
| Performance Measure | Mean (95% CI) | |
|---|---|---|
| Derivation | Validation | |
| Overall | ||
| Briera | 0.03 | 0.02 |
| R2 | 0.75 (0.71-0.79) | 0.72 (0.67-0.76) |
| Discrimination | ||
| Harrell C | 0.80 (0.79-0.82) | 0.79 (0.79-0.81) |
| Discrimination slope | 1.47 (1.41-1.52) | 1.41 (1.35-1.46) |
| Calibration | ||
| Calibration-in-the-large | 0 | 0.01 |
| Calibration slope | 1 | 0.96 (0.91-1.00) |
| Clinical usefulness | ||
| Net benefit at 7.69%b | 0.05 | 0.06 |
Model Validation
In the external validation data set there were 1010 transitions to psychosis (12 of which were observed in the ARMS [1.19%]; eTable 3 in the Supplement), a value that greatly exceeds the minimum of 100 events required for robust external validation.34 The model retained an overall good performance and was able to explain around 72% of the observed heterogeneity (Table 3). Model discrimination was fair to good, with a C index of 0.79 (Table 3). The mean risk of psychosis in the validation data set was lower than in the derivation data set, but there were no major miscalibration issues (Table 3; eFigure 3 in the Supplement).
Potential Clinical Usefulness of the Risk Calculator
At the reference threshold for recommending focused interventions to prevent psychosis, the use of the model was associated with significant net benefits in both the derivation and validation data sets (Table 3). The decision curve estimated in the validation data set (Figure 2) shows that compared with conducting no tests, testing on the basis of the risk calculator is associated with net benefits for a 1% to 50% range of threshold probability (risk of developing psychosis by 5 years; eResults 2 in the Supplement).
Decision curve analysis estimated in the validation data set, showing the potential clinical usefulness of the risk calculator at different threshold probabilities (risk of developing psychosis by 5 years) for focused interventions to prevent psychosis (assessment and care), compared with treating all patients or to treating no patients at all.
An online version of the risk calculator was built to facilitate numeric calculation of the predicted probability of conversion to psychosis in secondary mental health care (http://www.psychosis-risk.net).
Discussion
A total of 91 199 patients receiving a first index diagnosis of nonorganic and nonpsychotic mental disorder within SLaM were included in the study, either in the derivation (33 820) or validation (54 716) data sets, with a mean follow-up of 1588 days. The overall 6-year risk of psychosis in secondary mental health care was 3.02 (95% CI, 2.88-3.15) and was higher than the 6-year risk of psychosis in the local general population (0.62). Compared with the ARMS designation, all of the other ICD-10 diagnoses were associated with a lower risk of psychosis, with 2 exceptions. Bipolar mood disorders and acute and transient psychotic disorders showed a similar and higher risk of psychosis than the ARMS, respectively. The ARMS designation accounted only for a small proportion of transitions to psychosis (n = 52 of 1001; 5.19%), indicating the need for transdiagnostic prediction of psychosis in secondary mental health care. The prognostic risk stratification model based on preselected clinically based variables (index diagnosis, age, sex, age by sex, and race/ethnicity) showed good prognostic accuracy in the derivation data set. The risk calculator was externally validated, confirming good performance and potential clinical usefulness for the transdiagnostic prediction of psychosis in secondary mental health care.
This study has 3 significant clinical implications. First, we confirmed substantial psychosis risk enrichment in individuals accessing secondary mental health care. The 6-year risk of psychosis was 5-fold higher than in the local general population (3.02/0.62 = 4.87) and in individuals accessing primary care,37 highlighting a clear window of opportunity for the transdiagnostic prevention of psychosis within secondary mental health care.38
Second, we have shown that the ARMS designation, in particular its attenuated psychotic symptoms subgroup (footnotes to Table 2), is necessary to predict psychosis in individuals who have never experienced psychotic (eg, brief limited intermittent psychotic symptoms39) symptoms. The ICD-10 categories of substance use disorders, nonbipolar mood disorders, anxiety disorders, personality disorders, developmental disorders, childhood/adolescence onset disorders, physiological syndromes, and mental retardation showed a lower level of psychosis risk. Accordingly, the use of ICD-10 categories of comorbid mental disorders, such as anxiety or depression, is unlikely to be of any clinical usefulness to predict psychosis. The ICD-10 acute and transient psychotic disorders and the brief limited intermittent psychotic symptoms subgroup of the ARMS were both associated with a very high risk of developing psychosis, in particular schizophrenia spectrum psychoses, but only in individuals with remitting symptoms at the time of the index diagnosis.40,41 Similarly, bipolar mood disorders specifically predicted the onset of affective spectrum psychoses.
Third, we have demonstrated that the ARMS designation, although necessary, is not sufficient to intercept the overall burden of psychosis risk in secondary mental health care (eDiscussion 1 in the Supplement). In fact, although Outreach and Support in South London was established in Lambeth and Southwark several years before the start of the current cohort,42 only 314 of 33 820 individuals (0.93%) were receiving Outreach and Support in South London care, accounting for only 5.19% of the total cases of emerging psychosis across the 2 boroughs. More importantly, none of the patients outside of Outreach and Support in South London care had ever been assessed for an ARMS. This seems like a missed clinical opportunity because screening for an ARMS is specifically indicated for individuals “already distressed by mental problems”43 and accessing secondary mental health care,44 to prevent psychosis with indicated interventions,13 and improve outcomes in those who go on to develop psychosis (by reducing the duration of untreated psychosis, admission to hospital, and compulsory treatments45 or unnecessary treatment).46
Building on the aforementioned points, our findings highlight a significant unmet need for transdiagnostic prevention of psychosis in secondary mental health care, which is not addressed by existing ICD-10 categories (that are not specific enough for psychosis prediction) or the ARMS designation (which does not include most individuals at risk for psychosis). To overcome these limitations, this study developed a practical, individualized risk calculator tool for the transdiagnostic prediction of psychosis in secondary mental health care. A well-performing risk calculator was developed from easily collectable clinical and demographic predictor variables (age, sex, age by sex, race/ethnicity, and index ICD-10/ARMS diagnosis). The overall externally validated model was robust and achieved good performance, which is in the range of values for established calculators in use for cancer and cardiovascular, neurological, and endocrine diseases (Table 3 in Fusar-Poli et al5). The risk calculator was implemented online and designed to generate a number representing the probability of transition to psychosis, given a particular profile of input variables. A key advantage of the risk calculator is that it inherently accommodates heterogeneity in profiles of risk factors among high-risk individuals.47,48 At the same time, the risk calculator assumes that individuals have accessed secondary mental health care and that the predictor variables are coded as indicated in our methods (eg, ICD-10 categories for the index diagnosis). Therefore, the risk prediction tool would not be usable in primary care or the general population, or if other diagnostic criteria have been used (eg, DSM).
This tool is therefore most useful to clinicians using the calculator for patients who have accessed secondary mental health services. The online calculator could also be easily integrated into electronic case registers, such as the Clinical Record Interactive Search, to facilitate the automatic and individualized prediction of psychosis. Critically, risk determinations should be communicated to patients by clinicians who can help patients understand the meaning of the risk estimates and provide commensurate treatment recommendations. The decision curve analysis presented in our study can further help clinicians to tailor individualized focused interventions, such as selecting patients to be referred to ARMS services. Focused interventions may include a detailed clinical assessment for psychosis risk (ie, the ARMS assessment) combined with sequential testing,49,50 close-in clinical monitoring for the emergence of psychosis, and psychological treatments recommended to prevent psychosis.
Future studies are needed to refine the focused interventions targeting the high-risk individuals detected by our risk calculator. It is also possible that not all high-risk individuals, even when properly referred and assessed, would eventually meet ARMS criteria. For example, research in high-risk individuals with an index diagnosis of bipolar disorders may help to refine the ARMS construct and its ability to predict the onset of affective psychoses.51 Similarly, the effectiveness of preventive psychological treatments in individuals deemed at risk by our calculator but not meeting ARMS criteria should be further investigated.
Limitations
Limitations of this study are addressed in eDiscussion 2 of the Supplement and primarily involve the use of a clinical case register database that shows high ecological validity but lack of formal validation with research-based criteria. Furthermore, the transportability of our model to different clinical scenarios should be confirmed and refined by external replication studies.
Conclusions
Individuals accessing secondary mental health services are at enhanced risk of developing psychosis compared with the local general population. The use of this novel individualized risk calculator can be of clinical usefulness to improve the transdiagnostic detection of at-risk individuals and prevention of psychosis in secondary mental health care.
Notes
Supplement.
eMethods 1. CRIS
eMethods 2. Details of Study Measures
eTable 1. Primary Index Diagnoses of Non-Organic and Non-Psychotic Mental Disorder
eMethods 3. Exchange Rate and Decision Curve Analysis
eTable 2. The RECORD Statement
eResults 1. Overall Risk of Developing Psychotic Disorders in Secondary Mental Health Care
eResults 2. Net Benefit Analysis
eFigure 1. Cumulative Incidence (Kaplan-Meier Failure Function) For Risk of Development of Psychotic Disorders in 91199 Patients Accessing SLaM During 2008-2015.
eFigure 2 Smoothed Curve of the Baseline Hazard Function in the Derivation Database Using Kernel Density Estimation.
eTable 3. Raw Count of Events (Transition to Psychosis) Observed Over Follow-up, Stratified per Index Diagnosis
eTable 4. Statistics for Individual Predictor Variables in the Multivariable Cox Proportional Hazards Regression Analysis of Risk For Psychosis in the Derivation Dataset, After Removing the Acute and Transient Psychotic Disorder (ATPD) Subgroup From the Index Diagnosis.
eFigure 3. Calibration Plot of the Model Estimated in the Derivation Database With Resampling Methods Using the Hare Function Under the Calibrate Options of the Rms Package in R32 at 6-Year Follow-up.
eDiscussion 1. Clinical Utility For Improving Outcomes of First Episode Psychosis
eDiscussion 2. Strengths and Weaknesses of the Study
eReferences