Heterogeneity and Classification of Recent Onset Psychosis and Depression: A Multimodal Machine Learning Approach

Abstract

Diagnostic heterogeneity within and across psychotic and affective disorders challenges accurate treatment selection, particularly in the early stages. Delineation of shared and distinct illness features at the phenotypic and brain levels may inform the development of more precise differential diagnostic tools. We aimed to identify prototypes of depression and psychosis to investigate their heterogeneity, with common, comorbid transdiagnostic symptoms. Analyzing clinical/neurocognitive and grey matter volume (GMV) data from the PRONIA database, we generated prototypic models of recent-onset depression (ROD) vs. recent-onset psychosis (ROP) by training support-vector machines to separate patients with ROD from patients with ROP, who were selected for absent comorbid features (pure groups). Then, models were applied to patients with comorbidity, ie, ROP with depressive symptoms (ROP+D) and ROD participants with sub-threshold psychosis-like features (ROD+P), to measure their positions within the affective-psychotic continuum. All models were independently validated in a replication sample. Comorbid patients were positioned between pure groups, with ROP+D patients being more frequently classified as ROD compared to pure ROP patients (clinical/neurocognitive model: χ² = 14.874; P < .001; GMV model: χ² = 4.933; P = .026). ROD+P patient classification did not differ from ROD (clinical/neurocognitive model: χ² = 1.956; P = 0.162; GMV model: χ² = 0.005; P = .943). Clinical/neurocognitive and neuroanatomical models demonstrated separability of prototypic depression from psychosis. The shift of comorbid patients toward the depression prototype, observed at the clinical and biological levels, suggests that psychosis with affective comorbidity aligns more strongly to depressive rather than psychotic disease processes. Future studies should assess how these quantitative measures of comorbidity predict outcomes and individual responses to stratified therapeutic interventions.

Introduction

Treatments for mental illness are currently based on categorical structures built on patterns of syndromes and their course, rather than etiology.¹ The biological and clinical overlaps between these syndromes, and significant heterogeneity in outcomes, has become more apparent in recent years.^2–4 Advancement in both pharmacological and psychotherapeutic interventions has stalled, potentially as a result of continued focus on invalid disease categories.^5,6 The need for better treatments is particularly acute in psychosis and depression, which constitute major mental health challenges to the world’s population.^7–13 The legacy of a Jaspers based hierarchical approach to symptom structures suggests that positive psychotic symptoms are of primary importance within psychotic spectrum disorders.¹⁴ Yet the categorical and hierarchical division of psychotic disorders into affective and non-affective has been contested for decades,¹⁵ with a clear demonstration of the presence of affective symptoms in psychosis and psychotic symptoms in affective disorders.^8,11,16

Heterogeneity is particularly noticeable in early and developing stages of illness, with a high prevalence of affective symptoms across disorders.¹⁷ The comorbidity of depression in early psychosis has been largely regarded as secondary to the primary disorder (psychosis), reinforcing a categorical, hierarchical approach.^18,19 There are increasing calls to use empirical evidence to develop alternative aetiologically informed structures.²⁰ The use of multidimensional item response modeling to predict psychosis biotypes has been shown to transcend traditional diagnostic boundaries; with the suggestion of an underlying transdiagnostic dimension across psychotic diagnoses.^21–23 However, there are valid reasons why a categorical approach to mental illness has persisted; a significant number of individuals’ presentation will “fit” within distinct categories of mental illness and the course and outcome of their treatment can be predicted from such diagnostic structures.²⁴ When disorders are fully formed, course, as well as symptom structure, enables clearer distinction between categories. Imaging studies also show shared areas of interest, including the hippocampus and cerebellum,²⁵ the prefrontal cortex and insula²⁶ and both depression and psychosis have been associated with heightened brain activation in regions central to emotional processing^26–28 with similarities particularly prominent in the early stages of illness.^26–29

The distinction between depression and schizophrenia is possible by structural brain data, but also, more challenging in the early stages of illness when symptoms and course are more heterogeneous.²⁹ Transdiagnostic processes of mental health disorders are descriptively transdiagnostic (ie, being present in multiple disorders, without regard to how or why) or mechanistically transdiagnostic (ie, reflecting neurobiological, physiological, or functional mechanisms).^30,31 Both depression and psychosis are associated with transdiagnostic features of working memory, executive functioning, and verbal fluency deficits.^32,33 The importance of certain mechanistically transdiagnostic symptoms is potentially hidden in categorical structures, and they remain under-investigated.³⁴

Complex psychopathology and heterogeneity in developing mental health disorders present the opportunity of fuller exploration of the significance of potential transdiagnostic symptoms, to provide further insight into aetiopathogenetic pathways of symptoms and through this to advance diagnostic structures.^35,36 However, novel approaches and powerful statistical tools such as machine learning techniques could help provide this deeper understanding by detecting complex patterns of data across diagnostic structures, and the delineation of shared and distinct features of these illnesses at the phenotypic and brain levels. This may inform the development of more precise differential diagnostic tools and improve the development of new treatments.³⁵

This study aimed to identify prototypes of pure depression and pure psychosis to investigate the heterogeneity of depression and psychosis with common, comorbid transdiagnostic symptoms. We hypothesized that developed models would correctly classify diagnostic groups without comorbid symptoms, in keeping with evidence of the utility in categorical diagnostic structure, and that grey matter volume (GMV) would add classification accuracy. We further hypothesized that a reduction of classification accuracy would be seen in groups with comorbid symptoms. Exploration and the delineation of shared and distinct features at both the phenotypic and biological levels may potentially inform future development of more precise treatments.

Materials and Methods

Study Design

Data were taken from the discovery and replication samples of the PRONIA study, an EU-FP7 funded seven-center study aiming to optimize candidate biomarkers for the prediction and staging of mental health disorders. Details of the PRONIA study sites, recruitment protocol and quality control procedures are described in a previous publication³⁷ and in supplementary methods (1.1, 1.2, 1.3, tables S2 and S3).

Inclusion and Exclusion Criteria

The general inclusion criteria for the study were: (1) age between 15 and 40 years, (2) sufficient language skills for participation, (3) capacity to provide informed consent/assent. General exclusion criteria were: (1) an IQ below 70, (2) current or past head trauma with loss of consciousness (>5 min), (3) current or past known neurological or somatic disorders potentially affecting structure or functioning of the brain, (4) current or past alcohol dependence, (5) polysubstance dependence within the past six months, and (6) any medical indication against MRI. ROP and ROD inclusion criteria can be found in supplementary section 1.1.

Group Identification

Pure ROP: any ROP patient who had a Beck Depression Inventory-II (BDI-II) score of 13 or lower, which is indicative of absent or minimal depressive symptoms.^38,39 Pure ROD: any ROD patient who had a Positive and Negative Symptom Scale (PANSS)⁴⁰ positive subscale score of no more than 7 and no Structured Interview of Psychosis-risk Syndromes positive (SIPS-P) severity score of 3 or more on any item.

ROP with depressive symptoms (ROP+D): any ROP participant with a BDI-II score of 14 or more. ROD with psychotic symptoms (ROD+P): any ROD participant with a SIPS positive item score of 2 or more and a Schizophrenia Proneness Instrument-Adult version (SPI-A) Cognitive Disturbances (COGDIS) item score of 3 or more.

Twenty-four ROP and 31 ROD patients from the discovery sample and 21 ROP and 53 ROD patients were not included in the analysis due to not meeting group identification criteria or not having neuroimaging data.

MRI Imaging Data Acquisition, Quality Control, and Preprocessing

Participants underwent a multi-modal MRI protocol. A minimal harmonization protocol, which the MR sequences across the different scanners had to comply with is described in the supplementary methods (1.3). In the current study, T1-weighted structural MRI (sMRI) images of the participants were analyzed. The sMRI images of six healthy traveling volunteers who were scanned at all sites with same parameters were also analyzed as part of a calibration study. The images were processed using the open-source CAT12 toolbox (version r1155; http://dbm.neuro.uni.jena.de/cat12/) (see supplementary methods 1.4). Employing generalization theory,^41,42 a between-site voxel reliability map (G coefficient map) was computed from the analysis of the GMV maps that were derived from the calibration study. During our neuroimaging-based machine learning analyses the G coefficient maps were used for reliability-based voxel masking.

Classification Models

Using the pure ROP and ROD groups, a Support Vector Machine (SVM) classification model was built using individual item scores from broad clinical and neurocognitive tests that assess features commonly occurring in psychosis and depression including anhedonia, social functioning and cognition deficits (see supplementary methods 1.8). In total, 151 features were included in the model. The trained pure classification model was then applied to the comorbid ROP+D and ROD+P groups to determine the classification accuracy to their primary diagnosis.

A second model using GMV whole-brain voxel-wise data as features was developed in the pure ROD and ROP groups and then applied to the ROP+D and ROD+ P groups. The developed clinical/cognitive and GMV models were combined by using decision values from the pure clinical and pure GMV models (to build a model that learns from the meta-data) in a stacking-based data fusion framework.^43,44 Finally, all the models were applied to an independent replication sample.

Support Vector Machine Learning Analysis

The machine learning analysis of pre-processed data (see supplementary methods 1.5) was performed using NeuroMiner (version 1.0; https://github.com/neurominer-git). A repeated nested pooled cross-validation (CV) was used with 10 outer CV2 permutations, 10 outer CV2 folds, 10 inner CV1 permutations, and 10 inner CV1 folds.

Imbalanced learning was corrected for, by increasing the C value in the minority class by multiplying it by the inverse ratio of the training class sizes. A linear kernel was used with 11 C values (0.0156, 0.0312, 0.0625, 0.1250, 0.2500, 0.5000, 1, 2, 4, 8, and 16) to optimize the choice of C value and create an ensemble of predictive models to be applied to the CV₂ data to produce a single average robust prediction.

Balanced accuracy (BAC) regularized by SVM model complexity was used as a criterion for the hyperparameter optimization (see supplementary methods 1.6).

Finally, a stacking-based data fusion framework^43–46 was used to examine whether the combination of the clinical/neurocognitive and the neuroimaging-based models would provide a superior classification accuracy (see supplementary Methods 1.7).

Independent Validation

All our models were validated in our independent replication sample (N = 262) (see supplementary methods table S4) which was collected at a different timescale. The same group identification criteria were applied.

Supplementary Analyses

A number of supplementary exploratory analyses, including correlation analyses between decision scores from our models, association and comparison analyses between correctly and misclassified patients, GMV comparison between groups, decision score group comparisons, and regression analyses with 9-month functional outcomes were conducted and can be found in supplementary material (Section 2.6).

Results

Demographic Information

Data from 154 participants with ROP and 146 patients with ROD were included in the analysis as our Discovery sample. Thirty-eight ROP patients were included in the pure ROP group and 90 ROD patients in the pure ROD group. The mean age of the pure ROP group was 26.5 [SD 6.8]) and the mean age of the pure ROD group was 26.5 [SD 6.6]). There were 25 male and 13 female patients in the pure ROP group and 45 male and 45 female patients in the pure ROD group.

Ninety-two ROP subjects were included in the ROP+D group and 25 ROD subjects in the ROD+P group. The mean age of the ROP+D group was 26.5 [SD 6.8] and the mean age of the ROD+P group was 23.8 [SD 3.9]. There were 57 males and 35 females in the ROP+D group and 12 males and 13 females in the ROD+P group. A summary of demographic information is provided in table 1.

The independent validation sample consisted of 161 patients with ROP and 131 patients with ROD. Fifty ROP patients were included in the pure ROP group and 53 ROD patients in the pure ROD group. The ROP+D and ROD+P groups consisted of 90 and 25 patients, respectively. A full description is provided in supplementary table S4.

Machine Learning Analyses

Internal Validation of the Pure Group Differential Classifiers

Clinical and Neurocognitive Data.

A repeated nested pooled cross-validation model with classifiers of clinical and cognitive variables predicted pure diagnostic groups with a balanced accuracy (BAC) of 79.3%; 95% CI [77.2, 82.3] and an area under the curve (AUC) of 0.86 (table 2 and figure 1A). Assignment to the ROP category by the clinical classifier was driven by reduced scores in the RSA and elevated scores in the WSS, RSA, and SPIA. ROD group classification was informed by increased scores in the SPIA, WSS, together with reduced scores in the DSST. The contribution of the features was calculated by feature weights and by cross-validation ratio (figure 2).

GMV Data.

The repeated nested pooled cross-validation model using sMRI to predict diagnostic group in pure ROP and ROD produced a BAC of 62.5%; 95% CI [58.8, 64.0] and an AUC of 0.70 (table 2 and figure 1B). ROP patients showed pronounced reductions in the thalamus and the cerebellum, whereas depressed patients showed orbitofrontal, limbic and paralimbic volume reductions (figure 3).

Stacking.

Combining the outputs of the clinical predictors and sMRI using stacked generalization predicted diagnostic group with a BAC of 79.5%; 95% CI [77, 81.9] and an area under the curve (AUC) of 0.87 (table 2 and figure 1C).

Separability of Comorbid Groups.

Clinical/Neurocognitive Data

The trained pure classification system comprising the collection of 11 clinical/neurocognitive models generated by the repeated nested cross-validation scheme on pure groups was then applied to the comorbid groups (ROP+D and ROD+P) to produce decision scores measuring ROP vs. ROD likeness. This model had a BAC of 62.5% and an AUC of 0.66. Misclassifications showed a directionality toward the ROD group, with 63% of ROP+D patients being classified as ROD; Z = 1.276, P = 0.0385) (see table 2 and figure 1A). ROP+D patients were more frequently classified as ROD compared to pure ROP patients (χ² = 14.874; P < 0.001). In contrast, the assignment precision of ROD+P and ROD patients did not differ (χ² = 1.956; P = 0.162).

GMV Data.

The trained pure classification system (comprising of 11 GMV models generated by the repeated nested cross-validation scheme on pure groups) was then applied to the comorbid groups (ROP+D and ROD+P) to produce decision scores measuring ROP vs. ROD likeness. This produced a BAC of 47.8% and AUC of 0.43. Misclassifications showed a directionality toward the ROD group, with 80.4% of ROP+D patients being classified as ROD; Z = 0.713, P = 0.344) (see table 2 and figure 1B). Similarly to the clinical/neurocognitive model ROP+D patients were more frequently classified as ROD compared to pure ROP patients (χ² = 4.933; P = .026). In contrast, the assignment precision of ROD+P and ROD patients did not differ (χ² = 0.005; P = .943).

Stacking.

When applied to the comorbid groups, the combined model predicted a diagnostic group with a BAC of 58.5% and an area under the curve (AUC) of 0.66 (see table 1 and figure 1C).

Independent Validation.

Application of our models to the independent validation sample replicated findings very well (pure clinical and neurocognitive model BAC 76.2; pure imaging model BAC 49.9; pure stacking model BAC 78.2). Full results from the independent validation analysis can be found in supplementary table S5.

Supplementary Analysis.

See supplementary file S2 for additional exploratory analysis results.

Discussion

Using repeated nested cross-validation techniques we built classification models based on transdiagnostic clinical and neurocognitive features and GMV data, together with a combined model integrating all data modalities, to classify prototype diagnostic groups of ROD and ROP participants without comorbidity. Eighty-seven percent of patients with pure ROP and ROD were accurately ascribed to their diagnostic group. Applying this model to groups with comorbidity, 88% of patients with ROD and psychotic features were ascribed to their primary diagnostic group (depression) whereas only 37% of patients with ROP and depressive features were ascribed to their primary diagnostic group (psychosis). The shift of comorbid psychosis patients towards the depression prototype was observed both at the clinical and biological levels. This suggests that when comorbid with affective symptoms, psychoses align more strongly to the disease processes of depressive than psychotic disorders. Using GMV measures only for classification, comorbid groups ROP patients with depressive symptoms largely resembled pure ROD. Results were generalizable to our independent validation sample.

Our findings suggest that clinical and neurocognitive transdiagnostic symptoms may have differential weight within psychosis and depression presentation with and without comorbidity. In pure groups, these symptoms could be seen as under the hierarchical umbrella of psychosis or depression, and may accurately reflect underlying pathology in these groups. However, in the face of comorbidity, transdiagnostic symptoms lean more to the depression domain. Given the prevalence and importance of depressive comorbidity in early psychosis this may support a model where depression could be more intrinsically important than is currently considered in early phases of illness.^4,47 When participants with ROP exhibited even mild depressive symptoms, their GMV classification was more likely to lie within the depression group than psychosis suggesting a potential depressive biological phenotype that exists in the psychosis spectrum. The implications of these findings suggest that understanding heterogeneity of brain structures may need to include a specific focus on symptoms that may often be masked by more acute (eg, positive) symptoms and simple solutions that some symptoms may be transdiagnostic, potentially belie the complexity of individual etiology and psychopathology. Furthermore, these findings indicate a need to rethink current diagnostic classification to better reflect the biological reality and eventually develop better treatment options.

A classical diagnostic hierarchy in the structure of personal illness, reflected in current nosological classification systems, posits that mental disorders of the primary diagnosis have more weight and primacy over symptoms from lower classes, which are seen as secondary or comorbid.^18,19 We found that comorbid symptoms affect diagnostic structures in different ways. Sub-clinical psychotic-like symptoms appeared to not alter the signature of ROD patients whilst depressive symptoms had a profound effect on ROP patients’ classification accuracy.

Our findings add evidence to the debate around the validity of the system on which the DSM is built upon,⁴⁸ suggesting that a descriptivist position in the diagnosis of mental disorder is not sufficient and that a novel multivariate approach of mental disorder is more appropriate. Comorbidity in psychiatric disorders presents a clinical as well as nosological challenge. There may be a significant interplay between sets or clusters of symptoms over the development of disorder from prodrome via onset to potential chronicity. The frequency of depression within ROP may be a primary driver, rather than being a secondary symptom. If identified correctly, novel symptoms may be new targets that if treated effectively, ameliorate the other, eg, positive symptoms.

Recent onset disorders may constitute groups of phenotypically highly individual symptoms with underlying etiopathology, and it may be that personalized treatments could be tailored accordingly. Our SVM classification model found that different types of anhedonia (social and physical) were important in the classification of both psychosis and depression. Anhedonia has been suggested as a possible biomarker for depression⁴⁹ and has been found to be associated with decreased activation in ventral basal ganglia areas, the dorsal anterior cingulate, middle frontal gyrus, and medial frontal gyrus both in schizophrenia and depression.⁵⁰ Our GMV model revealed that orbitofrontal areas were higher weighted in the classification to the ROD group.

The relationship between psychotic and affective symptoms has been central to the dilemma of psychiatric classification. Substantial clinical and genomic evidence shows that schizophrenia and affective disorders may be distributed across a dimensional spectrum.⁵¹ However, the dimensional spectra model does not allow for either the clinical reality of a complex and changing symptom profile, nor the investigation of clinical features commonly seen across all disorders; some of which may be of primary importance. Concerning the neurobiology of schizophrenia and depression, the majority of previous studies are based in subjects with depression or psychosis, but only rarely in both^29,52 and not previously in highly mixed recent-onset comorbid disorders. Our results suggest that while GMV showed some distinction in prototype (pure) groups, when presented with complex comorbid groups, which may be the majority in clinical practice, there was a significant lack of any point of rarity between disorders. This builds on previous work suggesting distinction is more challenging, but also that neuroimaging based data do not support categorical classification in recent-onset disorders.²⁹ In recent onset disorders, early cognitive processes related to depression can both drive other more severe symptoms and/or be seen in isolation: for example, anhedonia could be an early indication of negative symptom clusters or a core feature of co-morbid depression.⁵³

Strengths and Limitations

The strengths of the present analysis include sufficiently large data, robust collection of clinical and imaging data from both depression and psychosis groups, independent validation analysis together with a novel approach to a challenging and essential clinically relevant research question, which speaks to the validity of diagnoses as the cornerstone of psychiatric practice. Our results however should be interpreted with certain caution due to the limitations of the study. Regarding our definition of the ROD+P group, we used a SIPS-P item score of 2 or more which is not a marker of formal psychotic symptoms, and thus would only measure low levels of psychotic-like symptoms. However, to supplement this, we used an SPI-A COGDIS item score of 3 or more. We did not include core symptomatology measures such as the PANSS and the BDI in our features because primary groups are defined with these measures, and therefore including them would risk a circular analysis. Finally, there were more subjects identified in the pure ROD and comorbid ROP groups; ideally, groups of equal size would have been used. Nevertheless, we addressed this imbalance in our analysis, by increasing the C value in the minority class by multiplying it by the inverse ratio of the training class sizes.

Conclusions

Findings from this large, multi-modal, replicated machine learning classification study in recent-onset disorders suggest that whilst there may be a small subset of prototypically pure individuals with clear categorical disorder, the majority of patients share a number of transdiagnostic features, primarily from the depression domain. The brain structure of psychosis patients with co-morbid depressive symptoms largely resembles that of depression. The increasing interest in heterogeneity of early disorders and transdiagnostic symptoms as novel treatment targets needs to be fully informed of potential depression related co-morbidity. Our analysis in recent-onset groups also highlights that both categorical and transdiagnostic approaches may ultimately fail at an individual patient level, as neither recognize the possibility of pluripotent pathways that are both stage and context-dependent. The implications, particularly for early intervention and prevention in mental health disorders is that ultimately a personalized medicine approach, encompassing the full potential of comorbidity, may be necessary to improve outcomes. Future studies should investigate the utility of targeting such transdiagnostic depression features to elucidate their prognostic value, and develop new treatments.

Acknowledgments

R.U. reports grants from Medical Research Council, grants from National Institute for Health Research: Health Technology Assessment, grants from European Commission - Research: The Seventh Framework Programme, and personal speaker fees from Sunovion, outside the submitted work. C.P. has participated on Advisory Boards for Janssen-Cilag, Astra-Zeneca, Lundbeck, and Servier. He has received honoraria for talks presented at educational meetings organised by Astra-Zeneca, Janssen-Cilag, Eli-Lilly, Pfizer, Lundbeck and Shire. All other authors report no conflicts of interest.

Funding

PRONIA is a Collaboration Project funded by the European Union under the 7th Framework Programme under grant agreement number 602152.

References

Author notes