Capturing Motor Signs in Psychosis: How the New Technologies Can Improve Assessment and Treatment?

Abstract

In Translation

Identifying and assisting individuals across the psychosis spectrum, from those at Clinical High Risk (CHR) to individuals experiencing full psychotic episodes, remission, or relapse, is essential to provide early intervention and monitoring progress. Traditionally, much research and clinical monitoring has relied on structured clinical interviews and a range of symptom domains. More recently, investigators have underscored the importance of motor signs in this space as well.^1,2 This category encompasses a range of different motor abnormalities falling into a large number of domains and categories. In particular, social motor behaviors (eg, nonverbal gestures and interactional synchrony), mechanistically relevant motor signs (eg, hyperkinesia and balance issues), and other motor abnormalities like neurological soft-signs and medication-induced motor side effects, have strong clinical potential for enhancing risk assessment, monitoring, and treatment approaches in psychosis. However, their application in clinical settings remains limited. Recent advancements in technology and methods present a perfect opportunity to use insights from the fields of Human Movement Sciences (a field dedicated to improving theoretical, methodological, and technical understanding of the control and organization of human movement) and Affective Computing (a field dedicated to understanding the processing of human emotions using multimodal signals) to enhance the analysis and assessment of motor abnormalities. This article will demonstrate the clinical relevance of these motor signs across psychosis stages and explain how the adoption of new technologies and interdisciplinary scientific approaches can improve the assessment and treatment for psychotic disorders.

Starting with the more ecological context, social motor behaviors, which include unprompted gestures, posture, facial expressions, and head and body movements during naturalistic conversations, have been analyzed and shown to be valuable in distinguishing individuals with schizophrenia from healthy controls. Research indicates that patients with schizophrenia display significantly fewer prosocial behaviors (eg, nodding, smiling), hand gestures, and displacement behaviors (eg, fumbling), but more flight behaviors (eg, looking away, freezing) than healthy controls.^3–5 Moreover, schizophrenia patients often demonstrate impairments in interactional synchrony,⁶ struggling to align movements in space and time with others, which affects the quality and outcome of their interactions.⁷ Improving assessments of social behaviors will be critical for better identifying and predicting psychotic symptoms. For example, social communication deficits in CHR are associated with an increased risk of conversion to psychosis,⁸ Furthermore, socio-motor coordination deficits in the first episode of psychosis have been recently linked to poorer outcomes after 10 years.⁹ As social dysfunction is a major contributor to disability in schizophrenia, gaining deeper insights into the mechanisms driving these impairments can guide the development of more precise and efficient interventions. For example, therapies that enhance nonverbal communication or interactional synchrony may improve social outcomes and reduce long-term disability. Notably, treatments incorporating body-oriented therapy have shown significant improvements in interactional synchrony for schizophrenia patients during interviews, with therapy outcomes predicting overall progress.¹⁰

In addition to social motor behaviors, distinct motor abnormalities that we termed here as mechanistically relevant motor signs, have been tied to vulnerability for psychosis, as well as disease-driving mechanisms. For example, abnormal involuntary movements, also termed hyperkinesia (eg, repetitive and irregular movements affecting predominantly the upper limb, neck, and facial muscles) as well as hypokinesia (eg, slowing and rigidity) characterize psychotic disorders.¹¹ These motor abnormalities occur independently of medication¹² and are believed to reflect vulnerability and later changes in the frontal-striatal dopamine system.² Hyperkinetic movements were found to distinguish high-risk individuals who later convert to psychosis.¹³ Thus, their assessment is crucial for improving the prediction and identification of psychosis. Indeed, by coding archive of childhood home movies, investigators have reported that dyskinesia is one of the earliest precursors of schizophrenia.¹⁴ Additionally, impairments in balance and equilibrium, associated with cerebellar-thalamic dysfunction, are frequently observed in schizophrenia.¹⁵ This motor coordination deficit in balance has been hypothesized as “cognitive dysmetria,” characterized by difficulties in coordinating mental activity due to cerebellar dysconnectivity.^16,17 While the cerebellum is crucial for motor control, it is also heavily involved in cognitive and affective processing as well.¹⁸ Emerging evidence indicates that cerebellar-motor abnormalities are present even in psychosis-risk populations,¹⁹ suggesting that cerebellar-motor dysfunction may serve as a marker of disease onset and progression in the prodromal phase. Indeed, research has shown that abnormal and delayed premorbid motor development are present in infants who later develop schizophrenia.²⁰ Extensive work has also established the prognostic value of motor abnormalities on clinical outcomes in psychosis.²¹ Furthermore, specific mechanistically relevant motor signs have been found to differentiate schizophrenia patients, from other psychiatric groups such as those with depression, bipolar disorder, or affective psychosis.²² Hence, these motor abnormalities are not only indicative of disease vulnerability and progression but may also aid in distinguishing psychotic disorders from other psychiatric conditions.

Apart from social and mechanistically relevant motor signs, other motor abnormalities hold considerable importance in the detection, monitoring, and intervention of psychosis. One significant example, indicative of nonspecific central nervous system dysfunction, is neurological soft signs (NSS), which encompass integrative sensory, motor coordination, and motor sequencing. Higher levels of NSS have been observed in individuals with schizophrenia compared to controls,²³ in those experiencing first-episode psychosis,²⁴ and in both medicated and naïve-drug patients.²⁵ Elevated rates of NSS have also been reported in individuals at high risk for schizophrenia compared to controls.²⁶ This evidence underscores the importance of improving NSS assessment to better predict psychosis. For instance, the presence of NSS in the general population has been associated with increased odds of adult-onset schizophrenia.²⁷ Additionally, a study demonstrated that NSS could predict increased negative symptoms 12 months later in CHR for psychosis,²⁸ highlighting its potential as a unique biomarker of psychosis and as a reflection of underlying factors driving the illness. Furthermore, NSS may also predict clinical response to specific antipsychotic medications, such as haloperidol²⁹ and risperidone,³⁰ making them valuable for tailoring treatment approaches. However, it is also essential to address extrapyramidal motor abnormalities that can arise as side effects of antipsychotic medication. While the prevalence of these medication-induced motor side effects has decreased with newer antipsychotics, they still limit treatment effectiveness. Monitoring these motor side effects is crucial to improving treatment outcomes, reducing re-hospitalization, and ensuring long-term adherence.

Despite growing awareness of the relevance of motor dysfunction in psychosis research, practical clinical applications have been limited. Challenges include the difficulty of implementing motor and gesture tasks in clinical settings, as well as the diversity and complexity of available methods. For example, assessing social motor signs often involves time-consuming manual annotation and coding. The most commonly used system is the Ethological Coding System for Interviews and allow to code for various types of social behavior such as smile, head nod, hand gestures, crossed arms, or scratching.³¹ Although, extensively used in patient-therapist interviews within clinical psychiatry, it has gradually been replaced by more precise automated capture tools, such as optical motion tracking systems.³² However, these tools remain expensive, demand complicated experimental setup and processing, and involve attaching reflective markers on different joints of the patients, which can be disruptive. Consequently, these limitations reduce the feasibility of using such tools in many clinical settings. Complementary, general gesture production have been extensively assessed with the Test of Upper Limb Apraxia (TULIA).³³ Although the TULIA efficiently assessed imitative and pantomime gestures in schizophrenia individuals,³⁴ it requires multiple gesture tasks and a time-consuming application based on video analyses. As such, it would greatly benefit from advancements in capture and processing technology. Concerning mechanistic-relevant motor signs, standard psychometric scales, such as the Abnormal Involuntary Movement Scale³⁵ or the Modified Rogers Scale,³⁶ face issues of scalability and require extensive clinician training. Additionally, NSS and medication-induced motor side effects have also traditionally been assessed using rating instruments including motor tasks such as the Neurological Evaluation Scale²³ and the Simpson-Angus Scale,³⁷ which can pose challenges for reliability and replication. For the last decades, multiple research groups have successfully measured many motor abnormalities in laboratory settings, offering greater sensitivity, and reduced subjectivity compared to traditional methods, using force sensor³⁸ or balance platform.¹⁵ Despite their advantages, their practical application in clinical research remains limited, primarily due to the setup and processing time, high cost, and the requirement of expert operators.

To complement or, in some cases, replace the traditional assessment described above, recent technological and methodological advancements, particularly in the fields of Human Movement Sciences have introduced valuable tools for motor signs assessments (see Table 1 for an overview). Programs enabling real-time, multi-person detection of human body, hand, and facial key points are now readily available online (eg, OpenPose⁵⁰; Mediapipe⁵¹) and require minimal technical expertise. These open-access tools require only a low-resolution video from basic devices such as webcams, cameras, or smartphones, and can extract 2D or 3D trajectories of bodily key points. These video-based motion tracking methods have been validated by comparing against high-performance wired motion tracking systems to study human movement (eg, Polhemus Liberty, Polhemus, Colchester, VT, USA).⁵² Furthermore, since the onset of the COVID-19 pandemic, a significant number of clinical interviews have taken place online, creating an influx of video data that holds valuable information.

This accessibility has transformed the landscape of motor sign assessment and offers a practical solution for tracking both social, mechanistic signs, and other motor abnormalities in individuals across the psychosis spectrum. For example, nonverbal social behavior can now be automatically captured and analyzed from video recordings, reducing the need for training time, and manual gesture coding. For example, previously video-based used methods like Motion Energy Analysis are even more effective now with the technological advances presented above. Alternatively, motion tracking combined with threshold-based methods can be used to detect the frequency of patients’ spontaneous gestures, as demonstrated on schizophrenia patients.³² Thanks to advances in Affective Computing, machine learning models have also recently enabled real-time recognition and classification of dynamic hand gestures from video streams, which could be applied to analyze patient interviews.³⁹ Although applications in psychosis research are still limited, studies on major depression disorder have used head motor features extracted from face videos to classify depressed patients from non-depressed participants.⁵³ More precisely, the authors extracted a 3D face model from 2D real-word videos and founded specific motor features in depressed patients like slower head movements, less change of head position, or longer duration of looking down. Then, they modeled a machine learning-based classifier with these motor features to automatically classify patients, supporting clinicians in their diagnosis and monitoring of clinical depression. In addition to movement cues, by combining audio features (eg, speech variability, articulation rate, and pause duration) with dynamic visual cues (ie, facial movement and head movement in 3D) and deep-learning-based video coding, it is now possible to automatically estimate depression severity from audio-videotaped clinical interviews.⁵⁴ Furthermore, multi-person interactions can now be analyzed automatically to quantify dynamic interactional synchrony among individuals, such as between hand and hand, or head and head,⁴⁰ as well as speech turn-taking during conversation.⁵⁵ Interpersonal distance between interactive partners can also be automatically extracted from videotaped interactions by detecting body landmarks (eg, head, shoulders, and hands), estimating individuals’ center of mass, and continuously computing the distance between both partners’ center of mass,⁵⁶ which can serve as a measure of rapport and prosocial attitude. However, it is important to note the influence of situational and contextual factors, such as the familiarity between participants or cultural norms governing gesturing and interpersonal distance, that can introduce variability and confound interpretation. Future studies should report these factors to minimize these confounds.

Similarly, automated methods are transforming the assessment of mechanistic-relevant motor signs. For example, a preliminary study have assessed dyskinesia in patients with Parkinson’s disease with video-based pose estimation algorithms applied on communicative tasks.⁴¹ In this study, movement trajectories of patients’ joints were extracted from recording videos of Parkison disease assessments using a machine learning-based body pose estimation program, like presented above. Specific features of body movements (eg, kinematic, frequency) were then used to train classification models to detect and estimate the severity of dyskinesia. In addition, postural control has extensively used balance or force platform, where investigators record the postural sway of a patient standing still or under different conditions (eg, eyes open vs closed).^15,57 As these force platforms do not exist in all clinical centers and laboratories, other alternatives have recently been explored, such as video-based methods. For instance, automatic tracking of linear and angular displacements of head and trunk movements allows to obtain a quantitative measure of postural sway.⁴² Authors also extracted the same features from a more precise 3D optical motion capture system and found agreements on postural sway measures in comparison to their video-based method, emphasizing the potential interest of using low technological capture systems, such as a basic camera, combined with video-based body tracking.

In addition, the assessment of other motor abnormalities, such as NSS, can benefit from recent technological and methodological advancements. For instance, specific movement patterns like overarm throwing have been analyzed in individuals with schizophrenia using a markerless motion capture system, with performances associated with NSS scores.⁴³ Results indicated that less mature movement patterns could serve as a marker for atypical neurological development in schizophrenia, supporting the use of simple gesture recording to enhance NSS assessments. Regarding medication-induced motor side effects, instrumental approaches based on handwriting movements have also been developed for quantification. For example, a handwriting task requiring participants to repeatedly write a sentence on a digitizing tablet enables the extraction of specific kinematic variables (eg, velocity, acceleration, jerk). This approach has revealed significant slowing and dysfluencies in patients with psychosis compared to healthy controls.⁵⁸ Handwriting movement abnormalities were also found to be positively associated with the neuroleptic daily dose. Subsequent studies have replicated these findings using various handwriting tasks, highlighting this instrumental approach as a useful mean of assessing neuroleptic response alongside other motor abnormalities, and facilitating greater translational applicability.^59,60

Furthermore, emerging movement technologies can also be used to explore additional domains and areas of interest in motor functioning. Wearable sensing has been particularly popular, employing accelerometers, gyroscopes, or magnetometers to record movements. These sensors, often packaged together as inertial movement units or partially substituted with smartphone-based systems, allow for versatile applications. For instance, motor coordination has been assessed by instructing participants to walk in a straight line for 12 feet heel-to-toe, with a smartphone in their pocket recording acceleration, rotation, and gravity data.⁴⁴ Wearable sensors, used in actigraphy, can continuously record movement activity over an extended period, providing quantitative and objective data on psychomotor activity (eg, velocity of movement, acceleration) and user mobility (eg, location frequency, distance traveled). For example, a study where patients with psychosis disorders wore a triaxial accelerometer on their hip for 5 days found that movement disorders were associated with reduced physical activity and increased sedentary behavior, both major contributors to mental health issues.⁴⁸ Continuous recording facilitates the collection of longitudinal data, offering insights into how motor functioning changes over weeks and months. These applications have significant implications for translational research. This can aid in evaluating the progression of motor signs, as well as medication-induced motor side effects,⁴⁶ studying specific patterns of motor behaviors relevant across psychopathology,⁶¹ and monitoring the impact of intervention,⁶² such as clinical progression in brain stimulation therapy.⁶³ Actigraphy also enables remote monitoring, which is particularly beneficial for patients who may have difficulties traveling to clinics for regular assessments. For instance, smartphone technology capturing passive data (eg, geolocation, acceleration, and screen state) and active data (eg, surveys) over a period of several months has been used to examine the feasibility of predicting relapse.⁶⁴ Similarly, remote measurement of motor functioning has been tested by analyzing the head movements captured through the front-facing camera of smartphones in patients with schizophrenia, significantly predicting the motor retardation item of the Positive and Negative Syndrome Scale (PANSS).⁶⁵ While digital health technologies still require robust validation, they offer a promising supplement to traditional assessment tools.⁶⁶ Moreover, these tools raise some challenges concerning the specificity or direction of findings, as they often encompass several overlapping constructs; these issues speak to the need for future studies to consider reporting the full array of associations between clinical and motor symptoms.^49,67 In addition, while the major advantage of these automatic tools is their superior ability to process large quantities of data efficiently, the field sorely needs translational studies as well as thorough analysis of their clinical and economic impact.⁶⁸

The integration of motor sign assessment into psychosis research holds immense translational potential, bridging the gap between laboratory findings and clinical practice. By leveraging recent technological advancements, such as automated video-based motion analysis, wearable sensors, and machine learning-based classifiers, we can enhance the precision and scalability of motor evaluations across diverse clinical settings. These innovations enable the continuous and unobtrusive monitoring of social, lower-level mechanistic motor signs, and various motor abnormalities, providing valuable insights into the trajectory of psychosis and the impact of interventions. Additionally, the objective data obtained from these tools can inform personalized treatment approaches, aligning with the goals of precision medicine. By tailoring technological devices to incorporate psychological and motor mechanisms, assessments and interventions can target both behavioral and emotional components, leading to more comprehensive treatment strategies.^69,70 As we continue to validate these technologies and integrate them into routine care, the comprehensive assessment of motor signs will become a key point of early intervention, ongoing monitoring, and improved outcomes for individuals across the psychosis spectrum.

Conflicts of interest

The authors declare they have no conflict of interest.

Funding

This work was supported by National Institute of Mental Health (grant number R01 MH120088 and R01 MH134369).

References