Validity assessment in Eastern Europe: cross-validation of the Dot Counting Test and MODEMM against the TOMM-1 and Rey-15 in a Romanian mixed clinical sample

Abstract

Objective

This study investigated performance validity in the understudied Romanian clinical population by exploring classification accuracies of the Dot Counting Test (DCT) and the first Romanian performance validity test (PVT) (Memory of Objects and Digits and Evaluation of Memory Malingering/MODEMM) in a heterogeneous clinical sample.

Methods

We evaluated 54 outpatients (26 females; M_Age = 62.02; SD_Age = 12.3; M_Education = 2.41, SD_Education = 2.82) with the Test of Memory Malingering 1 (TOMM-1), Rey Fifteen Items Test (Rey-15) (free recall and recognition trials), DCT, MODEMM, and MMSE/MoCA as part of their neuropsychological assessment. Accuracy parameters and base failure rates were computed for the DCT and MODEMM indicators against the TOMM-1 and Rey-15. Two patient groups were constructed according to psychometrically defined credible/noncredible performance (i.e., pass/fail both TOMM-1 and Rey-15).

Results

Similar to other cultures, a cutoff of ≥18 on the DCT E score produced the best combination between sensitivity (0.50–0.57) and specificity (≥0.90). MODEMM indicators based on recognition accuracy, inconsistencies, and inclusion false positives generated 0.75–0.86 sensitivities at ≥0.90 specificities. Multivariable models of MODEMM indicators reached perfect sensitivities at ≥0.90 specificities against two PVTs. Patients who failed the TOMM-1 and Rey-15 were significantly more likely to fail the DCT and MODEMM than patients who passed both PVTs.

Conclusions

Our results offer proof of concept for the DCT’s cross-cultural validity and the applicability of the MODEMM on Romanian clinical examinees, further recommending the use of heterogeneous validity indicators in clinical assessments.

Introduction

The importance of validity assessment in neuropsychological evaluations has been repeatedly highlighted by professional associations in the USA (Chafetz et al., 2015; Sweet et al., 2021). Furthermore, numerous empirical studies recommend the administration of multiple independent performance validity tests (PVTs) (i.e., ≥2) for determining performance (in)validity in clinical and forensic populations (Boone, 2009; Critchfield et al., 2019; Erdodi, 2021; Erdodi et al., 2014; Jennette et al., 2022; Schroeder et al., 2019a; Victor et al., 2009; Webber et al., 2020). Several solid arguments for this practice lie in the limited signal detection ability of a single PVT (Schroeder et al., 2019a; Soble et al., 2020) and the demonstrated permeability of some freestanding (Bailey et al., 2018) or embedded PVTs to genuine cognitive impairment (Erdodi & Lichtenstein, 2017; Ovsiew et al., 2020; Soble et al., 2019).

Variable base rates of invalid performance reported throughout the literature both in the US (Martin et al., 2020; Martin & Schroeder, 2020) and in Western Europe (Dandachi-FitzGerald et al., 2013; Merten et al., 2022; Merten & Dandachi-FitzGerald, 2022) further motivate the necessity of validity investigations in all examinees referred for clinical evaluations. In both the US and Western Europe, macro social phenomena such as immigration (Europa.eu, n.d.; Congressional Budget Office, 2023) increase cultural and linguistic diversity and raise the possibility of encountering examinees who are non-native speakers of English in neuropsychological assessment contexts (Crișan et al., 2023). Therefore, administering PVTs with proven validity in culturally diverse populations becomes key to a thorough clinical neuropsychological assessment (Dandachi-Fitzgerald & Martin, 2022).

However, in some Eastern European countries (i.e., Romania), validity assessment is sorely limited by insufficient research and the scarce availability of PVTs (Crişan et al., 2022; Crişan & Erdodi, 2022). Therefore, examining the cross-cultural validity of established PVTs and the development of new PVTs normed on this population represent nascent research directions in this cultural space. Both approaches are needed to ensure the viable development of culturally valid assessment practices in understudied populations (Ali et al., 2022). Since linguistic and cultural differences can often confound PVT results (Braw, 2021), administering PVTs with low verbal mediation (Crişan & Erdodi, 2022) may partially control such variables.

Are PVTs with Low Verbal Mediation Cross-Cultural?

Establishing the cross-cultural validity of PVTs originating from the US and normed on native English speakers has been an ongoing research preoccupation in validity testing (Nijdam-Jones & Rosenfeld, 2017; Rhoads et al., 2021; Salazar et al., 2007; Weiss & Rosenfeld, 2010). Gold standards such as the Test of Memory Malingering (TOMM; Tombaugh, 1997) or public-domain PVTs such as the Rey Fifteen Items Test (Rey-15; Boone et al., 2002a) have proven their signal detection performance in various culturally diverse samples with variable levels of education and literacy, often at similar cutoffs with those reported for English speakers (Crişan & Erdodi, 2022; Gasquoine et al., 2017; Merten et al., 2007; Nijdam-Jones et al., 2019; Nijdam-Jones & Rosenfeld, 2017; Rhoads et al., 2021; Vilar-López et al., 2008). However, such findings do not imply that PVTs with low verbal mediation are inherently valid across all populations and cultures (Ali et al., 2022; Erdodi et al., 2017) because a multitude of demographic, cultural, or contextual factors (Bailey et al., 2021; Braw, 2021; Weiss & Rosenfeld, 2010) can impact PVT performance, therefore limiting outcome generalizability.

For instance, the associations between education and PVT performance may vary within the same linguistic community. In a normative study on a large sample of native Spanish speakers from different South American countries, Rivera et al. (2015) found that the impact of education on the TOMM varied as a function of the native country, with some countries showing no effect of education and hence no need to adjust cutoffs to the educational level.

In the Eastern European cultural area, a first study on PVTs administered in a Romanian clinical sample (Crişan & Erdodi, 2022) found that the TOMM-1 and Rey-15 classified patients with noncredible performance at cutoffs similar to those reported for English-speaking samples and that failure rates were significantly higher in patients with external incentives to appear impaired, as reported for other cultures. In this study, participants’ mean educational level was representative of the general Romanian population (i.e., M = 11.8, SD = 3.2). While education showed no significant associations with performance on the TOMM-1, it was significantly positively associated with the Rey-15 indicators.

The Dot Counting Test (DCT; Boone et al., 2002b) is another non-verbal PVT subject to culturally related empirical inconsistencies. A study on a highly educated (M = 13.96; SD = 2.31) US veteran sample (Soble et al., 2018) found that DCT scores were not affected by education or bilingualism. Similar results were obtained by Gasquoine et al. (2017) in a community sample of bilingual Mexican Americans with less than 16 years of education (M = 11.62, SD = 1.57). However, a study on Indian Punjabi civil lawsuit claimants (Weiss & Rosenfeld, 2010) reported notably higher failure rates on the DCT, even in examinees deemed honest by clinician ratings. The same study found a strong negative correlation between the years of formal education and DCT performance in the context of no formal schooling for half of the sample. Similar inconsistencies were reported across Spanish-speaking samples. Several studies reviewed by Strutt & Stinson (2022) support the DCT’s robustness to age and education and its signal detection performance at cutoffs similar to English speakers (i.e., ≥17; Burton et al., 2012; Robles et al., 2015; Salazar et al., 2007; Vilar-López et al., 2008). For instance, Robles et al. (2015) reported that a cutoff of ≥17 maintained adequate specificity in a Mexican sample with lower levels of education (i.e., 0–10 years), suggesting no need for cutoff adjustment. In contrast, Rhoads et al. (2021) found that, despite its robustness to age or age of immigration, the DCT produced significantly higher failure rates than expected at cutoffs designed for English speakers. The same study found a significant negative correlation between education and DCT performance. Given these contradictory findings, more research is needed on understudied populations to determine the cross-cultural validity of the DCT.

Introducing the First Romanian PVT – MODEMM

The practical need for a PVT designed for and normed on Romanian samples has led to the construction of Memory of Objects and Digits and the Evaluation of Memory Malingering (MODEMM; Sava & Crișan, 2022). The MODEMM is not only the first Romanian PVT but also a test that allows the continuous evaluation of verbal and non-verbal working memory, learning, and executive performance. In this regard, the MODEMM comprises various indicators that assess both performance validity and cognitive performance. Yet, for the scope of the present article, we will refer only to MODEMM’s validity indicators. A detailed description of the three validity indicators (mean recognition, inconsistent recognition, and false positives) is provided by Crişan et al. (2022), who tested their accuracy individually and in combinations in two studies using an experimental simulation paradigm. These indicators rely on visual stimuli (12 pictures and words of common objects) and will be further described in the Methods section.

In the initial studies, comparing experimental malingerers with cognitively impaired patients demonstrated the highest accuracies for Recognition-based indicators (Mean RG), inconsistencies across the two recognition phases (Inconsistent RG), and inclusion errors (Inclusion false positives/FP). Two multivariable models significantly increased the classification accuracies of single indicators. Model A (Mean RG and Inclusion FP) and Model B (Inconsistent RG and Inclusion FP) classified simulators with 0.73–0.77 sensitivities at 0.91 specificity in the first study and 0.85–0.87 sensitivities at 0.93 specificity in the replication study (Crişan et al., 2022). Given these promising experimental results, the need for cross-validating the MODEMM against criterion PVTs in an ecological known-groups design was highlighted (Crişan et al., 2022).

This prospective study follows the above-mentioned directions by employing an established PVT (i.e., the DCT) and a newly developed PVT (i.e., the MODEMM) against two freestanding PVTs (i.e., TOMM-1 and Rey-15) in a mixed clinical sample. TOMM-1 and Rey-15 were chosen due to their proven cross-cultural validity in the Romanian clinical population (Crişan & Erdodi, 2022), despite the methodological limitations acknowledged by the cited study. Our objectives were to (i) investigate the cross-cultural validity of the DCT and the applicability of the MODEMM in a real-world Romanian clinical setting and (ii) compare the selected cutoffs for this sample with previously recommended cutoffs. We hypothesized that (i) validity indicators of DCT and MODEMM will classify noncredible performance as defined by the TOMM-1 and the Rey-15 at cutoffs similar to those previously reported; (ii) MODEMM multivariable models will render higher accuracies than single indicators in determining noncredible performance on the TOMM-1 and Rey-15.

Methods

Participants

A total sample of 54 outpatients (26 females; M_Age = 62.02; SD_Age = 12.3; M_Education = 12.41, SD_Education = 2.82) from a mid-sized city in the Western part of Romania was included in the study. Total education years ranged from 6 (1.9%) to 18 (1.9%), with most patients reporting 12 years (44.4%) followed by 10 years (14.8%) of formal schooling. Patients' self-reported ethnicities were 87% (n = 47) Romanian, 9.3% (n = 5) Hungarian, and 3.7% (n = 2) Serbian. All patients were fluent in Romanian, which was also the assessment language. They were recruited from a neurological diagnosis and rehabilitation center and the first author’s private practice. They were evaluated by the first author at their physician's recommendation (i.e., neurologist or psychiatrist) during January–December 2022. All patients reported subjective complaints of cognitive dysfunctions, and 50% of the sample reported external incentives to appear impaired (i.e., applying for disability pensions). Their mean MMSE/MoCA score was 26.31, SD = 2.58.

Diagnoses ranged from neurological to psychiatric disorders (Table 1). For patients with cerebrovascular accidents (CVA), the time because the last stroke ranged between 3 months and 2.5 years. To control potential confounds due to different locations of the stroke, we sought an even ratio of CVA patients with left and right hemisphere strokes (50–50%). Likewise, patients with mild cognitive impairment (MCI) were selected according to MCI subtypes (i.e., 50% amnestic and 50% non-amnestic MCI). Differences between categories of disorders (i.e., of neurological and psychiatric etiology) were computed to investigate the permeability of criterion PVTs to the probability of cognitive impairment. There were no significant differences between scores of patients with neurological disorders and patients with psychiatric disorders on the MMSE/MoCA, TOMM-1, or Rey-15 (Table 2). Patients differed in age, with neurological patients being significantly older than psychiatric patients, but not in education.

Materials

The following instruments were administered to all patients:

(1) The Romanian versions of the Mini-Mental State Examination (MMSE; Folstein et al., 1975) and the Montreal Cognitive Assessment (MoCA; Nasreddine et al., 2005). All patients applying for disability pensions completed the MMSE, a mandatory test for disability evaluations in Romania. Two patients also completed the MoCA, and 12 patients (with no expressed incentives) were administered only the MoCA. MoCA scores were converted into MMSE scores using the conversion table by Fasnacht et al. (2023).

(2) The Test of Memory Malingering Trial 1 (TOMM-1; Tombaugh, 1997) requires examinees to memorize 50 drawings of common objects and identify them from 50 picture pairs in which the target images are coupled with foils. The total number of correct responses represents the main indicator, with various cutoffs recommended for English-speaking samples (e.g., ≤39 (Fazio et al., 2017); ≤40 (Denning, 2012; Schroeder et al., 2019b); ≤41 (Martin et al., 2020)). Studies on Spanish-speaking samples reported a lack of consensus regarding the most appropriate TOMM-1 cut scores (Bailey et al., 2021; Rhoads et al., 2021; Rivera et al., 2015). Crişan & Erdodi (2022) reported a cutoff of ≤39 on the TOMM-1 to provide adequate accuracies in a Romanian mixed clinical sample.

(3) The Rey Fifteen-Item Test – Free Recall (FR) and Recognition Trials (Rey-15; Boone et al., 2002a). For the FR trial, the examinee is requested to memorize 15 symbols in 10 seconds, with the instruction to draw them from memory immediately afterward. The subsequent recognition trial requires the identification of the 15 target symbols from a card containing 30 symbols (the 15 target symbols mixed with 15 foils). A combination score (Rey-15 COMB) can be computed by adding the correctly recalled items in the FR trial with the correctly recognized items and subtracting the false positives. Cutoffs for the FR trial range between an initially recommended conservative cutoff of ≤8 (Lezak, 1995) and a more liberal cutoff of ≤11, recently proven to sustain acceptable accuracies (Ashendorf et al., 2021; Crişan & Erdodi, 2022; Poynter et al., 2019). For the Rey-15 COMB score, previously reported cutoffs hovered around ≤19- ≤ 21 (Boone et al., 2002a; Messerly et al., 2021), with a cutoff of ≤20 proposed for detecting noncredible performance in Romanian clinical samples (Crişan & Erdodi, 2022).

(4) The Dot Counting Test (DCT; Boone et al., 2002b) is a non-memory-based PVT with low verbal mediation that can be introduced as a reaction time task. It presents examinees with 12 cards with black dots, 6 grouped and 6 ungrouped, and requires them to count the dots as quickly as possible without disclosing the existence of grouped or ungrouped patterns. The examinator records the reaction time (in seconds) necessary for each card and any errors in adding the dots and computes the means for grouped and ungrouped items. The main validity indicator is the E score, calculated by adding the means of seconds spent counting the grouped and ungrouped items with the total number of errors. Previous studies endorsed a range of cutoffs, primarily diagnosis-dependent. The initial validation study reported cutoffs from ≥14 for depression, post-traumatic stress disorder (PTSD), anxiety disorders, and mild TBI, ≥15 for learning disorders, ≥19 for mild neurocognitive disorders, to ≥20 for schizophrenia and severe TBI (Boone et al., 2002b). Subsequent studies on US samples recommended cutoffs of ≥14 (McCaul et al., 2018), ≥17 (Critchfield et al., 2019; Hansen et al., 2023; Webber et al., 2020), or ≥ 18 for psychiatric and neurological patients (Rhoads et al., 2021), and ≥ 15 for veteran samples (Soble et al., 2018). Cross-cultural studies generally endorsed an E-score of ≥17 for Spanish speakers (Burton et al., 2012; Gasquoine et al., 2017; Rhoads et al., 2021; Robles et al., 2015; Vilar-López et al., 2008).

(5) Memory of Objects and Digits and the Evaluation of Memory Malingering (MODEMM; Sava & Crișan, 2022) – version M (Malingering) comprises three tasks: two free recall trials, one cued recall trial, and two forced-choice recognition trials. For the first two free recall trials, 12 pictures of common items are shown to the examinee with the instruction to name and memorize each picture, followed by the immediate recall of the items. Next, verbal cues are given (i.e., the first two letters of each item) to the examinee, who is requested to complete the word corresponding to each item. Finally, the 12 target items, paired with similar foils, are presented to the examinee, first as pictures and then as words, with the instruction to identify the original items. Three validity indicators have been previously highlighted as effective: cutoffs of ≤11.5 on Mean Recognition (i.e., mean of total correct visual and verbal recognitions), ≥2 on Inconsistent Recognitions (i.e., inconsistent responses across the visual and verbal recognition trials—e.g., a correctly identified item in visual recognition and the same item misidentified in verbal recognition), ≥1 on Inclusion False Positives (i.e., any other word than the 12 original items) demonstrated sensitivities of 0.59–0.85 at specificities ≥0.91 in the original experimental studies (Crişan et al., 2022).

Procedure

All patients were assessed with the MMSE/MoCA and the above-mentioned PVTs as part of their cognitive evaluation. They consented to the inclusion of their test scores in the study in an anonymous form. Informed consent was retrieved before testing by signing a consent form. Participants with neurological and psychiatric diagnoses were included in the study if they reported cognitive dysfunctions and had intact visual and auditory abilities and minimal reading and writing skills. We did not include patients with dementia, intellectual disability, or schizophrenia, as these disorders are associated with dysfunctions that can often confound PVT scores (Bortnik & Dean, 2021; Messa et al., 2022; Victor & Boone, 2021). From the total of patients with various diagnoses assessed during January–December 2022, the current sample was selected consecutively, as follows: for those diagnoses implying a categorization (i.e., MCI, stroke), every time a patient from one category was included (e.g., amnestic MCI), a patient from the other category (e.g., non-amnestic MCI) was sought out and included (if willing). In addition, because previous reports on the Romanian mixed clinical population (Crişan & Erdodi, 2022) outlined higher PVT failure rates in patients with known external incentives, a similar sampling procedure was employed: for each recruited patient with expressed external incentives, another patient with no known incentives was included until an equal number in each category was reached. There was no missing data.

This study was approved by the University’s ethics committee (No. 32023) and was not preregistered. Relevant ethical guidelines were followed throughout the project. All data collection, storage, and processing were done with the approval of relevant institutional authorities regulating research involving human participants in compliance with the 1964 Helsinki Declaration and its subsequent amendments or comparable ethical standards. All data have been made publicly available at the osf.io repository and can be accessed at https://osf.io/grxya/?view_only=1448af0f4a314a5f8f79b43d6516ac6d (doi: 10.17605/OSF.IO/GRXYA).

Data Analysis

Classification accuracies were computed for the DCT and MODEMM against the TOMM-1 and Rey-15 COMB. Receiver operating characteristics (ROC) were computed in SPSS 20. Areas under the curve (AUCs) with the corresponding confidence intervals (95% CI) and base rates of failure (BR fails) at various cutoffs, along with sensitivity (Sn), specificity (Sp), and overall concordance rates (OCC), were reported for each PVT. Cutoffs considered relevant were the first to reach the minimum acceptable specificity level (i.e., ≥0.90; Larrabee, 2008). Pearson correlations were computed between demographic variables, MMSE/MoCA scores, and PVTs, to show the association between age, education, degree of cognitive decline, and PVTs and the similarity between PVTs, respectively. Two groups were constructed (pass both vs. fail both), taking failures on the TOMM-1 and Rey-15 COMB as indicative of noncredible performance at previously proposed cutoffs for this population (i.e., ≤39 on the TOMM-1, ≤10 on the Rey-15 FR, and ≤ 20 on the Rey-15 COMB (Crişan & Erdodi, 2022)). Differences between group scores were computed using non-parametric tests (Mann–Whitney U). Corresponding effect sizes (Glass rank-biserial correlation coefficients; rg) were reported as relevant. Finally, differences between failures in each group were computed using Chi-square tests (χ²), and the association between PVTs was measured using Goodman and Kruskal tau (Φ²). Likelihood ratios (LR) were calculated to indicate the likelihood of failing PVTs in the fail both versus pass both groups.

Results

Classification Accuracies Against Two PVTs

All individual indicators produced outstanding classification accuracies against the TOMM-1 and Rey-15 (AUCs = 0.93–0.98), many achieving perfect sensitivities at specificities ≥0.90 (Table 3). The failure rates indicated by the DCT and MODEMM were identical for TOMM-1 + Rey-15 FR and TOMM-1 + Rey-15 COMB.

A cutoff of ≥18 on the DCT E was the first to achieve acceptable specificity (0.92), rendering good (0.80) to perfect (1.00) sensitivities and very high OCCs (0.907–0.926). More conservative cutoffs produced no increase in sensitivity despite a small specificity increment (0.96). More liberal cutoffs reached perfect sensitivities but were associated with unacceptably low specificities (≤ 0.84).

On the MODEMM Mean RG, a cutoff of ≤11 reached perfect sensitivities at 0.94–0.96 specificities against both criteria, with very high OCCs (0.944–0.962). Lowering the cutoff decreased sensitivities with no increase in specificities. Raising the cutoff to ≤11.5 maintained sensitivities of 1.00 but caused specificities to drop to 0.88–0.90. Lowering the cutoff to ≤10.5 produced a dramatic decrement in sensitivity (0.75–0.80) with no gain in specificity (0.94–0.96). The most liberal cutoff on the Inconsistency indicator (≥1) achieved perfect sensitivities at 0.90–0.92 specificities. A good balance was also produced by the more conservative cutoff of ≥2 (Sn = 0.80–1.00, Sp = 0.98, OCCs = 0.962–0.981). Making the cutoff more conservative further decreased sensitivities and OCCs with no gain in specificity. A score of ≥2 false positives in Inclusions was the first to reach acceptable specificities (0.92–0.94) at high sensitivities (0.75–0.80) and OCCs (0.907–0.926). The more liberal cutoff of ≥1 was associated with perfect sensitivities and below-standard specificities (0.76–0.78). Raising the cutoff to a conservative ≥3 approached perfect specificities (0.98–1.00) at the cost of dramatically reduced sensitivities (0.25–0.40).

The two multivariable models of the MODEMM rendered outstanding AUCs (0.900–1.00) when tested against the TOMM-1 and Rey-15. Model A (i.e., Mean RG + Inclusion FP) detected all noncredible cases at the most liberal cutoffs, leaving a rate of only 6%–8% false positives. Making the cutoffs more conservative achieved only a modest increase in specificities (0.96–0.98) at the expense of reduced sensitivities (0.75–0.80). Model B (i.e., Inconsistent RG + Inclusion FP) achieved a perfect overlap with the classification of the TOMM-1 + Rey-15 FR (Sn = Sp = OCC = 1.00) at liberal cutoffs. The more conservative cutoffs maintained perfect specificity but reduced sensitivities to 0.60–0.75 and the OCCs to 0.962–0.981.

Correlations Between Demographic Variables and PVTs

Age and education showed variable associations with PVT scores (Table 4). Age correlated significantly only with the TOMM-1 and MODEMM inclusions false positive scores in the predicted directions, explaining about 10% of their variance (r² = 0.091). On the other hand, education proved significant correlations with both Rey-15 indicators and the DCT E score, explaining about 12–13% of the variance in these indicators (r² = 0.126–0.130). Scores on the MMSE/MoCA correlated significantly with all PVTs, covering between 10% and 23% of the variance (r² = 0.102–0.228), except for two MODEMM indicators (the inclusions FP and the mean recognition score).

All PVTs were significantly associated with each other in the predicted directions (i.e., error scores correlated negatively with accuracy scores). The covered variance ranged from 10% to 77% (r² = 0.105–0.774), with indicators stemming from the same PVT (e.g., TOMM-1 and TOMMe10, MODEMM Mean RG and Inconsistent RG) showing the strongest associations. Two exceptions were noted: no significant correlations were found between MODEMM Mean Recognition and Rey-15 FR, and Mean Recognition and the DCT E score, respectively, showing their independence (possibly due to different detection paradigms).

Comparisons Between Groups

Finally, differences on the DCT and MODEMM were computed between patients who passed the TOMM-1 and Rey-15 and those who failed both PVTs (Table 5) and failure rates across different cutoffs were compared (Table 6). Significant differences and very large effect sizes (rgs = 0.85–0.95) were noted for all validity indicators, with the MODEMM indicators rendering the largest differences between groups (rgs = 0.92–0.95), followed by the DCT E scores (rgs = 0.85–0.86).

Depending on cutoffs, patients in the group failing the TOMM-1 and Rey-15 were up to 20 times more likely to fail the DCT and the MODEMM Mean RG and up to 40 times more likely to fail the Inconsistent RG and Model A of the MODEMM (p < .01).

Discussion

The present study sought to cross-validate the DCT and a new PVT, the MODEMM, against the TOMM-1 and Rey-15 in a Romanian heterogeneous clinical sample and compare the cutoffs with the highest accuracies with previously recommended cutoffs for exploring the cross-cultural validity of the DCT and the MODEMM’s applicability to real-world clinical examinees. We hypothesized that (i) selected cutoffs on the DCT and MODEMM will match previously reported cut scores and (ii) indicator combinations on the MODEMM will be more accurate in classification than single indicators.

Results partially supported our first hypothesis. A cutoff of ≥18 on the DCT E was the first to reach the required specificity threshold across both criteria combinations, rendering sensitivities of 0.80–1.00, with a constant failure rate of 14.8%. This cut score matches previously reported cutoffs for native English-speaking neurological samples (Rhoads et al., 2021). Nevertheless, a score of ≥17 – more frequently reported as relevant for several English (Critchfield et al., 2019; Hansen et al., 2023; Webber et al., 2020) and Spanish-speaking samples (Burton et al., 2012; Gasquoine et al., 2017; Rhoads et al., 2021; Robles et al., 2015; Vilar-López et al., 2008) – failed to reach the minimum specificity level. Related, a cutoff of ≥19, originally recommended for mild neurocognitive disorders (Boone et al., 2002b), maintained the same sensitivities as ≥18 whereas slightly increasing specificities and the OCCs and decreasing the failure rate to 11.1% across all criteria combinations. Similar to some reports on English-speaking samples, our findings suggest that cut scores of ≥18 and ≥ 19 can accurately discriminate between psychometrically defined credible and noncredible performance in the present Romanian clinical sample with neurological and psychiatric diagnoses, offering proof of concept for the measure’s cross-cultural validity. Of course, due to the mixed nature of the small sample, any delimitations of cutoffs for types of disorders should be inferred with maximum caution. Therefore, the cutoff recommendations can be treated as tentative and less likely to be extrapolated to the Romanian clinical population. Choosing between cutoffs may vary depending on disorder type or impairment severity, aspects that future studies on more representative samples are encouraged to explore.

Also similar to studies on US samples (Soble et al., 2018), age did not appear to correlate significantly with DCT performance in our sample. However, consistent with reports on other cultures (i.e., Indian Punjabi; Weiss & Rosenfeld, 2010; US Spanish speakers; Rhoads et al., 2021), we found a strong negative correlation between education and DCT E scores.

The MODEMM indicators proved highly accurate against both combinations of the TOMM and Rey-15, with sensitivities exceeding those reported in the original experimental studies (Crişan et al., 2022). However, cutoffs had to be adjusted to achieve the optimal balance between sensitivity, specificity, and OCC. While the experimentally established cut score of ≤11.5 on the Mean RG (Crişan et al., 2022) proved exceptionally sensitive to both combinations, it barely reached the 0.90 specificity threshold against the TOMM-1 + Rey-15 COMB. Therefore, a more conservative cut score of ≤11 proved the best solution to maintaining perfect sensitivity with a very low false positive rate (4–6%) and a very high correct classification rate (94–96%). Cutoffs also needed to be adjusted for Inconsistent RG and Inclusion FP. Unlike initial results that reported a cutoff of ≥2 recognition inconsistencies (Crişan et al., 2022), our findings showed that, in this sample, at least one inconsistency would be specific enough and more sensitive than two or more inconsistencies in classifying noncredible performance determined by the TOMM-1 + Rey-15. The reverse situation was noted for Inclusion FP: an initial cutoff of ≥1 (Crişan et al., 2022) proved highly sensitive but not specific enough to noncredible performance. Raising the cutoff to ≥2 provided the sole alternative to balancing sensitivity and specificity across both PVT combinations. Despite these adjustments, the high signal detection performance of all MODEMM indicators supports their utility in real-world clinical settings within the Romanian cultural context.

In line with data on US samples defining invalid performance as failures on ≥2 PVTs (Boone, 2009; Critchfield et al., 2019; Erdodi et al., 2014; Jennette et al., 2022; Schroeder et al., 2019a; Victor et al., 2009; Webber et al., 2020), our findings showed that all single indicators reached perfect sensitivities at specificities above 0.90 and very high OCCs against the two PVTs. Two MODEMM indicators based on the forced-choice paradigm proved most accurate: the newly proposed cutoffs of ≤11 on Mean RG and ≥ 1 on Inconsistent RG demonstrated perfect sensitivity at specificities and OCCs above 0.90. A single exception was noted for the Inclusion FP indicator of the MODEMM, which nonetheless achieved 0.75–0.80 sensitivities at ≥0.90 specificities across criteria combinations. Of note, due to the small size of the total sample and especially the group classified as noncredible (i.e., fail both n = 5), accuracy parameters and effect sizes must be interpreted strictly in the context of the present sample and serve only as a proof of concept that warrants further replication studies.

Our findings also offer proof of concept for indicators based on the recognition paradigm (i.e., MODEMM Mean RG), including error indicators derived from it (i.e., Inconsistent RG). These results align with recent meta-analytic findings (Crişan et al., 2023) and studies supporting the robustness of the recognition paradigm in populations with limited English proficiency (Ali et al., 2022; Gasquoine et al., 2019). Additionally, the MODEMM’s accuracies were comparable to or even higher than accuracies rendered by PVTs well-established in the Western cultural space (i.e., TOMM-1, Rey-15, DCT). The high correlations with consecrated PVTs support their convergence in detecting invalid performance despite relying on different paradigms or detection strategies (Crişan et al., 2023). Our study adds value to PVT research by including different modalities testing for noncredible examinees using strategies other than memory impairment (e.g., intentional errors, longer reaction time, performance inconsistencies) to exaggerate their deficits. Additionally, the MODEMM’s validity indicators showed few or no significant correlations with age, education, and the level of cognitive decline, which supports their relative robustness to cognitive impairment and demographic variables. These findings demonstrate the effectiveness of the MODEMM, at least in the present sample, and encourage its use in real-world clinical settings.

Our second hypothesis was also partially supported. Multivariable models of the MODEMM proved slightly higher accuracies than individual MODEMM indicators, but this was cutoff-dependent: using the most liberal cutoffs for Models A and B outperformed their component indicators up to Model B classifying all participants of the groups determined by the TOMM-1 and Rey-15 FR. Our results, therefore, support initial experimental findings that proved the effectiveness of MODEMM indicator combinations at the initially recommended cutoffs (Crişan et al., 2022). Hence, multivariable PVT models demonstrated their utility when applied to Romanian clinical samples like in English-speaking populations (Boone, 2007; Cutler et al., 2023; Larrabee, 2012). However, when more conservative cutoffs were used, sensitivities decreased without falling below 0.75 for Model A and 0.60 for Model B, with negligible or null specificity increases. The limited amount of additional variance captured by the MODEMM’s multivariable models is to be interpreted considering the strong signal detection power of the MODEMM’s individual indicators, which left little room for additional accuracy.

Also similar to reports on English-speaking (Erdodi, 2019; Hurtubise et al., 2020) and Romanian samples (Crişan & Erdodi, 2022), the group failing both the TOMM-1 and Rey-15 produced notably higher SDs on predictor PVTs, which could be an indication of greater intra- and inter-individual response variability associated with noncredible performance (Crişan & Erdodi, 2022; Erdodi et al., 2014). The distribution of disorder type (i.e., neurological or psychiatric) was relatively even in this group: three patients had psychiatric diagnoses (i.e., two MDD, one GAD), and two had neurological diagnoses (i.e., Parkinson’s disease and epilepsy). All expressed external incentives to appear impaired. In this vein, similar to previous reports on this population (Crişan & Erdodi, 2022), the likelihood of PVT failures seemed to be associated more with the presence of external incentives than with the type of diagnostic etiology.

Finally, when failure rates across the two groups were compared, results showed that the DCT and MODEMM correctly classified between 80% and 100% of patients in the noncredible group at the established cutoffs. The corresponding likelihood ratios indicated that patients failing the TOMM-1 and Rey-15 were significantly more likely to fail the DCT and MODEMM than patients passing the TOMM-1 and Rey-15. Similar to numerous other studies (Boone, 2009; Critchfield et al., 2019; Erdodi, 2021; Erdodi et al., 2014; Jennette et al., 2022; Schroeder et al., 2019a; Victor et al., 2009; Webber et al., 2020), our findings of higher invalidity prevalence among patients who fail two PVTs support the effectiveness of using multiple psychometric criteria to determine performance authenticity (Sweet et al., 2021) in Romanian samples.

Several noteworthy limitations should be mentioned. First, the small sample size recruited from a single geographical area restrains the generalizability of our findings to the general Romanian clinical population. Using two PVTs to classify such a small number of participants into credible and noncredible groups inevitably reduced the size of the noncredible group, further limiting the interpretation of accuracy parameters to the present sample. Therefore, future studies on larger, more representative samples are clearly needed to replicate results. Second, the heterogeneity of diagnoses (i.e., neurological and psychiatric) precludes any suggestion of optimal cutoffs for specific diagnoses, as our results should be interpreted as proof of concept rather than cutoff recommendations per se. In this vein, the potential effect of several confounds (e.g., demographic variables or genuine impairment) on sample PVT scores should not be dismissed. Third, considering the scarcity of PVTs validated on Romanian samples, we had to rely on the only available methods employed before in the Romanian clinical population (i.e., TOMM-1 and Rey-15; Crişan & Erdodi, 2022) to construct the credible and noncredible groups. In this vein, the limitations of the cited study (small sample size, partial criterion design mixed with differential prevalence design, use of a single criterion measure) might have influenced the internal validity of the present study through the choice of cutoffs for the TOMM-1 and Rey-15. Nevertheless, using these two PVTs to establish criterion groups in this study (as indicated by professional associations in the US) takes this line of research one step further toward establishing a methodologically sound validity research base in Romania.

Finally, we want to note the possible effect of modality specificity, which has received recent empirical attention (Erdodi, 2019; Erdodi, 2021; Martin & Schroeder, 2020; Schroeder et al., 2019a). It implies that the paradigm similarity between predictor and criterion PVT (e.g., both rely on forced-choice recognition) affects classification accuracy, posing a serious threat to studies’ internal validity (Abeare et al., 2019; Erdodi et al., 2018; Rai & Erdodi, 2021). Indeed, the TOMM-1, Rey-15 COMB, and MODEMM Mean RG share the recognition paradigm, which could have artificially increased accuracies for MODEMM’s single indicators and multivariable models. However, the fact that the DCT, a non-memory-based PVT, provided comparable accuracies brings supplemental evidence in favor of their classification ability. Nevertheless, future research is clearly needed to explore the actual impact of modality specificity on specific PVT accuracies in diverse, more representative samples.

Conclusions

The present study cross-validated the DCT and MODEMM against the TOMM-1 and Rey-15 in a heterogeneous clinical sample from Romania. Like in other cultures, the DCT demonstrated acceptable sensitivities at ≥0.90 specificities, but unlike the generally endorsed cut score (i.e., ≥17), a cutoff of ≥18 reached optimal accuracies. Also similar to other cultures, DCT performance was uncorrelated with age but negatively correlated with education. All MODEMM indicators proved mostly robust to demographic confounds and highly accurate, reaching up to perfect sensitivity, with multivariable models slightly increasing accuracies compared to single indicators. Our findings offer proof of concept for the robustness of established North American PVTs to cultural confounds, along with the necessity of empirically verifying PVT cutoffs in every new population, the incremental accuracy of at least two PVTs in classifying invalidity, and the cross-cultural validity of the forced-choice paradigm shared by various indicators. The present findings also contribute to the field of cross-cultural validity assessment by adding information on the Romanian understudied population, thus expanding the range of culturally diverse populations with empirically proven PVT cross-cultural validity and providing guidance for future research.

FUNDING

Support from Norway Grants and UEFISCDI (Executive Agency for Higher Education, Research, Development, and Innovation Funding) 2014–2021, under Project contract no. 17/2020.

CONFLICT OF INTERESTS

The authors hold intellectual property rights associated with the MODEMM and receive royalties from its sales, which may pose a financial conflict of interest with the present study.

AUTHOR CONTRIBUTIONS

Iulia Crisan (Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Validation, Visualization, Writing – original draft, Writing – review & editing) and Florin Sava (Conceptualization, Funding acquisition, Methodology, Resources, Supervision, Validation, Visualization, Writing – review & editing)

References