The cross-national measurement invariance of the health literacy for school-aged children (HLSAC) instrument

Abstract

Background

Health literacy (HL) is an important determinant of health and health behaviours, and there is a need to monitor HL levels among all population groups. It is therefore essential to develop instruments to assess HL during childhood and adolescence. The aim of this study was to examine the cross-national measurement invariance of the instrument Health Literacy for School-aged Children (HLSAC) in four European countries.

Methods

The data were collected via standardized self-administered anonymous questionnaires within classrooms in Finland, Poland, Slovakia, and Belgium. There were in total 1468 respondents (aged 13, N = 690; aged 15, N = 778). The HLSAC instrument was used to measure the subjective HL of adolescents in each country. A multigroup confirmatory factor analysis was applied to test measurement invariance.

Results

Configural and metric invariance was established, but scalar invariance did not hold. However, the instrument exhibited high internal consistency (α = 0.85) and showed adequate fit with the data. Moreover, the partial invariance allowed comparison of mean values across the countries in question. There were significant mean value differences between countries and age-groups.

Conclusions

HL mean values (as assessed via the HLSAC instrument) can be compared across countries. The instrument has utility for large-scale international HL studies on adolescents.

Introduction

Health literacy (HL) can be defined in various ways.^1–4 It can be understood as a capacity to obtain, process and construct health information, enabling people to make sound health decisions and to work on factors that constitute their own and others’ health chances. HL is both a constitutive determinant^5,6 of and an independent risk factor⁷ for health. Hence, HL may help in tackling global health challenges and in reducing health inequalities between and within population groups.^6,8 HL has consequences at both the societal and the individual level; hence, it is essential to promote and assess the HL of both populations and individuals, especially during childhood and adolescence, when the foundations of many health behaviours are laid. However, little is known about how HL is related to health disparities among adolescents.

Various suggestions have been made for developing an appropriate HL monitoring instrument. Pleasant, McKinney and Rikard⁹ emphasized that a comprehensive instrument should be based on key attributes, including a conceptual framework, multi-dimensionality and the treatment of HL as a latent construct. One tool along these lines is the instrument entitled Health Literacy for School-aged Children (HLSAC).¹⁰ The relevant conceptualization¹¹ emphasized the multidimensional nature of health literacy (involving several domains, a latent construct and multiple items).¹⁰ The instrument was developed to meet the needs of adolescent HL. It was validated using a nationally representative target sample in Finland, where its psychometric properties were shown to be at an adequate level.^10,12 However, the applicability of the instrument (including performance and feasibility) beyond Finland remained to be tested.

It is common in behavioural and social sciences to assess relevant traits, habits, behaviours, attitudes, or conceptions with reference to self-reporting multi-item questionnaires. Usually, the items in such questionnaires are developed to assess an underlying phenomenon (latent factor). In making meaningful and valid comparisons between different groups with reference to the means of a latent factor (as calculated from an instrument), it becomes critical to examine the measurement invariance (i.e. the factorial invariance, the measurement equivalence).¹³ The attribute of measurement invariance indicates that the instrument performs similarly in different groups or populations.¹⁴ There are several methods to inspect measurement invariance.¹⁵ In the present study, we applied multigroup confirmatory factor analysis, which is one of the most commonly used methods to test invariance.^16–18 Our aim was to examine the cross-national measurement invariance of the HLSAC instrument in four European countries, i.e. Finland, Poland, Slovakia and Belgium.

Methods

Participants and data collection

The research was conducted in collaboration with the Health Behaviour in School-aged Children (HBSC) Study Network.¹⁹ The data were collected in Finland, Poland, Slovakia and Belgium in the spring of 2016. All the participants responded voluntarily and anonymously to a standardized paper-and-pen questionnaire administered during one lesson. The pupils were aware of the confidentiality of the data, and they were informed that only group-level results would be published. Each country followed national ethical/legal requirements for the survey. The final data retained 1468 participants. Of these, 690 pupils were aged 13 (Finland N = 176, Poland N = 341, Slovakia N = 173). In Belgium the number of 13-year-olds remained low; hence, no Belgian 13-year-olds were included in the analysis. There were in total 778 pupils aged 15 (Finland N = 175, Poland N = 301, Slovakia N = 118, Belgium N = 184).

Measures

The Health Literacy for School-aged Children (HLSAC) instrument, which has been validated among 13- and 15-year-old pupils, was used to measure self-reported health literacy.¹⁰ The brief 10-item instrument includes five core components (theoretical knowledge, practical knowledge, critical thinking, self-awareness and citizenship), each component being tapped by two items (table 2). All the items took the form ‘I am confident that…’. Originally the response options (on a Likert-type scale) were (i) ‘not at all true’, (ii) ‘barely true’, (iii) ‘somewhat true’ and (iv) ‘absolutely true’. The HLSAC score range was thus 10–40. The scores were divided into three categories, covering levels ‘low’ (scores 10–25), ‘moderate’ (scores 26–35) and ‘high’ (scores 36–40).¹² However, it became clear that there is almost no semantic difference in the Slovak language between the options ‘barely true’ and ‘somewhat true’ and the second option ‘barely true’ was therefore changed to ‘not quite true’. All the countries ensured translation and back-translation of items by professional translators, to ensure parity between languages. In Belgium, the questionnaire was translated into Dutch.

To confirm adequate functioning of the original HLSAC items in different countries, 14 items were added to the survey, aiming thus to check whether there could be items that functioned better than the original ten items. The analysis started with inspection of the item distributions. Six items were removed because of low discriminatory power. Next, a confirmatory factor analysis was applied to the remaining 18 items. We inspected the item loadings, and considered the item contents, but found no evidence suggesting a need to change the original HL items. Using regression analysis, we also examined what proportion of the variance of the 18-item instrument was explained by the short 10-item HLSAC instrument. In Finland and Poland, the proportion of explained variance was 97%, in Slovakia 96% and in Belgium 95%.

Statistical analysis

The distributions were first screened carefully to ensure that each item had adequate variance in each country, and to avoid the biases that can be caused by ceiling/floor effects. A multigroup confirmatory factor analysis (CFA) was used to test measurement invariance, i.e. whether the HLSAC instrument functioned in a similar way in four European countries (Finland, Poland, Slovakia, Belgium). Multigroup CFA allows simultaneous estimations and direct statistical comparisons of the measurement parameters across groups; it is therefore a useful method to test the structure of a scale.²⁰ The fits of the CFA models were evaluated using the chi-square goodness-of-fit statistic (χ²), the comparative fit index (CFI), the root mean square error of approximation (RMSEA) and the standardized root mean square residual (SRMR). A good fit with the data is indicated when CFI is higher than 0.95, lower than 0.06 for RMSEA and lower than 0.08 for SRMR.²¹ We considered that the fit with the data would be adequate, even if the chi square test was not non-significant, so long as other indexes suggested a good fit (on the grounds that the chi-square test is sensitive to large sample sizes).²¹ The levels of measurement invariance were tested using the χ² difference test statistic.

In the first phase of the modelling, we inspected the magnitude of the factor loadings for each country and each age, and fitted the configural invariance model with freely estimated parameters.^22,23 The model was a one-factor model with ten items. Configural invariance requires that the factor structure should be similar across groups, i.e. the construct (latent factor) can be measured by the same items, but the factor loading magnitudes can differ across groups. A precondition for clear comparisons is that there should be the same valid CFA construct in each group.

In the second phase of the modelling we inspected the equivalence of the factor loadings, fitting a metric invariance model. To examine the equality of the factor loadings across countries, the latent factor mean was set at zero and the variance at one. Metric invariance (with the factor loadings equal across the groups) confirms that the measured construct has the same meaning for the various participant groups.²³ If the metric invariance model holds (via a non-significant χ² difference test), we continue by fitting a scalar invariance model (third phase). This is realized when both the factor loadings and the intercepts are equal across groups. If scalar invariance is not reached, one arrives at partial invariance.^24,25

In the fourth phase of the modelling, the equivalence of the HL mean values between countries was examined. To compare the mean values, the factor loadings were fixed at one and the corresponding intercept was set at zero, while the factor variance and the HL mean values were allowed to differ across countries. In setting the equality constraints, we first set all the participant countries to the model in each age group. After implementing group comparisons, we conducted all possible paired country comparisons. All the analyses were conducted using Mplus (version 7.3).

Finally, the internal consistency (reliability estimate) of HLSAC instrument in each country was estimated using Cronbach’s alpha. In addition, to examine whether the predictive power of the short 10-item HLSAC instrument was similar to that of the longer 18-item instrument, we ran a regression analysis, predicting the 18-item instrument with the 10-item instrument (using SPSS version 24).

Results

Table 1 presents the descriptive HL statistics for the different age-groups. The total HL mean score was 31.02. The highest mean score was found in Finland, and the lowest in Belgium. The HL mean score was higher among pupils aged 15 than among those aged 13.

The 10-item HLSAC instrument exhibited an adequate Cronbach alpha (0.85) for the data as a whole, and the internal consistency of the instrument was also at a high level in each participant country (Finland α = 0.90, Poland α = 0.85, Slovakia α = 0.80, Belgium α = 0.81). The CFA models for the 10-item instrument showed adequate fit with the overall data [χ²(35) = 200.65, P = 0.000; RMSEA = 0.06, CFI = 0.96, SRMR = 0.03], and also with the data for each country [Finland χ²(35) = 86.53, P = 0.000; RMSEA = 0.07, CFI = 0.97, SRMR = 0.03; Poland χ²(35) = 168.83, P = 0.000; RMSEA = 0.08, CFI = 0.93, SRMR = 0.04; Slovakia χ²(35) = 51.42, P = 0.036; RMSEA = 0.04, CFI = 0.97, SRMR = 0.04; Belgium χ²(35) = 69.23, P = 0.000; RMSEA = 0.07, CFI = 0.92, SRMR = 0.05].

Comparisons of the factor loadings

Table 2 indicates that the item loadings were in general at the same level in the different age-groups (pupils aged 13 and 15) and countries, although there were a few items for which the loadings seemed to vary slightly between countries.

The analysis of configural and metric invariance provided evidence that the corresponding common factors had the same meaning across groups. The baseline model (against which the restricted models were compared) had free loadings across all countries [χ²(105) = 204.60, RMSEA = 0.06, CFI = 0.95, SRMR = 0.04). The country comparison between Finland, Slovakia and Poland in the data for pupils aged 13 showed that while all ten factor loadings were equivalent in Finland and Poland [χ²diff(10) = 15.62, P = 0.11], some loadings were different in Slovakia. We continued the analysis by examining which of the factor loadings were different in the Slovakian data, setting each factor loading free one by one. The analysis showed that there was one loading that was lower in the Slovakian data (‘ability to judge how one’s own behaviour affects one’s health’). A model in which all ten factor loadings were set as equivalent in Finland and Poland, and in which all factor loadings except for one item were set as equivalent in Slovakia, Finland and Poland, proved to be as well fitted to the data as a model with all the loadings free in each country [χ²diff(19) = 21.01, P = 0.33].

The baseline model for pupils aged 15 had free loadings across all countries [χ²(140) = 261.54; RMSEA = 0.07, CFI = 0.95, SRMR = 0.05). The comparison of the factor loadings for pupils aged 15 indicated that the loadings were equivalent in Finland, Poland and Belgium [χ²diff(20) = 27.56, P = 0.12]. The model fit fell below the accepted limit when all the Slovakian factor loadings were set as equal to the loadings for the other countries [χ²diff(30) = 53.87, P = 0.01]. After releasing one item (‘having good information regarding health’) in the Slovakian data, the chi square difference test approached non-significance [χ²diff(29) = 44.64, P = 0.03]. After releasing three items in the Slovakian data (‘having good information regarding health’, ‘ability to follow the instructions given by doctors and nurses’, ‘ability to judge how one’s own behaviour affects one’s health’) the difference test showed non-significance [χ²diff(27) = 38.73, P = 0.07]. Note, however, that the loading for the item ‘ability to follow the instructions given by doctors and nurses’ was equivalent in Slovakia and Belgium. After releasing the item, ‘having good information regarding health’ in the Slovakian data, and setting the item ‘ability to follow the instructions given by doctors and nurses’ as equal in Belgium and Slovakia, the chi square difference test showed non-significance [χ²diff(28) = 37.80, P = 0.10]. These analyses suggest that the 10-item model with equal loadings across countries fitted well with the data, and particularly well in Finland, Poland and Belgium.

Examination of intercept equivalence across the three countries showed that the intercepts were not equal across countries, and that scalar invariance did not hold. This was the case both for the 13-year-olds [χ²diff(39) = 258.79, P < 0.001], and for the 15-year-olds [χ²diff(57) = 268.10, P < 0.001].

The analyses suggested that configural and metric invariance holds for comparisons between Poland and Finland (for 13-year-olds), and between Poland, Finland and Belgium (for 15-year-olds), and that models fitted the data well. Based on these analyses, partial invariance^24,25 holds. We considered this to be sufficient to conduct mean value comparisons between the countries in question. However, all the comparisons should be interpreted with caution, given the slight differences that emerged.

Comparisons of HL mean values

Comparison of the mean values for 13-year-olds (table 1) across the countries revealed differences [χ²diff(123) = 238.08, P = 0.000]. All the paired country differences were tested, and all the paired country comparisons were significant (FIN>POL, FIN>SLO, POL<SLO). Comparison of the mean values for 15-year-olds (table 1) also indicated differences between countries [χ²diff(167) = 310.92, P = 0.000). Subsequent paired testing showed significant differences between the following pairs: FIN>SLO, FIN>POL, FIN>BEL, POL<SLO. Poland and Slovakia showed no difference from Belgium. These findings were confirmed in the overall multigroup models. Thus, comparison of the baseline model (with all means free in all countries) to the model in which the means of Poland and Belgium were set as equal showed a non-significant chi-square difference [χ²diff(1) = 1.77, P = 0.18].

Discussion

The aim of this study was to examine the cross-national measurement invariance of the instrument Health Literacy for School-aged Children (HLSAC) in four European countries. The psychometric properties of the instrument were at a sound level, with configural and metric invariance accomplished. The original HLSAC instrument has been found to have adequate internal consistency in nationally representative Finnish HBSC data (α = 0.93)¹⁰ and in a Turkish study (α = 0.77).²⁶ This was the case also in the present study (total α = 0.85, variation between countries α = 0.80 – 0.90).

Furthermore, in this study, the 10-item HLSAC instrument predicted 95–97% of the variance of the longer 18-item instrument, and there was no compelling evidence (involving poor item distributions, factor structure, or reliability estimates) to change the items of the original HLSAC instrument in any of the countries.

In the CFA models for the 10-item instrument, the fit with the data was highest in Slovakia, and the fit with the other countries was at an adequate level. Metric invariance was established, and the tests showed that the factor loadings were equivalent in Finland, Poland and Belgium at age 15 and in Finland and Poland at age 13. The models at both ages were slightly poorer when the Slovakian factor loadings were set as equal to those of the other countries. It should be noted that it is very difficult to reach full measurement invariance in most empirical studies.¹³ From the perspective of measurement, it has been proposed that more liberal criteria in assessing invariance might be appropriate, on the grounds that the traditional criteria for testing exact measurement invariance may be too strict.²⁷ There are many issues that can cause variation in how respondents react to questions, and bias or lack of equivalence can cause issues, involving for example instrument adaptation (translation), social desirability, or familiarity with item response formats.^28,29 The questionnaire was translated from the original English version to the target language. Due the translation process or cultural differences, the meanings might have differed to some extent in the target languages.

Comparison of the HLSAC mean values showed significant differences between the countries in both age groups, except for Slovakia/Belgium and Poland/Belgium (15-year-olds). For both age groups, the self-reported health literacy was highest in Finland, followed by the 13-year-olds in Slovakia and Poland. The order was the same among the 15-year-olds (Finland, Slovakia, Belgium, Poland). A number of reasons could underlie the systematic statistically significant differences between Finland and the other countries. In Finland, health issues are taught at comprehensive school (grades 7–9, ages 13–15), within the subject of health education, which is an independent and obligatory school subject. The national core curriculum lays down the objectives of health education. Schools are obliged to follow these, and must also offer the same amount of health education to all pupils. In addition, pupils can gain health knowledge within other school subjects, and from general health promotion work in the schools (as part of the whole-school approach).

The present study has some limitations. It covered only four languages and one language per country. In future, it would be important to investigate measurement invariance in other countries and languages, as well as in different age groups. It is always challenging to obtain translations that are adequately similar, in such a way that participants can understand items in the same way, regardless of country, culture, or language. In future studies, it will be important to continue to pay careful attention to the translation process, as this could have been one key reason for not achieving scalar invariance in this sample.

To conclude, it is essential to develop HL measurement instruments, and to assess the levels across various settings, as this creates a basis for appropriate interventions and meaningful concrete practices. There is clearly a need for instruments such as HLSAC in the fields of public health and health promotion, in which group comparisons are frequently of interest. We suggest that the HLSAC Instrument has good applicability for these purposes, and strong potential for use in international contexts. In future, the aim will be to apply the instrument in more countries, and to examine the association between HLSAC and health behaviour. Surveys such as the HBSC study will offer good opportunities for such comparative research, with possibilities to develop the instrument further if necessary.

Acknowledgement

The authors thank Professor Thomas Abel and M.Sc. Laura Goeint for their help with the research.

Funding

Funded by Juho Vainio Foundation [#21000037611], Academy of Finland [#276239], and the Scientific Grant Agency of the Ministry of Education Science, Research and Sport of the Slovak Republic and the Slovak Academy of Sciences [reg. no. 1/0427/17].

Conflicts of interest: None declared.

References