Use of quality assessment tools within systematic reviews in orthodontics during the last decade: looking for a threshold?

Summary

Objective

To record the prevalence and extent of use of quality assessment/ risk of bias tools in orthodontic systematic reviews and to identify whether systematic reviews authors stipulated a threshold during the evaluation process of the primary studies included in systematic reviews, published across the previous decade and until now. Associations with publication characteristics including the journal of publication, year, the inclusion of a meta-analysis, design of primary studies and others, were sought.

Materials and Methods

Electronic search within 6 orthodontic journals and the Cochrane Database of Systematic Reviews was conducted to identify relevant systematic reviews from 1 January 2010 and 31 December 2020. The outcomes of interest pertained to the use, type and extent of quality appraisal/ risk of bias tools utilized as a standard process within the systematic reviews, and also whether a threshold had been stipulated by the systematic reviews authors. Predictor variables included journal, year of publication, geographic region, number of authors, involvement of a methodologist, type of systematic reviews, inclusion of meta-analysis, type/design of primary studies.

Results

A total of 262 systematic reviews were eligible for inclusion, with 41 quality appraisal/ risk of bias sets of tools being described either jointly or in isolation. One-third of the systematic reviews of the present sample (88/262; 33.6%) included a threshold, while this was mostly represented by the stipulation of sensitivity analyses in this respect (64/88; 72.8%). Journal of publication (non-Cochrane systematic reviews versus Cochrane systematic reviews: adjusted odds ratio, OR: 0.04, 95%CI: 0.01, 0.16; P < 0.001) and inclusion of a meta-analysis (adjusted OR: 8.76; 95%CI: 4.18, 18.37; P < 0.001), were identified as significant predictors for preplanning of thresholds.

Conclusions and implications

Quality assessment tools for primary studies are largely used and varied in orthodontic systematic reviews, while a threshold-level has been stipulated in only one third. Additional efforts should be endorsed by the scientific community, to embrace more straightforward adoption of the most rigorous reporting guidelines in this respect.

Introduction

Systematic reviews (SRs) and meta-analyses (MAs) are considered as the cornerstone of the evidence pyramid, with direct implications to decision making and formulation of guidelines for clinical practice (1, 2). The quality of conduct and reporting of SRs and MAs is of paramount importance on the interpretation of research findings to the clinical field, while channels of dissemination and scientific impact of publications hosts have been linked to the inherent quality of the SRs (3).

The GIGO ‘garbage-in, garbage-out’ notion has been key to the understanding and interpretation of aggregate findings stemming from syntheses, while it has been claimed that the quality of the studies included in an SR and the respective risk of bias is largely reflected on the outcomes and interpretation of an SR (4, 5). Several types of bias may be recognized within an SR, largely dependent on the conduct and reporting of the included primary studies. Fundamentally, the design of these studies is subject to certain types of bias, affecting even the most powerful and rigorous research designs, as framed by randomized controlled trial (RCTs). Selection, performance, detection, attrition, and reporting bias, judged as per reporting guidelines, may significantly compromise the results of an RCT (6, 7). Issues such as confounding, information or misclassification bias may additively negatively impact the results of non-randomized and/or retrospective study designs (8). In the same line, parameters like flow and timing of the participants, index and reference tests are also considered in diagnostic test accuracy studies (9).

To date, a number of quality or risk of bias assessment tools have been described and are currently in use, as a metric of domain-based and overall soundness of primary studies, included in SRs and MAs. These may be broadly divided into component type (with domain-based criteria), checklists and scale assessments. In an assessment of available tools, Moher and co-workers identified a bulk of 34 tools utilized for this purpose, back in 1995 (10). Later on, Sanderson et al., in 2005 (11), came across 86 tools (checklists and scales), only addressing susceptibility to bias arising from observational research; since then, numerous adjuncts have been developed, of varying complexity, even characterized by arbitrary, custom-made and non-validated scaling systems. Overall, and notwithstanding the more clear-cut picture in terms of an RCT appraisal, with the prevalent use of the Cochrane risk of bias tool (6, 7), the situation is bleaker in non-randomized studies.

Despite an almost universal, or at least large-scale use of quality assessment tools for rating the internal validity of the primary studies in SRs, there is general discordance with regard to the utilization of these tools, specifically within the stages of the review process. In essence, quality assessment thresholds and use of entry criteria based on the risk of bias may be framed and stipulated under the distinct steps of the qualitative and the quantitative process, within the pipeline of an SR. To this line, it might be possible, that a primary included study in an SR, may not qualify for the meta-analysis of a related outcome, due to methodological issues. Such practices have been considered as a safeguard pillar against a potential diffuse of bias to the outcomes of a meta-analysis and the respective interpretation of the findings (12). It has been suggested that interpretation of conclusions of SRs, disregarding the internal validity of the included studies, may constitute one of the most prevalent types of ‘spin’, included to orthodontic research (13) and beyond (14); the latter comprising the intentional or unintentional distortion of the research findings in terms of reporting, interpretation and extrapolation.

Thus, the present study aimed to identify the prevalence of use of quality assessment tools within SRs and to further investigate whether thresholds of methodological quality/risk of bias have been applied, during the evaluation process of the primary studies included in SRs. We focused on SR publications in orthodontic speciality journals and the Cochrane Database of Systematic Reviews (CDSR), over the last decade and until 2020. In addition, associations with relevant publication characteristics were sought.

Materials and methods

We searched the electronic contents of six orthodontic journals and the CDSR related to orthodontic research. In specific the journals were: the American Journal of Orthodontics and Dentofacial Orthopedics (AJODO), the Angle Orthodontist (AO), the European Journal of Orthodontics (EJO), the Journal of Orthodontics (JO), the Orthodontics and Craniofacial Research (OCR), and the Progress in Orthodontics (PO). Search strategy was all-inclusive, considering any SR within the contents of the journals, at a first level. Following this, only SRs pertaining to human research were further considered, with animal, and/or in-vitro SRs being excluded. Specifically for the CDSR, if more than one version of the review was identified, only the most recent one qualified. If an SR was designated as ‘withdrawn’ for any reason, it was ultimately excluded. The search included dates of publication between 1 January 2010 and 31 December 2020.

Data extraction was carried out based on bespoke standardized piloted forms and ultimately inserted into an electronic spreadsheet. Calibration was undertaken between the two assessors (FM, DK) on 20 articles. Any disagreement was settled after discussion until a consensus was reached. Inter-examiner agreement was estimated on an additional 20 papers, conditional on identification of thresholds. Whether the eligible SRs included quality assessment thresholds for their primary studies, was the primary outcome assessed. All recordings were made by the first author and confirmed by the second. In addition, a number of publication characteristics and predictor variables were assessed, as follows: journal, year of publication, geographic region based on the affiliation of the first author, number of authors participating in the publication, involvement of a methodologist (as documented by the affiliation of the authors), type of systematic review (interventional, epidemiologic, diagnostic), the inclusion of quantitative synthesis (meta-analysis), type/design of primary studies included (i.e. only RCTs, only non-RCTs/observational, both). Moreover, whether a quality assessment/ risk of bias tool was used, the number of tools and the tools’ specification was documented. With regard to the primary outcome, when a threshold was stipulated, the level at which this was applied was recorded; specifically, this pertained to the qualitative synthesis level (SR level), the quantitative synthesis level (meta-analysis level) and the post-synthesis level through sensitivity analysis (i.e., exclusion of studies with increased potential of risk of bias to assess the robustness of the findings), as described in the methodology of the SR.

Statistical analysis

Descriptive statistics were performed for the pre-defined variables. Cross-tabulations were constructed to assess the association between the use of quality assessment thresholds or otherwise and pre-defined publication characteristics. Univariable and multivariable logistic regression was performed to examine the effect of aforementioned publication characteristics including journal [as a binary variable: non-Cochrane and CDSR], year of publication, the inclusion of a meta-analysis and others, on the inclusion of a quality assessment threshold. The predictors were examined sequentially one at a time in the initial model and retained in the final multivariable model if P < 0.10. The type of SR and the type of included primary studies were tested for collinearity before inclusion in the model. The Hosmer–Lemeshow test was used to check model fit. In addition, an exploratory analysis was planned and performed, to assess the robustness of the results with regard to the stipulation of a threshold, when only SRs with meta-analyses were assessed. The unweighted kappa statistic was used to assess inter-rater agreement as per the primary outcome. A kappa value of 0.88 (95%CI: 0.64–1.00) was achieved, denoting almost perfect agreement. The predefined level of significance was set at P < 0.05 (two-sided). All analyses were conducted with Stata version 15.1 (Stata Corporation, College Station, Texas, USA).

Results

From an initial total of 284 SRs screened within a 10-year period [2010–2020] across the journals, 262 publication reports were deemed eligible for inclusion. The flowchart of the study selection process is presented in Figure 1. Based on initial descriptive statistics, the highest number of SRs in the present sample was published in the EJO (76/262; 29.0%), followed by the AJODO (55/262; 21.0%), while the CDSR contributed a total of 26 Reviews (9.9%). There was a variety of different quality appraisal/ risk of bias tools, utilized within the SRs for the assessment of primary studies, with an aggregate of 41 sets of tools or their combinations being retrieved (Table 1). Only 10 SRs were reported without the use of any tool (10/262; 3.8%). When a tool was used in isolation, the Cochrane risk of bias tool predominated (82/180; 45.6%), while a considerable amount of SRs based the assessment of their primary studies on custom-made, non-validated tools (45/180; 25.0%). When more than one tool was described within the same SR, again, 64 out of 72 SRs included the Cochrane risk of bias tool (88.9%). Custom-made tools were also prevalent (8/72; 11.1%) (Table 1). Interventional SRs embraced mostly the Cochrane risk of bias tool (114/159; 71.7%), while a non-negligible portion included custom-made tools for the assessment of their primary studies (17/159; 10.7%). On the other hand, in epidemiologic SRs, the use of arbitrary, custom-made tools predominated (24/87; 25.3%), while also, a comparable portion utilized the Cochrane risk of bias tool and/ or the ROBINS-I tool either jointly or in isolation (22/87; 25.3%). The QUADAS tool framed the largest part of the quality appraisal/ risk of bias assessment within diagnostic-type SRs (12/16; 75.0%).

The majority of SRs were published within the last 5-years (162/ 262; 61.8%), originated from Europe (131/262; 50.0%), lacked a formal involvement of a methodologist (230/262; 87.8%) and were mostly co-authored by 4–5 investigators (133/262; 50.8%). A quantitative synthesis of data (i.e. meta-analysis) was included in 122 SRs (122/262; 46.6%), while the primary studies contributing to the findings and conclusions of an SR were both RCTs and non-RCTs (i.e. observational) in design, within the same SR (138/262; 52.7%). The type of SRs assessed was mostly interventional (159/262; 60.7%) (Table 2).

Overall, one third of the SRs of the present sample (88/262; 33.6%) included a quality appraisal/risk of bias threshold within the Review process (Table 2). This was mainly framed under the pre-planning of a sensitivity analysis to estimate the robustness of the SRs results under inclusion of primary studies of diverse methodological quality/ risk of bias (Supplementary Table 1). In this respect, sixty four out of 88 (72.8%) SRs stipulated the use of a threshold though a sensitivity analysis. A quarter of SRs (23/88; 26.1%) predetermined in their methodologies, that primary studies with compromised methodological quality/ higher levels of risk of bias, would be a priori excluded from a quantitative synthesis (i.e. meta-analysis), while only 1 (1.1%) predefined this threshold at an early stage and planned to exclude right from the qualitative part of the Review (Table 3). When only SRs with a meta-analysis was assessed, the fraction of those with the inclusion of a threshold in their methodology was increased to 54.9% (67/122). The CDSR included the highest number of SRs which included pre-planned thresholds, with 23 out of 26 SRs (88.5%) belonging to this group (Table 2). The odds for stipulating a threshold in Cochrane SRs was 7.67 with an associated 95% Confidence Interval (95%CI) of 2.30 to 25.53. For the non-Cochrane speciality journals, the OCR and EJO ranked higher, with the respective odds and 95%CI being 0.47 (0.21, 1.05) and 0.46 (0.28, 0.75) (Figure 2). The breakdown of SRs, as per stipulation of a threshold (or otherwise), across publication years, is shown in Figure 3.

In the crude, unadjusted model, a number of publication characteristics qualified as significant predictors for the stipulation of quality appraisal/ risk of bias thresholds by the SR authors, namely the journal of publication (P < 0.001), the continent of origin (P = 0.002), the inclusion of a methodologist in the authors-list (P = 0.04), the inclusion of a meta-analysis (P < 0.001), the type of SR (P = 0.004) and the included primary studies within the SR (P < 0.001). In the multivariable model, there was strong evidence that SRs published in speciality journals presented 96% lower odds for inclusion of a threshold compared to SRs published in the CDSR (adjusted OR: 0.04, 95%CI: 0.01, 0.16; P < 0.001). Moreover, SRs including a meta-analysis presented 8.76 times higher odds for the stipulation of a threshold by the Review authors (adjusted OR: 8.76; 95%CI: 4.18, 18.37; P < 0.001). There was also scarce evidence that SRs including non-RCTs (i.e. observational primary studies), presented 74% lower odds for pre-planning a threshold (adjusted OR: 0.26; 95%CI: 0.07, 0.92; P = 0.05) (Table 4).

Discussion

Findings in context

The findings of the present meta-epidemiologic report elucidated the methodological structure and use of quality/risk of a bias assessment tools in orthodontic SRs, over the last decade. Most importantly, our findings designated the stipulation of quality thresholds for the inclusion of primary studies within further steps of the quantitative synthesis. A vast amount of appraisal tools were detected, either in isolation or in combinations, while the utilization of arbitrary or non-validated tools was also prevailing. In addition, a small proportion of orthodontic SRs vaguely lacked the use of any quality/risk of bias assessment tool for the included primary studies.

We concluded that only a third of the examined SRs included the stipulation of a threshold level of internal validity for their eligible primary studies, within their methodologies, to allow for a further interpretation of their findings, conditional on the underlying level of evidence. In essence, when only SRs with quantitative syntheses were considered, the overall picture appeared improved, however still suboptimal. Only about half of the included meta-analyses in our pool of studies reported a threshold level for study inclusion based on their methodological quality or risk of bias; this was implemented either in terms of a priori application of eligibility criteria for inclusion in the mathematical synthesis, or in terms of provision for post-synthesis conduct of sensitivity analyses; the latter pertained to an attempt to get an estimate of the robustness and consistency of the results, after exclusion of methodologically compromised studies. To further add in the overall uncertainty stemming from the aforementioned suboptimal stipulation of quality thresholds, we also identified a quarter of SRs embracing custom-made non-validating tools for the assessment of the internal validity of their studies, even when used in isolation; in the same direction, a small fraction of SRs did not use any at all. The latter is apparently alarming, as one might presume that the findings presented in the most recent systematic review articles in orthodontics are frequently misinterpreted by their authors, thus, rendering their value as decision making tools in clinical practice, questionable.

Our findings suggest that the practice of post-hoc examination of primary analysis and application of sensitivity thresholds, was almost thrice more prevalent pattern, than the a priori and pre-planned exclusion of studies pertaining to low methodological quality (i.e. high risk of bias). It might thus be presumed, that investigators tend to embrace, more readily and in the first place, the inclusion of any type of study, irrespective of its methodological quality, while they further opt to dismantle the contribution of diverse studies, in terms of their internal validity, on a second stage. The use of sensitivity analyses, to assess the robustness of the synthesized results, has also been highlighted in the newly formulated and published PRISMA 2020 statement for reporting of SRs (15).

The apparent superior performance of the Reviews published in the CDSR, in terms of active adoption and adherence to the utilization of quality assessment thresholds was typically anticipated. The Reviews published in the CDSR, frame a notably rigorous and sound piece of evidence, while their methodological quality and reporting overall, has been characterized as better than non-Cochrane Reviews published in standard biomedical journals (16, 17). In this respect, the onus to consistently update available evidence and data from already existing reviews has been identified as a key policy of the Cochrane Collaboration, which may be invariably considered as a determinant of novelty and progress (18). It is of note that all Reviews published from the CDSR, which adopted a relevant threshold, had, in fact, stipulated a sensitivity analysis to examine the rigour and robustness of their findings; the latter is in full alliance with the most recent guidelines in the field (15).

Prior research

To our knowledge, there is no previous study in the field either in orthodontics or in dentistry overall, that has examined the use of quality assessment tools, as well as stipulated thresholds for rating the risk of bias of primary studies within an SR. In this respect, we aimed to follow the methodological rigorousness of the orthodontic SRs and their dynamic to provide extra measures against a potentially flawed interpretation. A previous study has shown that only about 16% of orthodontic meta-analyses, included studies with a low risk of bias (2). Furthermore, a very recent empirical report from the orthodontic literature, indicates that about 35% of meta-analyses have included conclusions claiming beneficial effects of the intervention/ exposure of interest, disregarding the identified high risk of bias in their primary studies (13).

In fact, studies describing the methodological quality and risk of bias of SRs in orthodontics do exist (19, 20). The most recent reveals an overall good and sufficient methodological rigour of the orthodontic SRs over a 3- year period, between 2015 and 2018 (20). However, the aforementioned reports have used the original version of the AMSTAR [A MeaSurement Tool to Assess systematic Reviews] critical appraisal procedure (21), developed back in 2007, as a rating tool. Albeit the original AMSTAR tool includes an assessment of the evaluation of the methodological quality of the primary studies that contribute to an SR, there is no direct implication with regard to the role of a potential threshold on the robustness of the results of a synthesis, or an identified potential variation with regard to the impact of the inclusion/ exclusion of studies judged to be at the high-end of risk of bias level, on the recorded estimates. In essence, a critical addition in this respect has been included in the updated AMSTAR 2 tool. This is considered important with regard to the interpretation of the results and conclusions of a given SR (22). Apparently, there is no prior evidence for profound implications on the prevalence of adoption of this practice in the methodologies of the SRs, in orthodontics. Contrary to the lately documented improvement dynamics (20) in the quality of conduct of SRs in the field, and based on the findings of this study, we did not view a similar perspective.

Evidence from relevant empirical reports within dentistry and oral health has been sporadic and related to very recent studies on the critical analysis of the methodological quality of SRs with the use of AMSTAR 2. In specific, approximate of 40% of SRs pertaining to oral surgery and antibiotic efficacy on third molar surgery, failed to account for the potential impact of variant risk of bias levels, stemming from individual primary studies included in SRs (23). Furthermore, similar findings were confirmed within the prosthodontics research, with the majority of SRs, lacking analysis of the effects of risk of bias in evidence synthesis (24). In the broad spectrum of the biomedical literature, an assessment of a section of core clinical journals and inclusion of relevant SRs revealed that only around 13% of the studies pre-specified a threshold based on quality or bias. A considerable variation in the appraisal tools was also identified, with 58 combinations of tools recorded overall (12).

Strengths and limitations

We examined and tracked an array of quality appraisal or risk of bias tools, utilized within the evaluation process of an SR, across a wide cross-section of orthodontic SRs, constituting a fairly representative sample. This allowed for the formulation of inferences, both on time-dependent and publication dependent factors. The impact of the stipulation of a threshold on the conclusion of the SRs was not assessed and was out of the scope of the present empirical report; however, we examined its application on an earlier stage of the review process, which, if conducted on a suboptimal basis, or missing in the first place, then the effect on the reporting and interpretation of the Review findings is considered relevant.

We formally assessed the final publication report of the SRs, with no additional or primary evaluation of their protocols, if those existed, and were registered. Nevertheless, it is anticipated that the final report comprises the detailed methodological process followed by the authors of the Reviews. Furthermore, we merely reviewed publications related to orthodontic research published within journals of the speciality. While it is expected that we might have missed a number of orthodontic SRs published in general audience journals, this is considered to be a small fraction of orthodontic SRs, as speciality journals are more likely to provide state of the art evidence and a best-case scenario regarding orthodontic research. Intuitively, a small over-estimation of the identified effects might be speculated, however, the findings of the present study suggest that there is still substantial room for improvement.

Conclusions

Upon acknowledging all caveats of this work, and based on the reported findings, the following conclusions may be formulated: use of methodological quality/ risk of bias tools in orthodontic SRs seems rather universal at the SR procedure stage, however, the type and combinations are invariably diverse and abundant, while there are still arbitrary or non-validated, custom-made tools in use. Stipulation of a threshold based on quality/risk of bias of the primary included studies in orthodontic SRs seems suboptimal. This might render an interpretation of the findings from an SR problematic and potentially misleading. In this respect, editorial policies should become more stringent and potentially mandate close adherence to the most recent reporting guidelines, while actions should be endorsed to engage authors of SRs with already well-described, used and validated assessment tools, conditional on the type of study under evaluation.

Conflict of interest

None to declare.

Ethical approval

Not required. This is a meta-epidemiologic study without patients or records involvement.

Data availability

The data underlying this article will be shared at reasonable request to the corresponding author.

References