Application of a quantitative uncertainty assessment to develop ranges of plausible toxicity values when using observational data in risk assessment: a case study examining associations between PFOA and PFOS exposures and vaccine response

Abstract

Traditional approaches for quantitatively characterizing uncertainty in risk assessment require adaptation to accommodate increased reliance on observational (vs experimental) studies in developing toxicity values. Herein, a case study with perfluorooctanoic acid (PFOA) and PFOS and vaccine response explores approaches for qualitative and—where possible—quantitative assessments of uncertainty at each step in the toxicity value development process when using observational data, including review and appraisal of individual studies, candidate study selection, dose–response modeling, and application of uncertainty factors. Each of the 15 studies identified had uncertainties due to risk of bias in confounding, outcome, and exposure ascertainment, likely contributing to the observed inconsistencies within and across studies, and resulting in lack of candidacy for dose–response assessment. Nonetheless, 2 representative studies were selected to demonstrate possible methods to quantify uncertainty in the remaining steps. Data simulations indicated lack of a clear dose–response relationship; dose–response models fit to representative simulations indicated high uncertainty in both the magnitude and direction of effect with simulated benchmark dose and its lower limit values varying at least 66- and 86-fold for PFOA and PFOS. Uncertainty factor application added minimal uncertainty. Combined, a high level of uncertainty was observed, precluding the ability to confidently assess causal dose–response relationships with the observational data, alone. This case study highlights the need for quantitative uncertainty analysis when developing toxicity values with observational data and, importantly, emphasizes the need for application of additional techniques to directly assess causality and the specificity of dose–response when relying on studies of association in quantitative risk assessment.

Introduction

Characterization of uncertainty in risk assessment is recognized by several authoritative bodies as an important part of risk-based decision-making that supports better risk assessment, thereby allowing for improved risk management decisions (International Programme on Chemical Safety (IPCS) 2017; EFSA Scientific Committee et al. 2018; U.S. EPA 2022a). Some agencies, such as the European Food Safety Authority (EFSA), have developed one of the most comprehensive guidance documents to date on uncertainty analysis in scientific assessments. EFSA defines uncertainty as “a general term referring to all types of limitations in available knowledge that affect the range and probability of possible answers to an assessment question” and recommends quantifying the combined impact of as many identified uncertainties as possible and qualitatively characterizing the remaining unquantified uncertainties prior to providing conclusions to decision makers (EFSA Scientific Committee et al. 2018). The World Health Organization (WHO) and United States Environmental Protection Agency (USEPA) also have guidance on the assessment of uncertainty, but these are more limited and less prescriptive (IPCS 2017; U.S. EPA 1989, 2001, 2022a,). Other publications have addressed the need for uncertainty analysis and provide recommendations or guidelines for addressing uncertainty in risk assessments (NASEM 2014; Beck et al. 2016; Wikoff et al. 2019; Maertens et al. 2022), including Grading of Recommendations, Assessment, Development and Evaluation (GRADE) guidelines for assessing the certainty in modeled evidence utilizing systematic review methods to inform health care-related decisions (Brozek et al. 2021).

Much of the guidance and practical experience in conducting uncertainty assessments has occurred prior to the stark increase in reliance of epidemiological data in risk assessment. Human health risk assessments (HHRAs) are increasingly using epidemiological evidence (vs studies in experimental animals) for the identification of critical effects, dose–response modeling, and determination of a point of departure (POD) from which a reference dose (RfD) or concentration (RfC) is derived. Recent examples include the USEPA Toxicological Reviews of perfluorooctanoic acid (PFOA) and perfluorooctane sulfonic acid (PFOS), TCDD, formaldehyde, and ethylene oxide (EtO) (U.S. EPA 2012, 2016a, 2022b, 2024a, 2024b) and EFSA’s published human health risk assessment of PFAS in food (EFSA CONTAM Panel et al. 2020). There are multiple advantages to the use of epidemiological evidence, namely, the evaluation of potential effects in human populations largely eliminates questions of the biological relevance of the effects and reduced uncertainties attributable to interspecies variability. However, the type of epidemiological data available to risk assessors are typically observational, not experimental (e.g. clinical trials with controlled exposures and experimental conditions, which would be the study design equivalent to experimental animal studies).

It is well-recognized that associations with exposure–response are evaluated in observational epidemiological studies rather than causal relationships, thus presenting challenges in utilizing data from these types of studies to confidently characterize dose–response. When evaluating associations and potential for causality from experimental animal or clinical trial evidence, which are typically conducted under controlled experimental conditions, according to standardized guidelines, risk assessors can confidently assess “treatment-related” effects simply due to the study design. Conversely, outcomes and exposures in observational epidemiology studies may be uncertain due to the lack of a controlled environment or population, and therefore the results of a study may potentially be attributable to chance, bias, or confounding (Office of Health Assessment and Translation (OHAT) 2019). This does not mean that observational data are not useful in risk assessment, rather, it means that the field of toxicology must consider additional steps or approaches to be able to utilize these data for the purposes of risk assessment. For example, methods exist to evaluate confidence in determinations or inferences regarding the causal nature of the observations (Lash 2007; Lash and Ahern 2012; Lash et al., 2014; Cox 2021; Fox et al. 2021). Such methods can help to inform not only the direction of a potential exposure–response relationship but also the preciseness of such a relationship when accounting for residual or uncontrolled biases attributable to study design. These aspects directly contribute to an uncertainty assessment, whereby the confidence in the causal nature of the exposure–response relationship is critical to understand in risk assessment (Schünemann et al. 2013; Burns et al. 2014; Lash et al. 2016; Shimonovich et al. 2021).

To contribute to the ongoing research and conversations regarding best practices for the use of observational data in risk assessment (e.g. National Academies of Sciences, Engineering, and Medicine (NASEM) 2022; BfR 2023), the objective herein is to conduct a case study that demonstrates a potential application of uncertainty assessment for toxicity values based on observational data. The case study utilizes 2 RfD values derived by the U.S. EPA (2024a, 2024b) for 2 per- and poly-fluoroalkyl substances (PFAS), PFOA and PFOS; these RfDs are based on observational studies assessing decreased vaccine response and are the datasets considered for one of the co-critical effects considered by the EPA in their assessment of non-cancer effects. The specific goals include demonstrating the importance of utilizing systematic review for characterizing risk assessment uncertainty due to systemic bias in individual studies, paired with quantitative assessment of impacts of such uncertainty on the risk assessment process. It is anticipated that this assessment can contribute to ongoing discussions on best practices for the use of observational data in risk assessment.

Methods

Case study selection

For the case study herein, only decreased vaccine response was evaluated; uncertainty in the RfD values for increased total cholesterol, low birth weight, and elevated alanine aminotransferase (ALT) were not assessed, but the approach described could be applied to these or other endpoints alike to quantitatively assess uncertainty.

Uncertainty assessment methods

Uncertainty analysis is an important part of risk assessment and is a recognized step in developing toxicity values. For example, the U.S. EPA (2022a) Handbook specifically describes this step as part of the process, involving qualitative—at a minimum—but ideally quantitative assessment of principal sources of uncertainty, such as consistency in the overall database, dose-metrics, model uncertainty, and statistical uncertainty in the POD. The uncertainty analysis approach described by Wikoff et al. (2019) which, in turn, is based on recommendations by Beck et al. (2016) and NASEM (2014), was used for assessing qualitative and quantitative uncertainties in the case study. This approach involves “deconstructing” a toxicity value into the contributing components and assessing uncertainty in each step. Herein, this included: Uncertainty in identification and appraisal of relevant literature, selection of key and candidates, selection/dose–response modeling of a POD, conversation to human equivalent doses (HEDs) (for metric conversion of internal to external dose), and selection and application of uncertainty factors (Fig. 1).

Step 1: approach to assessing uncertainty in evidence identification through systematic review

This review focused on epidemiologic studies in which antibody response to vaccines was reported in populations with measures of PFOA and PFOS exposure based upon the following specific research question—“What is the dose–response relationship between PFOA and PFOS and antibody response to vaccines in humans?” The review team was composed of epidemiologists (MJV, STP), risk assessors (DSW, LCH, MJV, MMH), toxicologists (WDK, SF), a statistician (HR), and systematic review methodologists (SF, DSW).

The primary method for identification of human epidemiologic studies was handsearching previous government regulatory assessments (U.S. EPA 2016b, 2016c, 2021a, 2021b, 2024a, 2024b; Agency for Toxic Substances and Disease Registry (ATSDR) 2021; EFSA CONTAM Panel et al. 2020; OEHHA 2021) as well as an independent literature search of PubMed and Embase databases through May 2023 (see Supplementary File S1); given the objective, which was a demonstrative characterization of uncertainty, updated searches were not conducted prior to publication. Studies were included in the review if they met the Population, Exposure, Comparator, and Outcome (PECO) criteria: Observational studies (i.e. cohort, case-control, case series with n ≥ 20 cases, cross-sectional, and ecologic studies) of human populations with exposures to PFOA or PFOS, compared with either unexposed persons or persons with less exposure, with measured antibody responses to vaccines. Titles and abstracts of the articles were screened for relevance. Relevant review articles that were identified in the initial search (but not included in the systematic literature review per the inclusion criteria) were cross-checked for individual references that may not have been identified using the search syntax. The full text of the remaining relevant articles were reviewed by 2 separate epidemiologists. For each included article, information on the study design, population, vaccine of interest, exposure assessment, antibody response data, and other relevant results were extracted (see Supplementary File S1 for extracted information). Dose–response results, including statistical significance, were also extracted, as well as information on confounding factors considered and limitations discussed by primary study authors. Quality control was ensured by having 2 or more epidemiologists involved in the review process of the extracted data. Any discrepancies were resolved by consensus adjudication within the review team. By conducting this independent review of epidemiological studies, uncertainties in evidence identification and, ultimately, the impact of screening and inclusion or exclusion of evidence on hazard and risk characterizations, were assessed.

Step 2: approach for systematically characterizing uncertainty for individual studies using critical appraisal of risk of bias

Risk of bias (RoB) was assessed using the approach described in the U.S. EPA (2022a) IRIS Assessment Handbook, refined to the research question in a manner that facilitates the evaluation of the confidence in the exposure/outcome for the purposes of dose/response and risk assessment. To assess uncertainty, independent assessment (vs reliance on appraisals provided by EPA) was necessary to critically appraise studies in a manner that was specific to the evaluation of vaccine response and PFAS exposures, as well as to appraise studies for the specific purposes of uncertainty in dose–response and risk assessment. The U.S. EPA (2022a) Handbook includes assessment of 7 domains: Study selection, exposure assessment, outcome ascertainment, confounding, analysis, selective reporting, and sensitivity PECO-specific refinements of the RoB criteria developed a priori (see Supplementary Table S1; Supplementary File S2). Such refinements were necessary because the U.S. EPA Handbook only provides general instruction and prompting questions; as is described by the National Institute of Environmental Health Sciences (NIEHS) Health Assessment and Translation group (formerly OHAT 2019), RoB should be refined in context of the exposure–response relationship under investigation. For example, when considering the reliability of the outcome measurement and the potential for RoB, there are attributes unique to vaccine response that would vary from other outcomes, such as cardiovascular or birth defects.

The criteria for assessing bias in domains specific to exposure assessment, outcome ascertainment, confounding, and sensitivity were refined to account for factors specific to the exposure and outcome of interest, as well as potential confounders that specifically affect these associations. Potential confounders were identified through targeted literature searches to identify factors associated with vaccine response, in general, in addition to factors that specifically affect tetanus and diphtheria vaccine responses. Searches were conducted in PubMed for relevant review papers using combinations of keywords such as immune response, vaccine response, antibody response, factor, tetanus, and diphtheria. As an example, one of the key confounders identified specific to this research question was whether the child was breastfed; breastfed babies have been shown to have a higher antibody response to many vaccines than formula-fed babies, possibly due to maternal antibodies (Zimmermann and Curtis 2019). The sensitivity domain was excluded from evaluation herein because these topics were considered in modifications to other domains, including the validity of the method used to measure exposure and the timing of exposure. Consistent with the process in the U.S. EPA (2022a) Handbook, categorizations of Good, Adequate, Deficient or Critically Deficient were applied to each domain. Subsequently, these ratings were considered together to reach an overall study confidence rating of High, Medium, Low, or Uninformative based on guidance described by U.S. EPA (2022a) in which reviewer judgment is used to assess the study across domains, considering the likely impact the noted deficiencies in bias (and sensitivity) have on the outcome-specific results. Herein, reviewer judgments for evaluating domain-specific RoB and overall study confidence ratings were specifically made in context of utilizing the study for quantitative assessment of causal exposure(dose)–response relationships used in setting toxicological benchmarks. For this reason, most studies that had at least 1 key domain (exposure, outcome, or confounding) rated critically deficient were given an overall confidence of “Uninformative” for quantitative exposure–response evaluations.

Though consideration of relevance in study design and applicability are critical concepts in risk assessment, they are not readily transferable to systematic review tools designed for assessing interventions (vs adversity). For example, the U.S. EPA (2022a) added a sensitivity domain to the RoB tool to assess aspects of study design, whereas other entities or groups have addressed these aspects through specific evaluations of construct validity (e.g. Henderson et al., 2013). Herein, to address the risk, assessors need to characterize both the relevance and reliability of individual studies. Each study was also assigned a rating for construct and external validity per the methods described in Wikoff et al. (2019) for use in the overall evaluation of the confidence in the body of evidence. Assessment of these aspects of validity on an individual-study basis provides a more transparent and objective evaluation of individual study confidence (as well as uncertainty), which is consistent with risk assessment approaches that base most quantitative health-based benchmarks on single study versus a body of evidence. Wikoff et al. (2019), OHAT (2019), and U.S. EPA (2022a) stressed the importance of evaluating study construct, or study design, when critically appraising studies because the design alone has different strengths and weaknesses that should be considered in risk assessment. Similarly, external validity or generalizability of individual studies was also considered important, given the objective is to assess uncertainty a toxicity value, which, for these case studies, is based on individual studies and not the evidence base as a whole. Assessment of construct and external validity also informs uncertainty factor selection, discussed in a subsequent step.

Step 3: approach to systematically assess uncertainty in hazard classifications

Uncertainty in hazard classifications was conducted qualitatively. Hazard classifications were determined using the process described in the U.S. EPA (2022a) IRIS Assessment Handbook. The evidence base was synthesized by focusing on factors that increase or decrease certainty in reported finding as evidence for hazard (Table 1). Each of these considered factors (including confidence in study findings [i.e. RoB and sensitivity], consistency across studies, dose/exposure–response gradient, strength [effect magnitude] of the association, directness of outcome or endpoint measures, and coherence) was qualitatively evaluated in assigning an initial and final summary judgment.

Step 4: approach to assess uncertainty in candidate study selection

Studies were rated as high, medium, low, or uninformative confidence to determine which studies are appropriate for consideration as candidate studies for toxicity value derivation. Per the U.S. EPA (2022a) Handbook, studies rated as high or medium confidence are preferred over the use of low confidence studies, but low confidence studies can be used for toxicity value estimation with appropriate justification. Uninformative studies should not be used for risk assessment. For this demonstrative assessment, uncertainty in candidate studies was evaluated by carrying the candidate studies selected by U.S. EPA (2024a, 2024b) forward for comparison of the resulting toxicity value uncertainty ranges for the explicit purpose of demonstrating the conduct and results of an uncertainty assessment.

Step 5: approach to assess uncertainty in dose–response modeling of candidate studies

For the purposes of this case study, and to show proof-of-concept, all candidate studies of high or medium confidence, in addition to Budtz-Jørgensen and Grandjean (2018) and Timmermann et al. (2022), which were selected as candidate studies for POD derivation by U.S. EPA (2024a, 2024b) in their draft IRIS assessment, were included in the evaluation of uncertainty in dose–response modeling and carried forward for POD derivation through benchmark dose (BMD) modeling to calculate a BMD and its lower limit (BMDL). U.S. EPA ultimately selected Budtz-Jørgensen and Grandjean (2018) as the basis for the RfD for PFOA based on measurements of PFOA in serum at age 5 and decreased tetanus and diphtheria antibody levels in children at age 7; therefore, this uncertainty analysis focuses on these specific datasets as a case study.

Although USEPA did not have access to the raw data from Budtz-Jørgensen and Grandjean (2018) or Timmermann et al. (2022) to perform independent dose–response modeling, it made assumptions regarding the distribution of antibody responses in the full population and used the estimated standard deviation (SD) of antibody responses and reported model coefficients from Budtz-Jørgensen and Grandjean (2018; ref 10328962 HERO) to derive BMD(L)s for PFOA and PFOS using Equation (1).

where m(Response) is the median response (predicted by the dose–response model) at the BMD, the SD is the SD of the distribution of response in the population, and the benchmark response (BMR) factor (BMRF) is either 1 or 0.5, depending on the sensitivity of the chosen endpoint or BMR (U.S. EPA 2022b).

Traditionally, U.S. EPA’s (2012) BMD guidance states that the BMR is a change in SD from the control response (as shown in Equation 2).

where m(0) is the median response of controls (or zero dose) and the SD is the SD at the control (or zero dose) group. This analysis quantifies the impact or uncertainty of using the SD of the antibody response in the total study population in place of a control as the BMR through quantitative comparison of the BMD(L)s derived using Equations (1) and (2).

Due to the limited reporting in Budtz-Jørgensen and Grandjean (2018) and in the supplementary model output available in the USEPA HERO database (ref 10328962 HERO), model fit and the impact of BMR and/or model shape could not be evaluated. Therefore, simulated datasets were created to quantify the potential impact of variations in modeling approaches (i.e. model selection and shape and use of alternative BMRs) and uncertainties associated with the USEPA’s use of the SD of the vaccine response instead of a control response as a BMR. Specifically, simulated datasets were used to evaluate variations in derived BMD(L)s for PFOA and PFOS due to (i) BMR selection; (ii) model type; (iii) impact of log₂ instead of natural logarithm transformation of response; and (iv) application of non-traditional BMD methods based on summary statistics and assumptions of the data. Additionally, the simulated data were used to consider model fit.

Development of simulated datasets for quantification of uncertainty in POD

Detailed methods for development of simulated datasets are described in Supplementary File S3. Briefly, for each exemplar dataset (i.e. PFOA and tetanus antibody, and PFOS and diphtheria antibody with exposures measured at age 5 and antibody responses at age 7), 1,000 random iterations (or datasets) with 408 observations each were simulated assuming a lognormal (base 2) distribution for antibody response and normal distribution for PFOA or PFOS exposure. Due to limited reporting, the mean and SD of the lognormal distribution for tetanus and diphtheria were estimated based on the log₂ of the median response and an assumed SD of the interquartile ratio (IQR) divided by 1.35, consistent with the approach used by U.S. EPA (2024a, 2024b). For PFOA and PFOS serum measurements, the geometric mean as reported by Budtz-Jørgensen and Grandjean (2018) was assumed as the mean, and the SD was also assumed to be IQR/1.35. These pieces of information were gathered from Grandjean et al. (2012), Budtz-Jørgensen and Grandjean (2018), and model output available in USEPA HERO database (see Supplementary File S3).

All data simulations were performed using Stata version 17 (StataCorp. 2021. Stata Statistical Software: Release 17. College Station, TX: StataCorp LLC). For each simulated iteration (i.e. dataset), a linear model was fit, predicting log(base 2) antibody concentration associated with PFOA (or PFOS) exposure. A BMD(L) was estimated for each simulated iteration using the method presented by Budtz-Jørgensen and Grandjean (2018) for direct comparison to published results (Equation 3).

For the purposes of quantitatively estimating approximate uncertainties in BMD(L)s that would hypothetically be derived through independent modeling, we selected a single simulated dataset for PFOA and PFOS, respectively, from which the exposure–response (i.e. slope) best matched the parameters (i.e. linear slope and lower bound) of the dataset described by Budtz-Jørgensen and Grandjean (2018)’s model output (ref 10328962 HERO) (comparisons shown in Table 2).

Derivation of BMD(L)s based on simulated datasets

Linear and non-linear models were applied to the simulated datasets for PFOA and PFOS using BMDS (version 3.3.2) and a BMRF of 1 and 0.5 (Equation 2) in accordance with U.S. EPA’s (2012) BMD guidance. Detailed methods are described in Supplementary File S3. Individual models were not selected for POD determination; instead, all models that converged and calculated a BMD and the 95% lower limit (BMDL) were used to evaluate underlying model variability and model dependence, and the potential impact that the resulting PODs may have on the overall RfD.

In order to provide direct comparison to the BMD(L)s derived by the U.S. EPA using Budtz-Jørgensen and Grandjean (2018), BMD(L)s from the simulated datasets for PFOA and PFOS were also calculated in R (version 4.2.2) using the alternative approach shown in Equation (1). Non-linear models were also developed using R (version 4.2.2); hill, power, exponential, and polynomial models were selected to match continuous models offered within BMDS.

Alternative BMRs of 5% and 10% Extra Risk were also considered using BMDS. U.S. EPA (2024a) estimated that 2.8% of measured tetanus antibodies in the Faroese population studied by Budtz-Jørgensen and Grandjean (2018) were below the adverse cutoff (i.e. <0.1 IU/ml) at age 7. Therefore, the tail probability of the simulated dataset for PFOA and tetanus response was also assumed to be 2.8%, as the simulated data was intended to mimic the data modeled by Budtz-Jørgensen and Grandjean (2018). No comparable estimation was provided for the percentage of the population with lowered diphtheria antibodies, and therefore BMD(L)s based on Extra Risk were not estimated for PFOS.

Models within BMDS were run with and without assumptions of homogenous variance. BMDS outputs are included as Supplementary Files S4–S7.

Step 6: approach to assess uncertainty in calculation of HED

To establish HEDs for PFOA and PFOS from internal serum levels in children, U.S. EPA (2024a, 2024b) used a slightly modified single compartment model from Verner et al. (2016). The modifications adopted by USEPA were focused on updating the model for use in their evaluation (i.e. updating software code, incorporating updated physiological parameters, and using chemical-specific PBPK parameters). Importantly, this model explicitly assesses the child’s exposure by taking into consideration the mother’s exposure through life and in utero exposure of the child, followed by postnatal exposure. USEPA performed a comprehensive review of the literature for available data on PBPK input parameters. Rather than applying a range of values for each chemical-specific parameter to test the effect on the resulting HED, USEPA only applied a single value for each parameter to calculate HEDs for PFOA and PFOS.

To quantitatively assess uncertainty in HED calculations, all PBPK input parameters and their underlying sources summarized in U.S. EPA’s (2024a, 2024b) assessments for PFOA and PFOS were reviewed. As noted by both Verner et al. (2016) and U.S. EPA (2024a, 2024b) in their sensitivity analyses, besides the duration of breastfeeding, which had the highest impact, the main drivers of variability in HED calculations were chemical-specific parameters. As such, the chemical-specific parameters were evaluated in the current assessment to quantify uncertainty in HED calculations. Reported ranges in the literature for values of volume of distribution, half-life, maternal to infant partitioning, and milk to serum partitioning were reviewed (Supplementary File S8). Importantly, similar criteria as used by the USEPA were considered in the current evaluation for selecting appropriate ranges of chemical-specific parameters. For example, a wide range of values are reported for the half-life of PFOA (i.e. 0.53 to 22 years). As the experimental approach of the data collection greatly impacts the derivation of half-life (e.g. amount of initial dose and time of follow-up), some values are more appropriate than others.

Once appropriate ranges for each chemical-specific parameter were identified, alternate values for each parameter were first run individually in USEPA’s modified Verner et al. (2016) PBPK model to understand the directionality of the change in the resultant HED from modifying the parameter. From this initial evaluation, low and high body burden scenarios for PFOA and PFOS exposure were developed and are listed in Supplementary File S8. High and low body burden values established in the current assessment for each chemical-specific parameter were then applied to all BMDLs for PFOA and PFOS estimated by USEPA from summary information from Budtz-Jørgensen and Grandjean (2018) and Timmermann et al. (2022) to develop upper and lower-bound ranges of HEDs. For simplicity, only the BMDLs selected by U.S. EPA (2024a, 2024b) were used to investigate the uncertainty in chemical-specific PBPK parameters, as the magnitude of uncertainties due to varying BMDL estimates is already captured in Step 5 (Dose–Response Modeling) and these reflect only expected, not actual, uncertainties due to the reliance on simulated data. The resulting HED ranges for PFOA and PFOS equate to a quantifiable impact of uncertainty on the HEDs (and ultimately the RfDs) for PFOA and PFOS (e.g. the numerical difference between the upper and lower-bound HED values). These ranges do not reflect uncertainties predicted through modeling of the simulated data.

Step 7: approach to assess uncertainty in the application of uncertainty factors

Uncertainty factors were applied using standard risk assessment guidelines (U.S. EPA 2002). The underlying systematic review and body of evidence was used to directly inform selection of values. Because only human data for a single outcome were assessed herein, the focus of this step was in determining how the underlying evidence base informed human variability and inclusion of sensitive subpopulations as it related to the uncertainty factor for intraspecies variability.

Results

These results describe the quantitative and/or qualitative impact of uncertainty at each step of the risk assessment process as related to development of a toxicity value developed on observational data alone with this case study of PFOA/PFOS and vaccine response.

Step 1: uncertainty in evidence identification through systematic review

A total of 15 relevant studies evaluating PFOA and PFOS that met PECO inclusion criteria were identified from the peer-reviewed literature and through inventorying of relevant regulatory documents; this includes 10 cohort and 5 cross-sectional studies (Grandjean et al. 2012; Granum et al. 2013; Looker et al. 2014; Mogensen et al. 2015; Stein et al. 2016a, 2016b; Grandjean et al. 2017a, 2017b; Pilkerton et al. 2018; Abraham et al. 2020; Timmermann et al. 2020, 2022; Porter et al. 2022; Shih et al. 2022; Zhang et al. 2023). Extracted details from each of these studies are summarized in Supplementary File S1. In brief, these studies encompass a variety of exposure scenarios and populations, including: Persons from highly-exposed populations, such as the C8 participants (Looker et al. 2014) and 3M PFAS production workers (Porter et al. 2022); persons from a variety of geographic populations (e.g. the Faroe Islands, Germany, Norway, Greenland, Guinea-Bissau, and the United States); and persons in various age groups, including pediatric populations and adults. The 15 studies reported on antibody responses to a variety of vaccines—tetanus, diphtheria, measles, mumps, rubella, Haemophilus influenza B (Hib), and hepatitis types A and B.

Each of these studies were also identified in U.S. EPA’s (2024a, 2024b) toxicological assessments of PFOA and PFOS. Overall, because 2 independent searches identified the same set of literature, it was determined there was no uncertainty in evidence identification.

Step 2: systematic characterization of uncertainty for individual studies using critical appraisal of RoB

The 15 identified studies were critically appraised for internal RoB as well as construct and external validity, collectively informing the reliability and relevance for use in risk assessment. The findings of the independent critical appraisal and RoB assessment are summarized in Fig. 2 (see Supplementary File S9 for additional details supporting the summarized RoB judgments).

Critical appraisal of internal RoB

All studies measured serum PFAS; therefore, all studies received a confidence rating of “Good” for the domain of Type of Exposure Measurement. The second exposure criteria, Timing of Exposure Measurement, considered whether a single measure or multiple PFAS measures were taken in individuals prior to exposure. All but 2 studies received a confidence rating of “Deficient” or “Critically Deficient” for the timing of exposure measurement, because either a single PFAS measurement was taken prior to vaccination, the timing of the exposure measurement was unclear, or maternal serum measurements were taken later in pregnancy when serum measurements are highly uncertain or inaccurate due to pregnancy hemodynamics and serum volume expansion (Monroy et al. 2008; Kato et al. 2014; Steenland et al. 2018). Therefore, the unmeasured variability in exposures over time and the inability to establish a temporal relationship with response are potential sources of uncertainty and bias in dose–response analyses derived from these studies.

For example, in the cross-sectional study of tetanus and diphtheria antibodies in Greenlandic children between 7 and 12 years old (Timmermann et al. 2022), PFAS and antibody concentrations were measured from the same blood draw, which prevents establishment of the temporality of the exposure and response relationship. Studies that used NHANES populations were also considered “Critically Deficient” due to single serum PFAS measures with unclear timing relative to vaccine antibody measurement (Stein et al. 2016a, 2016b, Pilkerton et al. 2018), limiting both establishment of temporality and understanding of intra-personal variability in PFAS exposures; differences in PFAS elimination half-lives may impact relative PFAS concentrations in serum, and so multiple measurements are considered to be a more accurate representation (ATSDR 2021). Additionally, multiple studies collected maternal serum measurements in late pregnancy, including the third trimester (Grandjean et al. 2012) or “about two weeks after the expected term date” (Grandjean et al. 2017b), which indicates that exposures may be underestimated due to changes in maternal serum volume and GFR during pregnancy (Salas et al. 2006; Cheung and Lafayette 2013), this potential bias would lead to over-estimation of exposure–response gradients and risks.

The outcome domains consisted of 3 criteria, type of outcome measurement, timing of outcome measurement, and outcome measurement methods. For type of outcome measurement, all studies were rated as “Good” since all studies used laboratory-defined antibody measurements. The methods of measuring antibody levels were rated as “Good” or “Adequate” in all but 1 study because these studies described validated methods used to measure vaccine antibodies; Looker et al. (2014) described the antibody measurement but did not clearly describe whether the method was validated. Timing of Outcome Measurement was either rated as “Good” or “Adequate” in just over half (8/15) of the studies evaluated because measurements of antibody concentrations were taken both before and after either original vaccination or booster. Timing was rated as “Critically Deficient” in 7 studies because it was either unclear when antibody levels were measured or antibody levels were only measured post-vaccination, rather than measuring antibody levels both pre- and post-vaccination to ascertain changes in antibody levels after vaccination (Granum et al. 2013; Stein et al. 2016a; Pilkerton et al. 2018; Abraham et al. 2020; Timmermann et al. 2022; Porter et al. 2022; Zhang et al. 2023). Uncertainties in the measured vaccine responses and unmeasured variability in responses over time would lead directly to uncertainties in quantitative exposure–response relationships.

Adequate control of confounding factors was lacking in all studies, with 14 studies rated as “Deficient” and 1 as “Critically Deficient.” This was due to either inadequate control for key factors or failure to account for co-exposures to other PFAS. Antibody responses to specific vaccinations have been found to be affected by co-morbidities, location, and other chemical exposures, and these factors vary with the type of vaccine (further discussed in Supplementary File S2). PFAS compounds are often highly correlated, with the highest statistically significant correlations reported for PFHxS (perfluorohexane sulfonic acid), PFOS, PFOA, PFNA (perfluorononanoic acid), and PFDA (perfluorodecanoic acid) (Olsen et al. 2017). Because these compounds are correlated, it is important to account for these exposures in the analyses. Most studies did not control for co-exposures to other PFAS, giving them limited ability to differentiate the effects of PFOA or PFOS exposure from other PFAS. For example, in a study of immune response to the FluMist influenza vaccination in US adults, 4 PFAS compounds were measured in serum samples taken at the time of vaccination (PFOS, PFOA, PFHxS, and PFNA) (Stein et al. 2016a). However, Stein et al. (2016a) included only age, sex, and race/ethnicity as covariates in regression analyses. Uncontrolled confounding may bias observed associations away from the null hypothesis (i.e. “no effect”) and impact the confidence and certainty in the observed associations. The directionality and magnitude of residual biases, such as uncontrolled confounding, are important considerations for risk values derived from epidemiological observations.

A total of 11 of the 15 studies were rated as “Deficient” or “Critically Deficient” in the Participant Selection domain because they either lacked clear inclusion and exclusion criteria or had low enrollment from the eligible population without explanation as to whether the study population differed in any way from the source population, leading to possible (or likely) selection bias. As an example, none of the Faroe Islands cohorts describe the proportion of births from the respective hospitals that comprises the study population (Grandjean et al. 2012; Mogensen et al. 2015; Grandjean et al. 2017a, 2017b; Shih et al. 2022). Use of a biased samples may impact observed exposure–response associations and can limit generalizability of the findings to broader populations.

Studies were largely considered “Adequate” or “Good” for the 2 other domains: Analysis and Selective Reporting. However, 3 studies were rated as “Deficient” in analysis for failing to report confidence intervals for risk estimates (Abraham et al. 2020), not addressing the large amount of missing or non-recruited eligible individuals (Granum et al. 2013), or inconsistently reported P-values for some but not all analyses with no justification provided (Pilkerton et al. 2018).

All studies received an overall confidence rating of either “Low Confidence” (n = 8) or “Uninformative” (n = 7) (Fig. 2). Each of the studies had a RoB related to participant selection, timing of exposure and outcome measures, and/or confounding; collectively, the potential for systemic and residual bias from these observational studies results in a high level of uncertainty related to the reliance of such data for the purposes of risk assessment—and in particular, uncertainty in utilizing reported associations to represent a precise dose–response relationship for toxicity value derivation.

Critical appraisal of construct and external validity

Study construct was a limitation in several studies. Five studies were of cross-sectional design—these types of studies cannot determine temporality between exposure and outcome because both are measured simultaneously, making these studies automatically uninformative for the association between PFOA or PFOS exposure and vaccine response. Other studies were of limited external validity due to the evaluation of very small, isolated populations with unique characteristics or exposures, which may not be generalizable to larger populations in different settings. For example, the Greenlandic children in the Timmermann et al.’s (2022) study had exposure to high levels of mercury and polychlorinated biphenyl (PCB) concentrations (attributable to diet) when compared with American children, and the measured PFOS concentrations in Greenlandic children were twice as high as those measured in American or Faroese children. Use of small, isolated populations may allow for the observation of exposure–response in sensitive populations or considerations of other extremes in environmental or cultural practices. However, there is no indication that the studied populations of the Faroese or Greenlandic children represent a sensitive population, or if they represent the population of interest (or, e.g. in the case of an RfD for the EPA, the US population).

Step 3: systematic assessment of uncertainty in hazard characterization

Due to the uncertainty and high RoB, the 7 observational studies ranked as “uninformative” would not be incorporated into evidence synthesis and integration judgments according to the U.S. EPA (2022a) Handbook. The U.S. EPA Handbook guidance states that studies rated as “Critically Deficient” or “Uninformative” based on the limitations identified are not incorporated into evidence synthesis and integration judgments and are not useable for dose–response analyses. Studies with “Low Confidence” should be used only with proper justification and caveats, especially when high- or medium-quality information (including toxicological evidence from animal studies) is available (U.S. EPA 2022a). Therefore, based on the analysis herein, none of the identified studies could be confidently used in the derivation of an RfD; in this case study, where there are no medium- or high-quality studies for consideration, this analysis highlights the importance of applying uncertainty analysis in the assessment to evaluate the impact of assumptions and caveats.

The initial confidence is defined as neutral, or “indeterminate” judgment based on the study design per standard practice when the study design does not include controlled exposures. Considerations specific to the literature on PFAS and vaccine response that up- or down-grade certainty in the evidence regarding PFOA or PFOS and vaccine response hazard classifications are summarized in Table 3. Overall, the identified evidence is not consistent, with a reported mix of significant and non-significant associations identified both within and between studies for PFOA and PFOS, even among results for a specific vaccine type or population. For instance, among the Faroe Islands cohort 3 (i.e. individuals born in 1 Faroese hospital between 1997 and 2000), mostly non-significant associations were reported (Grandjean et al. 2012). Among the associations examined, only 1 statistically significant result was found for each vaccine type; statistically significant decreases in tetanus antibody levels (28.2% decrease, 95% CI: –42.7, –10.1%) at age 7 were associated with a 2-fold increase in PFOA levels at age 5, after adjusting for age, sex booster type, and antibody levels at age 5 (Grandjean et al. 2012). However, these associations were not observed at age 7 when other PFAS were considered (Mogensen et al. 2015), nor were associations observed between decreased tetanus antibody levels at age 13 and PFOA serum levels at ages 7 or 13 (Grandjean et al. 2017a). Similarly, heterogeneous findings were reported for diphtheria in this population. Supplementary Table S2 shows the mixed findings in vaccine response for tetanus associated with PFOA exposures within and across the studied populations from the Faroe Islands. The same inconsistencies were also observed for vaccine response associations with PFOS exposures (data not shown).

Among the studies of highly exposed cohorts, significant associations between PFOA/PFOS and vaccine response were not observed. Looker et al. (2014) examined PFOA or PFOS exposure and the response to influenza vaccination among participants of the C8 Health Project, who lived in an area with high levels of PFOA contamination in the drinking water. Among those participants who volunteered to receive a flu vaccine, there were no statistically significant decreased concentrations of antibodies for influenza type B, type A/H1N1, or type A/H3N2 per unit increase in serum PFOA or PFOS concentration. Analyses by quartile of PFOA or PFOS exposure also reported non-significant findings. Among workers at a 3M PFAS manufacturing facility who received a COVID-19 vaccine, there was a decreased antibody response with increasing PFOA (–1.47%, 95% CI: –8.63, 6.26) and PFOS (–3.14%, 95% CI: –6.48, 0.32) exposure that was not statistically significant (Porter et al. 2022). The lack of significant associations between exposure and vaccine response in these populations with higher exposures is not concordant with the findings from the Faroese populations. The lack of concordance and lack of clear exposure–response gradient across populations increases uncertainty in the observed associations.

In addition to a lack of concordance regarding the significance of the observed associations between vaccine response and PFOA/PFOS exposures, the direction of the associations observed between PFOA/PFOS and vaccine response were also heterogeneous. Specifically, studies reported both decreased and increased antibody measurements in response to vaccination (see Table 2 as an example within the Faroe Islands cohort). For example, Shih et al. (2022), who examined a different Faroe Islands birth cohort of individuals born in 1986 to 1987 in 3 hospitals, evaluated serum PFOA and PFOS at birth, ages 7, 14, 22, and 28 years with changes in antibody levels at 28 years old after 4 booster vaccinations—hepatitis A, hepatitis B, tetanus, and diphtheria. There was no consistent pattern to the direction of changes in any of the antibody levels measured up to age 28 with PFOA or PFOS concentrations at different ages. In another example, among participants in a Norwegian birth cohort, the effect of prenatal PFAS exposure on immune response to several pediatric vaccines at 3 years of age was examined (Granum et al. 2013). Bivariate regression analyses reported statistically significant decreased immune responses to rubella, non-statistically significant decreased immune response to measles and Hib for both PFOA and PFOS; the authors also reported and a non-statistically significant increased and decreased response to tetanus for PFOA and PFOS, respectively.

The available evidence indicates a lack of consistency both within study populations and among studies with different population demographics. Additionally, the magnitude of observed effects is relatively low, where the magnitude of change in observed antibody response is likely not clinically or biologically meaningful when compared with the WHO clinical cutoffs for antibody response (see Supplementary Table S2).

Temporality can be established in some studies in which vaccine response is measured over time, subsequent to a prior PFAS serum measurement (e.g. Grandjean et al. 2012, 2017a, 2017b). However, much of the observational evidence is cross-sectional and therefore evaluates only a single snapshot of exposure and response at a single point in time; temporality and causality cannot be established from this evidence (e.g. analyses of NHANES by Stein et al. 2016b; Pilkerton et al. 2018; and Zhang et al. 2023). Due to the cross-sectional nature of a portion of studies and the lack of consistent association in response, temporality is not clearly established and therefore does not increase or decrease confidence in the body of evidence.

Overall, the high RoB, lack of consistency, numerous uncertainties related to confounding and co-exposures, and lack of biological significance and coherence across studies indicates that there is indeterminate evidence to demonstrate an effect (i.e. the evidence base is inadequate in its ability to determine an association between PFOA or PFOS exposure and vaccine response). As such, use of these data to represent a critical effect in toxicity value development would be associated with a very high level of uncertainty.

Step 4: assessment of uncertainty in candidate study selection

Qualitatively, none of the evaluated studies were considered appropriate for selection as a candidate study, and therefore a toxicity value should not be derived from the observational evidence. Use of any studies receiving a low or critically deficient ratings in the study evaluation would be associated with a very high level of uncertainty; U.S. EPA (2022a) specifically recommends against using such. Regardless, for the purposes of continuing this case study through each step of RfD derivation, the candidate studies selected by the U.S. EPA were carried forward, despite being categorized as “low confidence” or uninformative in the critical appraisal step: Budtz-Jørgensen and Grandjean (2018), which pooled the findings from Grandjean et al. (2012) and Grandjean et al. (2017b) for dose–response analysis, and Timmermann et al. (2022).

Step 5: demonstrative assessment of uncertainty in dose–response modeling of candidate studies

Individual-level information from candidate studies is not available, which limits ability to reproduce analyses or evaluate uncertainties in modeling approaches. Based on the reported summary-level information from the published literature, U.S. EPA (2024a, 2024b) calculated a total of 20 and 18 BMDLs for PFOA or PFOS, respectively, from 6 datasets: Tetanus or diphtheria antibody levels associated with PFOA or PFOS exposure at ages 7 to 12 (Timmermann et al. 2022), tetanus or diphtheria antibody levels at age 7 associated with PFOA or PFOS exposure at age 5 (Grandjean et al. 2012; Budtz-Jørgensen and Grandjean 2018), and tetanus or diphtheria antibody levels at age 5 prebooster associated with perinatal PFOA or PFOS exposure (Grandjean et al. 2017b; Budtz-Jørgensen and Grandjean 2018). Only 4 BMDLs for each PFAS were carried forward for candidate RfD derivation: 2 BMDLs from Budtz-Jørgensen and Grandjean (2018) and 2 BMDLs from Timmermann et al. (2022). As such, these datasets were carried forward in the present assessment, despite the limitations to these studies described herein.

Notably, each of the USEPA’s BMDL and subsequent RfD estimates were derived from published summary information due to the lack of availability of the underlying raw data for additional dose–response modeling. As described in the methods, simulated datasets were generated for the analysis described herein based on reported descriptive statistics and the distribution of tetanus and diphtheria antibody responses at age 7 from Budtz-Jørgensen and Grandjean (2018)’s modeling output (ref 10328962 HERO). These simulated datasets allow for exploration of model fit, model dependence, and model uncertainty. Across the 1,000 iterations of simulated datasets for PFOA and PFOS, the mean slope estimate was approximately 0 (e.g. –0.01 IU/ml tetanus per ng/ml PFOA and 0.00 IU/ml diphtheria per ng/ml PFOS; see Table 2) with an approximately equal chance of predicting an increase or decrease in antibody levels with increasing exposure (see Fig. 3A for a graphical illustration of simulated PFOA datasets and PFOS in Supplementary Fig. S1). Because of this uncertainty in exposure–response directionality, when considering all simulated datasets, the ranges of simulated BMD(L)s for PFOA and PFOS are very broad, with an upper limit of infinity accounting for the simulations with positive (or non-adverse, i.e. predicting an increase in antibody levels) slope estimates. When limiting comparisons to only those simulations with a negative (or adverse) slope estimate, the mean BMDLs (calculated using Equation 3 for direct comparison) across all iterations (5.14 and 24.6 ng/ml for PFOA and PFOS, respectively) are approximately 20-fold greater than the BMDL estimated by Budtz-Jørgensen and Grandjean (2018) (0.25 and 1.25 ng/ml for PFOA and PFOS, respectively) (Table 2).

The simulated datasets selected for PFOA and PFOS demonstrate high variability, lack of dose-dependent clinical significance, and lack of model fit for both linear and non-linear models (see Fig. 3B and C for graphical illustration of the simulated dataset for PFOA, PFOS not shown). Dose–response trends are observed; however, the predictivity of the models is limited; the 95% prediction intervals for the linear model are broad, and the linear model predicts only approximately 1% of the observed variability (adjusted R² = 0.011 for PFOA and 0.0095 for PFOS) (Fig. 3C). Application of non-linear dose–response models does not improve the predictivity (Fig. 3C).

BMD(L)s for PFOA and PFOS from the selected simulated datasets were derived using Equations(1) and (2), described above, and calculations based on extra risk for linear and non-linear models. A summary of derived BMDLs from the selected simulation is shown in Table 4. BMDLs estimated from the selected simulated dataset ranged from 2.9 to 190 ng/ml for PFOA and 1.1 to 95 ng/ml for PFOS across all tested conditions (Table 4, see additional details in Supplementary File S3). This indicates that the quantitative uncertainty around the BMDLs generated by USEPA for PFOA and PFOS may vary by as much as 66- or 86-fold, respectively, depending on the assumptions made regarding BMR estimation and model shape or function. Of note, had all simulated datasets been considered for quantitative uncertainty analysis (recognizing that the models predicted an equal probability of decrease and increase relationships), the range of BMDLs would be even broader.

Across the tested assumptions of BMR (both type and magnitude) and model shape, model shape has the largest impact on BMDL estimation, with BMDLs typically having an approximately 2- to 3-fold difference across models under a given BMR assumption, with some exceptions (Table 4, see additional details in Supplementary File S3). Changes in the method of transformation of response data (i.e. use of natural log vs log2 response) have negligible impact on BMDL estimation (Table 4). Changes in the method used for calculating the BMR (i.e. Equations 1 vs 2) have minimal impact on the estimated BMDLs, with estimates typically having a less-than-1-fold difference (Table 4, see additional details in Supplementary File S3). Changes in the magnitude of the BMR (i.e. from ½ to 1 SD) also impact the estimated BMDLs, but the BMDL range remains comparable across magnitudes (i.e. a less than 1- to 2-fold difference).

Step 6: uncertainty in calculation of HED

Accounting for the uncertainty in chemical-specific PBPK parameters resulted in HEDs (calculated from USEPA BMDLs from Budtz-Jørgensen and Grandjean (2018) and Timmermann et al. (2022)) that differed by up to 1 order of magnitude for PFOA (1.83 × 10⁻⁷ to 1.31 × 10⁻⁶mg/kg-d) and 2 orders of magnitude for PFOS (8.77 × 10⁻⁷ to 1.15 × 10⁻⁵mg/kg-d) (Fig. 4 for PFOA, see Supplementary File S8 for PFOS). Therefore, the contribution of uncertainty from HED estimations is minimal compared with other components of PFOA and PFOS RfD derivation.

Step 7: uncertainty in assignment of uncertainty factors

The U.S. EPA (2024a, 2024b) applied a composite uncertainty factor (UF) of 10 to its HEDs to account for intra-species variability. A composite UF of 10 was also considered appropriate and conservative for the current assessment. Although Faroese and Greenlandic populations are isolated and pose advantages for evaluating a subset population, it has not been assessed whether these populations are more susceptible to decreased vaccine response; the studied populations were chosen based on convenience and cultural dietary exposures unrelated to PFAS (e.g. PCBs), not due to susceptibility to environmentally mediated immune effects. Studies for vaccine response also include children, a sensitive sub-population; therefore, a reduced UF for intraspecies variability may be suitable with additional evidence suggesting that children are the most susceptible group.

Therefore, for this case study, there is very low uncertainty attributable to differences in UF application.

Assessment of overall uncertainty in RfDs for PFOA and PFOS

After application of a 10-fold UF, the estimated RfDs differ by up to 1 order of magnitude for PFOA (1.83 × 10⁻⁸ to 1.31 × 10⁻⁷mg/kg-d) and 2 orders of magnitude for PFOS (8.77 × 10⁻⁸ to 1.15 × 10⁻⁶mg/kg-d). The cumulative uncertainties identified through each step of this analysis are summarized in Fig. 4 and Table 5. This includes both qualitative and quantitative estimates of uncertainty.

Results from this uncertainty assessment determined that the human evidence base for vaccine response and subsequent candidate study selection has the greatest impact on uncertainty in PFOA and PFOS RfDs, as the evidence was indeterminate to demonstrate an adverse immune effect and therefore should not be used in human health risk assessment. Uncertainty in the directionality and significance of the evidence across and within studies (i.e. across subsets of data within each candidate study) could qualitatively increase the upper end of the RfD range to infinite exposure (i.e. no effect or a positive increase in antibody response) if different candidate study (i.e. Mogensen et al. 2015) were used for POD derivation.

Assuming the evidence base was considered suitable to carry forward for use in BMD modeling, the next greatest source of uncertainty in RfD derivations comes from the methods used for POD calculations. Although the raw data for the candidate studies for vaccine response are unavailable, dataset simulations using Budtz-Jørgensen and Grandjean (2018)’s modeling outputs demonstrate large ranges in BMD(L)s (from 2.67 to infinity ng PFOA/ml and 12.7 to infinity ng PFOS/ml, see Table 2), with an equal chance (i.e. 50%) of the exposure–response relationship showing a positive relationship (i.e. antibody levels increase with increasing PFOA or PFOS levels) versus a negative relationship (i.e. antibody levels decrease with increasing PFOA or PFOS levels) based on the reported distribution of observations in Budtz-Jørgensen and Grandjean’s (2018)  supplemental information (ref 10328962 HERO).

Consideration of high and low body burden PBPK modeling parameters for HED calculations indicate this step of RfD derivation contributed only minor variability in RfD estimates, with resulting RfDs for PFOA and PFOS ranging from 1 to 2 orders of magnitude, respectively. No variability was expected from application of uncertainty factors in this specific case study.

Based on these uncertainties captured in this analysis, we find that there is high uncertainty in the RfD estimations for PFOA and PFOS. Ultimately, the evidence is indeterminate for evaluating the exposure–response between PFOA and PFOS exposures and vaccine response. However, if considered as a critical effect for risk value derivation, the estimated RfDs (based on BMDLs estimated by USEPA) are expected to fall within the range of no effect (i.e. data do not support development of a toxicity value) to 1.83 × 10⁻⁸mg/kg-d for PFOA and no effect to 8.77 × 10⁻⁸ for PFOS based on the ranges of selected studies and body burden calculations. Based on the observations from simulated data, these estimates may vary by up to 66- to 84-fold factor for PFOA and PFOS, respectively, if uncertainties in dose–response methods and BMDL derivation were calculable.

Discussion

The case study described herein provides a demonstration of how uncertainty can be assessed both qualitatively and quantitatively when utilizing observational epidemiological data for the purposes of developing toxicological reference values in risk assessment. For PFOA and PFOS and decreased vaccine, overall uncertainty was high, with the largest sources of uncertainty arising from uncertainties in the exposure–response relationships from the individual studies themselves, primarily due to aspects inherent to the observational study design and the high likelihood of residual and/or uncontrolled bias and confounding. A large amount of uncertainty was quantified in the dose–response assessment, demonstrating that in addition to having an equal probability of a lack of a response (and thus no dose–response), plausible RfD values differed by up to a 66- to 84-fold factor for PFOA and PFOS, respectively. This case study highlights the need to continue adapting our risk assessment methodologies to better incorporate observational data, recognizing that existing risk assessment frameworks and approaches have largely been established using experimental data and do not accommodate aspects of uncertainty inherent to observational design.

Through the application of systematic review methods, risk assessors are utilizing RoB to critically appraise studies. Such methods are good tools for qualitatively identifying the potential for systemic and/or residual bias—aspects that would reduce confidence or increase uncertainty in individual studies. As currently practiced, however, RoB tools are not sufficiently refined to the topic; this is most likely because risk assessors are tasked with assessing a wide variety of health outcomes in a single assessment. As is demonstrated herein, use of a “one-size-fits-all approach” to adjustment for confounding, exposure, and outcome assessment may not adequately address biases. In the case study, a wide variety of vaccines were examined in the evidence base, and the variability in the confounders that may affect a particular vaccine response are important to assessing the confidence in the association. This is especially important given that a candidate study is meant to demonstrate a dose that causes a specific effect for the purpose of establishing a toxicological reference value that is protective of such effects. In this case study, e.g. it is well-understood that comorbidities such as diabetes can affect antibody responses to certain vaccines; children with diabetes have lower antibody responses to HepB vaccination, and possibly PPV23, rubella, and measles vaccination, but not to diphtheria, Hib, PCV7, pertussis, or tetanus vaccination (Zimmermana and Curtis 2019). Adults with diabetes also have been shown to have lower antibody responses to HepB vaccination (Zimmermann and Curtis 2019) and malaria infection has been found to affect immune response to tetanus vaccine (Dietz et al. 1997). The lack of topic-specific consideration of these confounding factors is a clear uncertainty in the context of utilizing such data for dose–response assessments. The U.S. EPA did not apply such topic-specific refinements to confounding and thus did not fully consider aspects that could impact the dose–response; this and other differences, such as appraisal relative to timing of exposure, explain why the RoB assessments herein are different than the USEPA appraisals.

Chemical exposures other than PFAS can also affect the immune response differentially by vaccine type (e.g. differential response to polychlorinated biphenyls to various vaccines reported by Gascon et al. 2013; Stølevik et al. 2013; Jusko et al. 2010, or differential response to various vaccines and associations with arsenic as reported by Raqib et al. 2017; Welch et al. 2019, 2020). In the case study, few studies assessed herein accounted for co-exposures to other immunotoxicants, let alone co-exposure to other PFAS. This highlights the importance of accounting for the differences in the purpose of the conduct of a given study relative to its use in risk assessment—even if these studies are well-conducted using commonly accepted practices in the field of epidemiology, there are limitations regarding use of these studies for the purposes of quantitative risk assessment. Many of the environmental factors identified above that have demonstrated associations with immune responses (e.g. arsenic and PCBs) often co-occur with PFAS in drinking water; studies that do not account for such are likely to have residual and/or systemic biases that impact confidence in exposure–response association (and substantially limit confidence in using such to quantify causal relationships). For example, the Faroese population, a key study population selected by the U.S. EPA (2024a, 2024b), was originally evaluated for associations with PCB exposures attributable to population-specific diets high in PCBs (i.e. whale blubber) (Grandjean et al. 2001; Heilmann et al. 2006, 2010). These co-exposures may provide explanation, at least partially, for the inconsistent findings reported within and among the epidemiological evidence evaluating vaccine response in relation to PFOA and PFOS exposures alone. Notably, adjustment for other PFAS attenuated some observed associations (e.g. Mogensen et al. 2015)—an example where methods to consider evidence from multiple angles, such as triangulation, can also be informative when considering use of observational data for the purposes of quantitative risk assessment.

Aspects of exposure measurement can introduce bias in several ways, including the type and timing of the measurement. Guidance from the USEPA Handbook stated that to assess exposure, this domain “require[s] customization to the exposure and outcome (relevant timing of exposure)” (U.S. EPA 2022a). This domain was modified (details in Supplementary File S2) to assess these aspects of bias in exposure measurement in the current assessment. In U.S. EPA (2024a, 2024b), customization of domains was not evident, which may account for differences in ratings between the assessment herein and that of U.S. EPA (2024a, 2024b). As an example, the cross-sectional study of tetanus and diphtheria antibodies in Greenlandic children (Timmermann et al. 2022), a key study selected by U.S. EPA (2024a, 2024b), was rated as Adequate by U.S. EPA because serum samples were measured, and the cross-sectional measurements of exposure and outcome were considered acceptable based on the long half-life of PFAS. For the assessment herein, the exposure domain was split into measurement and timing aspects to address all aspects of exposure assessment bias. Measurement of PFOA in serum was determined to be Good, in agreement with USEPA. However, the timing of the measures was found to be Critically Deficient in the current assessment due to single measurements of PFAS and unclear timing of the serum draw. Multiple PFAS measurements prior to outcome ascertainment would have lessened the probability of biases due to intra-individual variability, which was not considered by USEPA. Temporality could not be established due to the cross-sectional study design, so a causal effect could not be determined from this study. Modification of the assessment tool to consider all aspects of the domain systematically led to a more accurate evaluation of the biases within each study.

Tools such as directed acyclic graphs or quantitative bias analyses described by Fox et al. (2021) and Lash et al. (2014, 2016) can qualitatively and quantitatively support estimation of uncertainties attributable to the impact of residual biases from population selection, exposure misclassification, or confounding. Quantitative bias analyses are recommended for scenarios in which inferences regarding causality are based on evidence that is small in magnitude or sensitive to bias, especially when the evidence is used to support regulatory policy making; these tools allow for incorporation of uncertainties in observed associations in hazard and subsequent risk characterizations (Lash et al. 2014). Due to the limited reporting in the currently available literature and the lack of reporting of measures of association (e.g. OR or RR) by Grandjean et al. (2012, 2017a, 2017b), a post-hoc bias analysis was not feasible for this assessment. Generally, until risk assessors readily apply the epidemiological tools for assessing potentially biasing pathways, refinement of critical appraisal (RoB) tools are adequate for qualitative considerations of bias in a manner that is fit for risk assessment, if refined to be specific to the research question, as was demonstrated herein.

The case study herein also highlights the limitations in using observational data from epidemiological studies simply because original data cannot be made (or are not) available. Since the underlying data from Budtz-Jørgensen and Grandjean (2018) and Timmermann et al. (2022) are not available for independent evaluation (herein or by any authoritative body)—an uncertainty in and of itself that data and analyses cannot be independently reproduced—neither this assessment (nor the assessment by U.S. EPA 2024a, 2024b) can quantify the true range of uncertainties or confidence in the derived BMD(L)s. The simulation of datasets, based on summary reporting, highlights the impact of dataset and modeling uncertainties. In this case study, an equal probability of an opposite direction of response was observed in the simulations, which indicates imprecision of the underlying exposure–response observations. The impact of these uncertainties could be substantial given the low magnitude of response modeled by the USEPA and the high likelihood that uncontrolled and/or residual bias impact the association. The simulated datasets presented in this assessment are intended for the purposes of comparison and quantification of the magnitude of uncertainty, only, and are not intended to represent accurate recommendations for public health guidelines. However, the analyses provide an example of the limitations when original data cannot be provided to authoritative bodies developing regulatory values.

The impact of these multiple unaddressed uncertainties are exemplified in a comparison of the USEPA’s toxicological assessments for PFOA and PFOS (U.S. EPA 2024a, 2024b) to EFSA’s published human health risk assessment of PFAS in food (EFSA CONTAM Panel et al. 2020). As part of its evaluation, EFSA developed a health-based guidance value (HBGV) for the sum of 4 PFAS, including PFOA and PFOS. This HBGV was based on a study by Abraham et al. (2020) that found an inverse association between the sum of PFOA, PFNA, PFHxS, and PFOS and diphtheria antibody levels at age 1. EFSA CONTAM Panel et al. (2020) considered decile and quintile data summary results from Grandjean et al. (2012) for dose–response modeling; however, the agency determined that BMD modeling of the Faroe Island data did not provide a BMDL that was suitable for risk assessment due to “wide BMDL-BMDU intervals” after extrapolating to zero exposure. EFSA CONTAM Panel et al. (2020) conducted a qualitative uncertainty assessment as part of its risk assessment (including evaluation of the impact of hazard characterization, exposure estimate uncertainties, BMD modeling, PBPK modeling, and risk characterization) and concluded that the impact of uncertainties on the risk assessment of PFOA, PFNA, PFHxS, and PFOS was “high.” EFSA CONTAM Panel et al.’s (2020) findings are consistent with this current analysis; however, they did not quantify the impact of these uncertainties or the range of plausible toxicity values. Although the U.S. EPA (2024a, 2024b) considered uncertainties in dosimetric adjustments, use of regression coefficients for deriving BMDs, and susceptible population identification, U.S. EPA did not account for uncertainties in hazard characterization or study selection, nor did USEPA quantify the impact of these uncertainties on toxicity value derivation. Collectively, these types of differences in assessment decisions, as well as others, lead to the resulting range of benchmarks developed for PFAS compounds (e.g. Burgoon et al. 2023).

Strengths of this assessment highlight how the efforts from systematic review methods can be better utilized to inform the risk assessment process. That is, the underlying systematic review is helpful not only in narrowing the risk assessment decisions around candidate studies but better allows for the use of the larger body of evidence to inform uncertainty in assessment findings—information that is critical to the risk managers and other users of toxicological reference values. It is a recognized limitation of this assessment that only observational data were considered in the evaluation of uncertainty, as the purpose was to provide a case study demonstration for the conduct of uncertainty assessment using observational data. Other types of data, such as experimental data from studies in rodents, or mechanistic data, could have been used to inform the biological plausibility of a causal relationship, as well as to inform on the presence and/or potency of any potential dose–response relationship. These data could also have been used to assess confidence and/or uncertainty in the case study herein but, given the volume of data and purpose of the assessment herein, were considered out of scope. Further, additional uncertainties cannot be quantified in this (or any) assessment based on the summary-level data that are publicly available; these include the impact of correlated co-exposures to other PFAS, limitations of control for confounding or effect modification in multivariate exposure–response models and their impacts on estimations of dose coefficients for BMD analyses, and the generalizability of findings from the Faroese islands studies to immune responses in the US population. Probabilistic and Bayesian methods to address exposure uncertainties when applying BMD analysis to observational epidemiological evidence (e.g. Allen et al. 2020a, 2020b; De Pretis et al. 2024) may also have been helpful, but were considered beyond the scope of the assessment. As a future activity, probabilistic methods and Bayesian approaches to characterize uncertainty in risk assessment to allow for a more holistic assessment of uncertainty rather than assuming worst-case scenarios estimated through pooling potential contributions of uncertainty from models used in the assessment, as proposed by Maertens et al. (2022), would be helpful if and when the reporting of the available epidemiological literature is sufficient to conduct such quantitative evaluations.

In summary, application of formal uncertainty assessment methods that characterize uncertainties qualitatively and/or quantitatively in each step of the risk assessment process, as shown in this case study and in EFSA CONTAM Panel et al.’s (2020) qualitative approach, are particularly important when considering observational data for quantitative risk assessment purposes. Although this case study considers only the endpoint of vaccine response for PFOA and PFOS, the approach presented herein could be applied to qualitatively and quantitatively characterize uncertainty for any substance or outcome when utilizing observational data.

Acknowledgments

Authors thank Alex Blanchette for his assistance with PBPK sensitivity analysis.

Author contributions

Authorship was determined using ICJME guidelines. Conceptualization: Daniele S. Wikoff and Laurie C. Haws. Design: Daniele S. Wikoff, Melissa M. Heintz, Melissa J. Vincent, Susan T. Pastula, Laurie C. Haws. Analysis: Melissa M. Heintz, Melissa J. Vincent, Daniele S. Wikoff, Susan T. Pastula, Heidi Reichert, and William D. Klaren. Interpretation: Daniele S. Wikoff, Melissa J. Vincent, Melissa M. Heintz, Laurie C. Haws, and Susan T. Pastula. Draft manuscript: Daniele S. Wikoff, Melissa M. Heintz, Melissa J. Vincent, Susan T. Pastula, and Heidi Reichert. Revisions: Daniele S. Wikoff, Melissa M. Heintz, Melissa J. Vincent, Susan T. Pastula, and Laurie C. Haws.

Supplementary material

Supplementary material is available at Toxicological Sciences online.

Funding

This work was supported in part by 3M.

Conflicts of interest. The authors are employees of ToxStrategies, LLC, a private consulting firm that provides services to private and public organizations for toxicology, epidemiology, and risk assessment issues. The work reported in this article was conducted during the normal course of employment, and no personal fees were received. Funding for this work was supported in part by 3M. 3M was not involved in the study design, analyses, interpretation of results, or manuscript preparation. 3M did not provide comments, edits, or any other direction regarding the content and conclusions of the manuscript. The contents of this manuscript solely reflect the view of the authors. D.S.W. is an associate editor at Toxicological Sciences.

References