Method Matters: The Underreporting of Intimate Partner Violence

Abstract

1. Introduction

It is challenging to accurately measure sensitive attitudes, behaviors, and experiences such as political preferences, prejudice, risky sex, and intimate partner violence (IPV). Often, researchers and policy makers must rely on self-reported survey data, yet such data may not be accurate. For sensitive topics in particular, there are many reasons why survey respondents may be reluctant to tell the truth, including a fear of being judged, endangered, or legally penalized (Blair, Coppock, and Moor 2020). If misreporting bias is significant and systematic, researchers and policy makers risk over- or underestimating the prevalence and welfare costs of sensitive issues and drawing inaccurate conclusions about their causes and potential solutions.

This paper compares women’s self-reported experience of IPV under different survey methods. In a survey experiment in Nigeria, women were randomly assigned to answer questions about their experience of emotional, physical, and sexual IPV using one of two methods: an indirect method (list experiment) or the standard survey method (face-to-face questions asked by an enumerator). Compared to the direct face-to-face method, the list experiment provides additional confidentiality as respondents do not directly disclose their experience of violence. Instead, they give a response that allows researchers to calculate average prevalence across the whole sample, essentially making respondents’ answers anonymous (Blair and Imai 2012).

Results show that estimated IPV prevalence varies depending on the survey method used to measure it. The most widely used method, face-to-face interview, resulted in the lowest IPV rates, while the anonymous list method had higher rates.¹ Women’s reported experience of emotional and physical IPV was 31 and 35 percent greater when measured using the list method compared to the direct method. However, we find no statistically significant difference between the list and face-to-face methods for sexual IPV. Anecdotal evidence and the similarly high rates of sexual IPV reported for both methods (26 and 28 percent) suggest this lack of difference for sexual violence could potentially be explained by a context and interview setting where reporting forced sexual intercourse within marriage was less stigmatized.

This paper finds some similar patterns of misreporting in a related experiment in Rwanda, where we randomly assigned women and men to answer violence questions using one of three methods: the indirect list experiment and two direct survey methods (face-to-face questions asked by an enumerator, or audio computer-assisted self-interview (ACASI) on an electronic tablet). The ACASI method affords more privacy than enumerator-administered methods as respondents answer questions themselves by listening to pre-recorded questions in a headset and then selecting their answer on a tablet. In Rwanda, as in Nigeria, we find women report more physical IPV when interviewed under the list experiment than the direct methods.² In this paper, we primarily focus on the Nigeria experiment as it uses the more common list experiment design and IPV questions based on the DHS. The Rwanda experiment is discussed in the supplementary online appendix.

In addition to significant differences in reporting rates, this paper also finds that misreporting is systematic and correlated with characteristics that have been theorized as causes of IPV and targeted in development and female empowerment programs. The characteristics that correlate differently with IPV depending on the measurement method used include indicators of women’s education, vulnerability, and marital relationship quality. For example, literate Nigerian women are 8 percentage points less likely than illiterate women to experience emotional IPV when violence is measured using the face-to-face method, but 12 percentage points more likely if IPV is measured using the list method. This suggests that more- educated women underreport IPV when asked directly but that the list method’s anonymity may make them comfortable enough to report it. This accords with results from other contexts (Joseph et al. 2017; Agüero and Frisancho 2022).

Results suggest that the standard, direct methods currently used in most surveys risk generating significant underestimates of IPV prevalence. Further, social scientists risk estimating biased correlations and, if treatment is correlated with misreporting, estimating biased treatment effects and drawing inaccurate conclusions about IPV’s causes and the effectiveness of interventions to prevent it.³

This paper contributes to the growing social science literature studying alternative survey methods that may reduce misreporting (Karlan and Zinman 2012; Blattman et al. 2016; Blair, Coppock, and Moor 2020; Chuang et al. 2021), including on IPV measurement. Recent studies on IPV measurement have compared respondents’ experience of violence across the face-to-face and list experiment methods. They found mixed results, with one study in Peru finding the same rates of IPV between face-to-face and list methods (Agüero and Frisancho 2022) and others in Vietnam, India, and Burkina Faso finding a difference (Joseph et al. 2017; Bulte and Lensink 2019; Lépine, Treibich, and d’Exelle 2020).⁴ However, the latter three of these studies asked respondents about their IPV experience using two methods within the same survey about the same or similar, but slightly different, violence acts. This potentially limits comparability across methods as respondents may answer differently the second time they are asked the same question. To avoid this issue, we randomly assign respondents to answer questions using only one method rather than asking respondents two overlapping questions with different methods. This paper also contributes to the limited evidence on who misreports on sensitive questions, the implications of misreporting for bias, and the trade-offs between different survey methods that seek to address misreporting.

We contribute to recent debates about the list experiment’s accuracy and efficiency in generating prevalence estimates, and its relevance given evidence that misreporting under direct methods can be small. For example, Chuang et al. (2021) identify challenges with the consistency of the prevalence of sexual and health behaviors estimated using list experiments, finding that prevalence sometimes differed depending on the other questions included on the list. Other questions about the list experiment concern its trade-off between potential bias reduction from minimizing misreporting, and the efficiency of its estimates given it adds noise to the data (Lépine, Treibich, and d’Exelle 2020). Finally, a recent review of reporting on sensitive topics in political science found that the difference in prevalence estimates between list and direct methods is often only around 5 percentage points (Blair, Coppock, and Moor 2020). If also true for IPV, there may be a limited upside to using the method. However, we show that face-to-face underreporting on IPV is likely to be substantial in some contexts. This highlights the continued need to carefully design and test list experiments, while also identifying new methods or interviewing techniques that minimize misreporting, are consistent, precise, and feasible to implement in resource-constrained settings. While the ACASI method holds some promise, it produced broadly similar IPV estimates to the face-to-face method in our experiment in Rwanda.

This paper contributes to the nascent literature comparing IPV prevalence using multiple survey methods that offer respondents different levels of privacy and anonymity. It is also one of few papers to study IPV misreporting in Africa, and to demonstrate that misreporting is substantial in some contexts, varies by the type of violence, and is correlated with characteristics commonly targeted in women’s empowerment interventions.

2. Measuring Sensitive Topics

There is robust evidence that people underreport the truth about sensitive topics when they are directly asked in a survey. Such research compares self-reported survey answers to observed or independently verifiable outcomes such as tax records, receipt of welfare benefits, drug prescriptions for mental illness, and voting in an abortion law referendum (Gottschalk and Huynh 2010; Rosenfeld, Imai, and Shapiro 2016; Bharadwaj, Pai, and Suziedelyte 2017; Murray-Close and Heggeness 2018; Meyer and Mittag 2019).

However, reliable, unbiased observational data are rare for many sensitive topics, including IPV.⁵ This is particularly true in developing countries, where there is limited administrative data from the medical or legal services that IPV survivors may use. Non-survey data sources that address the risk of reporting bias, such as emergency and domestic violence hotline calls, hospital and police administrative records, and Google search data (Leslie and Wilson 2020; Ravindran and Shah 2020; Bullinger, Carr, and Packham 2021), were widely used to study the effect of COVID-19 lockdowns on IPV. However, they typically capture the more extreme cases of IPV and might still be biased in other ways, for example by the types of people selecting into services. For example, in Nigeria, only 1 percent of women who experienced physical or sexual IPV reported it to the police (DHS 2019). While administrative and other non-survey data sources can be useful for studying extreme IPV cases and to validate survey data in some contexts, there is still a need for accurate, representative survey data from lower-resource settings and for less extreme cases.

As a result of these non-survey data limitations and concerns about underreporting on face-to-face surveys, researchers have considered alternative survey methods to minimize misreporting bias. A popular alternative method, the list experiment broadly aims to address misreporting driven by shame, embarrassment, a preference for anonymity, or fear of repercussions if one’s answer becomes known.

2.1. The List Experiment

Researchers have used the list experiment to improve data accuracy when measuring sensitive behaviors and attitudes.⁶ The standard list experiment works as follows: respondents are randomly assigned to one of two groups to answer the list question. In the “non-sensitive” or “control” group, enumerators read respondents a set of three non-sensitive statements and ask the respondent to tell them how many but not which statements are true for the respondent, with possible answers between zero and three. Meanwhile, the “sensitive” or “treatment” group is read the same list of statements, with the addition of a fourth sensitive statement. They are given the same instructions, to say how many, between zero and four statements, are true. Given the sensitive and non-sensitive list groups are randomly assigned, the two groups are, in expectation, equivalent and the only difference is the inclusion of the sensitive statement. The prevalence of the sensitive item across the whole sample is thus the difference in the mean number of items reported by the two groups.

Unlike with direct methods, anonymity is ensured because the lists are designed in such a way that very few people in the list sensitive group have all four statements as either true or not true.⁷ This way, a respondent is not revealing their true answer to the sensitive item to the enumerator or anyone with data access. This could mitigate underreporting by respondents concerned about shame or consequences if someone were to find out their answers, or those who doubt the confidentiality of their data.

There are potential drawbacks to the list experiment. One challenge is that, by design, it cannot be used to identify individuals who respond affirmatively to the sensitive question of interest, potentially limiting its use where individuals need to be referred to support services. Also, with list experiment answers ranging from zero to four, it typically has greater variance than direct questioning methods (but also likely lower misreporting bias because of respondent anonymity). Thus list experiments may reduce bias at the cost of efficiency (though there are approaches such as the double list experiment that can reduce the method’s variance) (Blair and Imai 2012).⁸ Given these power concerns, the list method may not be ideal for use in impact evaluations as it may leave outcome analysis, and heterogeneity analysis, underpowered. Further, the method does not easily lend itself to further question complexity, for example follow-up questions to measure frequency and injuries from violence, or aggregation of several questions to create indicator variables of “any experience of violence.” Finally, Chuang et al. (2021) find evidence that prevalence estimates under the list experiment can be internally inconsistent, with rates that are sensitive to the other questions used in the list.⁹

However, given the list method’s potential to reduce misreporting, researchers have begun using it to measure IPV and other forms of gender-based violence. In several studies, IPV rates are compared between the face-to-face and list methods. The study most related to this paper, conducted in urban Lima, Peru, found no difference in overall IPV prevalence by survey method (Agüero and Frisancho 2022); studies in other contexts did find a difference. In Sri Lanka, Traunmüller, Kijewski, and Freitag (2019) found the self-reported experience of wartime sexual violence (not IPV) to be higher under the list method than face-to-face, and in India and Burkina Faso, Joseph et al. (2017) and Lépine, Treibich, and d’Exelle (2020), respectively, found higher rates of IPV under the list method than face-to-face.¹⁰

In these studies, respondents were either (a) randomly assigned to answer questions about the same experience of IPV using the direct face-to-face method or the list method (Agüero and Frisancho 2022), or (b) all respondents were asked about their IPV experience using both methods within the same survey—either with similar, but slightly different, question wording (Bulte and Lensink 2019) or the same question wording (Joseph et al. 2017; Lépine, Treibich, and d’Exelle 2020).¹¹ One issue with approach (b) is that being asked the same question in two ways may bias responses to the second question. In our Nigeria experiment, we take approach (a), asking individuals each question only once to minimize this risk.

2.2. Misreporting and Truth

Tourangeau and Yan (2007) argue that for sensitive questions where responding in the affirmative is taboo, the survey method that finds the highest prevalence is closest to truth. In this paper, particularly given the stigma surrounding the issue of IPV, we assume that in most cases, the method with the highest prevalence is closest to truth and that most misreporting is underreporting.

Without objective, non-self-reported data, we cannot, ultimately, confirm which method is closest to truth. However, the alternative survey methods tested in this paper are designed to address motivations for underreporting when interviewed with the face-to-face method—shame and fear. These alternative methods are unlikely to introduce new reasons for people to report IPV that they have not experienced. This is particularly true for the list experiment, where answers are anonymous by design, hence individuals have no incentive to overreport given they cannot be followed up with to provide aid or program services.¹² Therefore, it seems likely that the method with the highest reporting indicates the method that best addresses respondents’ reasons for misreporting and is closest to truth.

This assumption is supported by empirical evidence from the only study we identified that compares an observed sensitive behavior with self-reported behavior measured using direct and indirect methods including the list experiment. Rosenfeld, Imai, and Shapiro (2016) found that when they were asked directly, people underreported their past voting behavior on a sensitive abortion referendum in Mississippi. However, rates were higher and closer to truth when they were asked using the list experiment. No method yielded rates higher than the true “socially undesirable” outcome of voting against tightened abortion restrictions.

Overall, evidence on the measurement of sensitive topics suggests that misreporting appears to be highly issue and context dependent (Tourangeau and Yan 2007; Langhaug, Sherr, and Cowan 2010; Blair, Coppock, and Moor 2020), presumably affected by factors such as the interview setting, cultural norms, security and safety, enumerator skill, and confidence in data privacy. Therefore, this paper primarily relates to misreporting about IPV, and in contexts similar to those studied here.

3. Experimental Design

3.1. Context, Data, & Design

Countries in Sub-Saharan Africa have high rates of IPV compared to the global average. According to data gathered using the direct face-to-face method, 14 percent of Nigerian women report having experienced physical or sexual IPV in the previous 12 months (DHS 2019). This compares to 3 percent of women in the United States, 21 percent in Rwanda, 23 percent in Kenya, 18 percent in India and Bolivia, and 10 percent in South Africa (Global Burden of Disease 2017; DHS 2021).

Our study took place in one of Nigeria’s poorest and most conservative regions, Kebbi state in Northwest Nigeria. Kebbi has the country’s highest acceptance of wife-beating, with 67 percent of women believing a husband is justified in using physical violence if a wife burns the food. Yet it also has the country’s lowest reported rates of IPV (DHS 2019). Only 13, 7, and 1 percent of women report experiencing emotional, physical, and sexual IPV in the previous 12 months (compared to the national average of 26, 12, and 5, respectively). In this region, IPV is not criminalized. The penal code governing Northern Nigerian states does not outlaw marital rape or physical violence against a wife “so long as the woman is not seriously harmed” (Benebo, Schumann, and Vaezghasemi 2018).¹³

The survey experiments described in this paper were conducted in accordance with WHO guidelines on the safe and ethical collection of data on violence against women (WHO 2001, 2016).

In Nigeria, the measurement experiment was conducted during the endline survey of a cluster randomized control trial studying the effects of the Feed the Future Nigeria Livelihoods Project, a multi-component cash transfer and livelihoods program.¹⁴ The sample consisted of ultra-poor households, with approximately 15 percent of women self-reporting as literate, 30 percent in polygamous marriages, and 84 percent identifying as Muslim.

The eligibility criteria for the measurement experiment were that the respondent was female, over 17 years of age, and married and living in the same household as a male. In total, 2,817 women met these criteria and were randomly assigned to answer three experimental questions about their experience of IPV using either the face-to-face or list method. Assignment to the method groups was stratified by geographic and poverty-related strata, as well as treatment status in the randomized control trial, to ensure balance.

The three IPV questions used for the measurement experiment were either directly from or combined several violence questions from the DHS.¹⁵ The experimental questions were “In the past 12 months, has your husband

1. said or done something to humiliate you in front of others?

2. slapped you, pushed you, shaken you, or thrown something at you?

3. physically forced you to have sexual intercourse with him when you did not want to?”

Respondents were interviewed one-on-one by a female enumerator in a private setting, usually in the household compound. Enumerators were instructed to pause if anyone came within earshot of the interview, not proceeding until privacy was ensured. The survey structure is outlined in table A.1.

3.2. The Measurement Experiment: Calculating Prevalence

We assess misreporting bias by comparing rates of IPV across the face-to-face and list experiment methods (with a third method, ACASI, also tested in Rwanda and discussed in the supplementary online appendix). Given that assignment to the survey method was random, we assume that the only difference in reported violence between the groups is due to the method used.

Under the list method, prevalence of the sensitive statement is calculated by taking the difference in the mean number of statements reported as true by the two groups (Blair and Imai 2012). To administer the list experiment to the non-sensitive group, respondents are read the same list, bar the fourth sensitive statement, and asked to report how many of the three statements apply to them.¹⁶

In Nigeria, 1,405 women were randomly assigned to answer the sensitive questions using the list method group, and 1,412 using the face-to-face method.¹⁷ Power calculations indicated that the sample in Nigeria was large enough to detect a difference of approximately 6 percentage points between the groups, and in Rwanda of approximately 5 percentage points, with a significance level of 0.05, power of 0.8, and base prevalence of 20 percent.¹⁸

Overall, the groups were well balanced on key characteristics, as shown in table A.3. While several of the raw differences are statistically significant at conventional levels, the differences are small, and they are within the bounds of what might be expected given the number of variables tested for balance. No normalized differences are greater than 0.25 standard deviations, a suggested rule of thumb indicating good balance (Imbens and Rubin 2015). In additional specifications, we estimate prevalence including controls selected using post-double selection LASSO, and variables with baseline imbalance (Belloni, Chernozhukov, and Hansen 2014).¹⁹ These do not significantly change results.

4. Results

4.1. Method Matters: IPV Reporting Differs by Survey Method

This paper assesses whether there is a difference in reported rates of IPV across survey methods. A difference in prevalence implies misreporting under at least one of the methods.

Table 1 shows IPV prevalence by survey method in Nigeria. Column 3 shows the difference in prevalence between methods. The p-values for these prevalence difference tests were adjusted for multiple hypothesis testing. Sharpened q-values were computed using the approach proposed by Benjamini and Hochberg (1995) and Anderson (2008).²⁰

Results show that the survey method used can substantially affect IPV reporting. Overall, the most widely used survey method, face-to-face interviewing, resulted in the lowest reported IPV rates, with the list method having higher rates. In Nigeria, emotional and physical IPV rates were 31 and 35 percent greater, respectively, when measured using the list method (emotional violence: 39.2 percent; physical violence: 25.9 percent) compared to face-to-face (emotional violence: 29.9 percent; physical violence: 19.2 percent). The differences between the methods were statistically significant at the 5 and 10 percent levels respectively. However, there was no statistically significant difference between direct and indirect rates of sexual IPV, with reported prevalence of 28.1 and 26.1 percent respectively.

The second experiment, conducted in Rwanda and discussed in the supplementary online appendix, shows a similar pattern of results. There, the third method tested, ACASI, produced similar or slightly higher rates of sexual and intimate partner violence experience (for women) or perpetration (for men) than the face-to-face method.

Our rates of directly reported IPV are much greater than equivalent rates from Nigeria’s DHS, shown in table 2. For example, in the DHS, only 15.2 percent of women say their partner said or did something to humiliate them in front of others; 9.2 percent of women say they have been slapped; and 4.8 percent say they have been pushed, shaken, or had something thrown at them in the previous 12 months by their partner (DHS 2019). Four percent say that in the past 12 months, they have been physically forced to have sexual intercourse with their partner when they did not want to. While the DHS is conducted in a different, representative sample and thus is not directly comparable to our targeted, highly economically vulnerable sample, the contrast in IPV rates with both our direct and list experiment estimates is striking. Our much greater face-to-face rates than those in the DHS may be due to our different sample, our very private interview setting, or our two-hour long survey covering many other non-sensitive issues that gives more plausible deniability to respondents and allows enumerators to build a strong rapport before asking IPV questions.

Our results are comparable to those from several related studies. Lépine, Treibich, and d’Exelle (2020) find significant face-to-face underreporting of physical IPV in rural Burkina Faso (16–20 percentage point difference between face-to-face and list method prevalence), as do Bulte and Lensink (2019) in Vietnam (10 percentage point difference), and Joseph et al. (2017) in India (9 percentage points). However, our results diverge from those of the one other study that also adopts our one-method-per-person approach. In Lima, Peru, Agüero and Frisancho (2022) found no overall difference between face-to-face and list method physical and sexual IPV prevalence. These divergent results indicate that reporting bias is, unsurprisingly, strongly affected by contextual factors. Differences in misreporting may be due to the social norms and degree of stigma women faced in each context (e.g. urban Lima versus rural Nigeria), or differences in the interview context (e.g. privacy of the interview setting or respondents’ faith in the privacy of their data). Significant reporting differences have also been found in other surveys, for example underreporting of condom use or sexual assault during war (Jamison et al. 2013; Traunmüller, Kijewski, and Freitag 2019), suggesting that sexual and health topics in particular may be prone to reporting bias; the direct method–list method differences appear to be much smaller in many political science studies (Blair, Coppock, and Moor 2020).

Overall, our results suggest that women’s experience of emotional and physical IPV may be substantially underreported when measured using direct methods. If the difference in prevalence between the list and direct methods is a proxy for fear, shame, or social desirability bias, then these results also suggest that questions about emotional and physical IPV were more sensitive than sexual IPV in our setting. This surprising result could potentially be explained by the context. In Nigeria, a quarter of the women interviewed using the list method and a quarter interviewed using the face-to-face method reported experiencing sexual IPV in the past year. Qualitative interviews and focus groups suggested that there may be less stigma associated with reporting sexual IPV than physical IPV. Participants in our predominantly Muslim study sites indicated that religious leaders cautioned male followers not to use physical violence against their wives, but that sexual IPV was viewed differently and was not considered violence. Further, DHS statistics show that women in this region have an extremely high acceptance of violence related to sex, in particular. In Kebbi state’s statistics on acceptance of wife beating, the worst in the country, 87 percent of women believe it is acceptable for a man to beat his wife if she refuses to have sex with him, while 67 percent believe the same if she burns the food (DHS 2019). This very widespread belief that husbands are entitled to sex from their wives suggests that sexual violence may not be particularly stigmatized in this context and therefore women may feel as comfortable reporting it face-to-face as with the list method. The more normalized reporting of sexual IPV may also be a coping mechanism in a context where this form of IPV is widespread and tolerated. A potential alternative explanation could be that some might consider sexual violence as resulting from a husband’s passion rather than anger, potentially making it less shameful to report than physical or emotional violence, which can be seen as disciplining errant wives (Benebo, Schumann, and Vaezghasemi 2018).

In the study setting, IPV, including marital rape, is not criminalized. It seems likely that the different legal, religious, and social treatment of emotional, physical, and sexual IPV may contribute to their relative reporting sensitivity.

4.2. Characteristics of Misreporting

In this section, we briefly consider whether misreporting is random, and if not, identify the characteristics that predict it. To do this, we assess correlations between respondent characteristics and IPV, as reported under each of the survey methods. When the IPV-characteristic coefficient differs across methods, this implies systematic misreporting under at least one of the methods that is associated with the characteristic. This analysis provides suggestive evidence on the type of person that might misreport, potential reasons for misreporting, and the risks of drawing biased inferences from such data.

4.2.1. Estimation: Characteristics Associated with Misreporting

4.2.2. Results: Respondent Characteristics Predict Misreporting

We analyze misreporting by respondent characteristics that fall into three overlapping categories. These characteristics include (a) those most likely to motivate respondents to misreport (to understand why people misreport IPV), (b) variables typically targeted in IPV prevention interventions (to evaluate risks of estimating biased treatment effects), and (c) characteristics frequently cited as IPV risk factors. The first two categories of characteristics include perceptions of village gender norms, poverty and vulnerability, marital relationship quality, and personal gender attitudes. Additional risk factors considered include women’s education and literacy, age, labor-force participation, and other indicators of women’s bargaining power, where theory and empirical evidence are mixed on the relationship with IPV (Eswaran and Malhotra 2011; Bobonis, González-Brenes, and Castro 2013; Heise and Kotsadam 2015; Buller et al. 2018; Erten and Keskin 2018; Haushofer et al. 2019).

Table 3 shows the relationships between IPV and respondent characteristics by survey method. Results show several differences in coefficients across the methods, implying that a number of characteristics are associated with systematic misreporting of IPV.²² These results are also shown graphically in fig. A.1. For ease of interpretation, in table A.4 we also show the differences in mean IPV prevalence for respondents above and below the median of the characteristic, or by their indicator variable characteristic. However, given we are likely underpowered to detect the differences in this table, results should be interpreted with caution.

This subsection presents the results of descriptive analysis and, as such, the usual caution should be applied when interpreting results given that omitted variables or other sources of bias may be driving observed correlations.²³

Our results suggest that vulnerable women were more likely to report physical IPV under the list method than face-to-face. For emotional and sexual IPV, the same pattern existed but the differences were not statistically significant.

Greater marital relationship quality, measured using questions about satisfaction, empathy, and trust in marriage, was also associated with underreporting. We found that women who reported greater relationship quality were more likely to report IPV when interviewed using the list method than face-to-face; the method differences were statistically significant on two out of the three IPV questions considered. A potential reason for women who report being in a more trusting, supportive relationship to underreport IPV when asked directly is because the direct question forces her to confront an uncomfortable truth about her marriage, or she may feel disloyal by directly admitting her husband has been violent.

These results highlight a potential measurement bias issue for the growing number of policies being assessed on their IPV impact. For example, an intervention that improves relationship quality, as several IPV prevention programs seek to do, may have no impact on actual IPV rates. However, it may lead to a reduction in a wife’s willingness to directly report IPV for the reasons discussed above. This would bias treatment effect estimates, suggesting a reduction in IPV that was, in fact, an artefact of the survey method used.

We also examined whether common correlates of IPV remain stable across the method used. For some indicators of women’s intrahousehold bargaining power, we find some evidence of systematic misreporting of IPV, though no clear pattern across different indicators.

Concerningly, we observe some opposite-signed relationships, where the sign on the relationship between IPV and the respondent’s characteristic is positive for one method and negative for another. In Nigeria, these variables include wife’s vulnerability, literacy, and assets brought into the marriage. For example, we find suggestive evidence of a positive relationship between IPV and women’s education when IPV is measured directly, but a negative relationship when using the list method. We observe a similar relationship in Rwanda. This is also similar to results in Peru and India, where researchers found women’s education to be positively correlated with IPV when measured using the list method but negatively when measured face-to-face (Joseph et al. 2017; Agüero and Frisancho 2022). The opposite-signed IPV–education associations would imply opposing theories about women’s bargaining power as a cause of IPV.²⁴

Overall, the sensitivity of the relationships between IPV and several bargaining power and other indicators to the survey method used suggests we should be cautious when using direct IPV measures to understand descriptive and causal relationships, as misreporting may bias correlations and treatment effect estimates. The anonymity of the list experiment may minimize this reporting bias, though at the cost of precision.

4.3. Robustness Checks

In this section we report the results of robustness checks on the list method, followed by checks on the overall analysis.

4.3.1. Checks on List Method Assumptions

There are two assumptions that underpin the validity of statistical analyses of list experiments, and which, if they hold, should result in an unbiased difference-in-means estimator (Blair and Imai 2012). The first assumption is that of “no design effect,” meaning that the inclusion of the sensitive item in the list does not affect answers to non-sensitive items. The second assumption is of “no liars,” or that the answers given for the sensitive item are true.

Blair and Imai (2012) propose a likelihood ratio test to assess the probability that the first assumption is violated. The test is based on the assumption that the inclusion of a sensitive statement should not change the sum of affirmative answers to the non-sensitive statements, and that cumulative proportions of different respondent types should be non-negative (Bulte and Lensink 2019). If one of the proportions is negative, then the no-design-effect assumption has likely been violated. The rejection of the null hypothesis of no design effect using Blair and Imai’s statistical test would suggest there was a design effect. The test returns a Bonferroni-corrected minimum p-value of the test (Blair and Imai 2011).²⁵

In Nigeria, the test returned a minimum p-value of 1, suggesting there was no design effect for the three list questions.²⁶

There is no explicit test for the second “no liars” assumption. However, the most obvious threat to this assumption is when a respondent’s anonymity for their answer to the sensitive statement is compromised. This “ceiling effect” exists when a respondent’s true answer is affirmative for all of the statements, thus their affirmative response to the sensitive statement will no longer be hidden. The proportion of respondents answering with the maximum number of items (three and four for the non-sensitive and sensitive list groups, respectively) is within the range of other studies.²⁷ The presence of a ceiling effect likely indicates downwards-biased list estimates because other respondents would be likely to have understated their true number of affirmative answers to conceal their sensitive answer (Blair and Imai 2012).

4.3.2. Checks on Overall Approach

The two key assumptions underpinning the unbiasedness of the list experiment estimator have parallels for the experiment as a whole.

First, we checked balance on survey questions asked of all respondents after the experiment questions. If respondents were differentially affected by one survey method, perhaps indicating a design effect, we may observe them answering subsequent survey questions differently. We found very few statistically significant differences in answers to subsequent survey questions despite the many tests for significance, and any differences were small.

Second, we analyzed the main list method prevalence regressions with enumerator fixed effects, controls for pre-experiment imbalance, and controls selected using post-double-selection LASSO (Belloni, Chernozhukov, and Hansen 2014). Estimates remained consistent, with few differences in coefficients or significance, as shown in table A.7.

Third, although a general “no liars” assumption is untestable, we compared refusal rates across methods to indicate the potential relative risk of misreporting. We find very low refusal rates, the highest being 0.64 percent for the face-to-face sexual violence question. Refusal rates are shown in table A.8.

Additional indications that these results are robust are supported by the broadly similar patterns of misreporting by survey method observed over different violence types, countries, and list method designs, and in Rwanda, different genders, and perpetration and victimization questions.

5. Conclusion

This paper provided some of the first evidence that the prevalence of intimate partner violence can vary substantially depending on the survey method used to measure it. By randomizing individuals to answer using only one method, we avoid some pitfalls from earlier studies. Our results suggest that in some contexts, standard survey methods are likely to result in significant underestimates of the prevalence and welfare costs of IPV. The most widely used method, face-to-face interview, resulted in the lowest prevalence estimates, followed by the more private ACASI method (tested in Rwanda). Finally, the list method, which allows respondents to report anonymously, resulted in the highest prevalence. In Nigeria, women’s experience of physical IPV was 35 percent greater when measured using the list method than the direct method, and 180 percent greater than physical IPV rates reported in Nigeria’s DHS.

This paper also showed that misreporting was systematic and was correlated with indicators that theory suggests are risk factors for IPV and those often targeted in women’s empowerment programs, including vulnerability, marital quality, and education.

As other studies have found, direct measures of IPV can produce biased treatment effect estimates, for example, finding programs caused a reduction in IPV if measured face-to-face, but an increase if measured with the list experiment (Bulte and Lensink 2019). Given the growing number of studies measuring program impacts on IPV, our results highlight the potential risk of collecting measurement-error-prone data in generating unbiased correlations and treatment effect estimates.²⁸

Our finding of significant and systematic underreporting for some types of IPV in our rural African contexts contrasts with findings from Lima, Peru, where researchers found no overall difference between face-to-face and list method prevalence (Agüero and Frisancho 2022), and with research on sensitive topics in political science, which typically find lower levels of misreporting (Blair, Coppock, and Moor 2020). However, our findings are consistent with IPV misreporting observed in rural Burkina Faso, Vietnam, and India (Joseph et al. 2017; Bulte and Lensink 2019; Lépine, Treibich, and d’Exelle 2020), and in other sensitive health and sexual behavior surveys (Jamison et al. 2013; Traunmüller, Kijewski, and Freitag 2019). Overall, our results indicate that misreporting bias is, unsurprisingly, strongly affected by contextual factors, but can be substantial, and varies by type of violence.

How then should we measure IPV, given our results suggest that IPV is likely to be underreported when measured using the status quo face-to-face method? Policy makers and social scientists measuring sensitive topics could use similar randomized survey experiments to compare IPV reporting rates between the list experiment and direct methods to identify the magnitude of stigma, the types of people most likely to misreport, and the risk of generating biased treatment effects. For example, social scientists researching the impact of interventions on IPV might use such an experiment to sign potential reporting bias in their treatment effect estimates. Also, given DHS’s large sample sizes and use in multi-country analysis, perhaps such surveys should measure several IPV questions using the list method as well as the face-to-face method. This would allow data users to assess the likely extent of reporting bias by context, allowing them to make adjustments to inferences about true levels and correlates of IPV.²⁹

While the list experiment may reduce reporting bias, the method presents some analytical and administration challenges that may undermine its suitability for measuring IPV in some circumstances, and suggests the need for research into alternative approaches that also reduce misreporting. The larger variance associated with list experiment prevalence estimates makes it difficult to use as an outcome measure in smaller studies. Further, it is more time consuming to administer and requires patient, well-trained enumeration. Finally, there are ongoing debates about the method’s internal consistency and optimal design.

We suggest that social scientists measuring IPV consider using both direct questions and the list experiment to test the likely magnitude of underreporting; use the double list experiment to both reduce variance in prevalence estimates, and to apply the internal consistency test proposed by Chuang et al. (2021); and test alternative methods and survey administration conditions that may minimize reporting bias (for example, testing the effect of enumerators more clearly explaining data confidentiality and ensuring interview privacy, or of incentivizing and monitoring empathetic enumeration). Further research could also explore the feasibility and reliability of alternative survey methods such as the ballot method, or of other non-self-reported indicators of IPV, such as brain imaging or stress hormones.

6. Data Availability

The data underlying this article were provided by the World Bank and Government of Rwanda by permission. Data will be shared on request to the author with permission of the World Bank and Government of Rwanda.

Appendix

Figure A.1.

Nigeria: Relationships between IPV and Respondent Characteristics across Survey Methods

Source: Author's analysis from author's data.

Note: The figure shows correlation coefficients between IPV and respondent characteristic, by survey method.

Open in new tabDownload slide