A simple diagnostic measure of inattention bias in discrete choice models

1. Introduction

Survey quality has been declining for some time, with a primary concern being measurement error (Meyer, Mok and Sullivan, 2015). Non-response rates are often used as a primary signal of survey quality (Johnson and Wislar, 2012), but there are concerns even about the quality of data attained from respondents. One concern is inattention, particularly when choices are hypothetical (Murphy et al., 2005; Carlsson, 2011). Survey methods can lead to ill-considered selections, as the participant might be concerned with the minimum amount of effort required to finish the survey to obtain their participation fee or to devote time to other activities.

While comparing observed outcomes to random choice has been proposed to test the hypothesis of utility maximisation using revealed preference data (Bronars, 1987), there is, at present, no simple metric available to infer issues related to inattention bias in survey research. This problem is particularly acute for analysts relying on stated preference choice data because there are no universally agreed upon measures of model fit and performance, such as the R² used in linear regression models. Although there are a variety of commonly reported performance metrics, such as the percent of correct predictions, the likelihood ratio index (also called McFadden’s R² or the pseudo R²) and various indices suggested by authors such as Ben-Akiva and Lerman (1985), Cramer (1999) and Efron (1978), all have shortcomings as evidenced by the lack of consistency in terms of reporting across studies employing discrete choice models (that is, if any measure of model performance is reported at all).

This paper proposes a simple, easy-to-use method that produces an intuitive fit metric that is comparable across samples and studies. In particular, we propose a metric we call the random response share (RRS) that ranges from 0 to 1, where a lower number indicates a more informative model. This note is most related to recent research that has used instructional manipulation checks, or ‘trap questions’ to check whether an individual is reading questions on the survey. Where trap questions create an opportunity for revealed participant inattention, the objective of this article is to describe a method for researchers to infer participant inattention.

The remainder of this research note is organised into three parts. In the following section, we describe how the RRS fits into the data quality and inattention literature. We then validate the RRS concept using stated preference choice data where we can compare RRS across people who have and have not missed a trap question. The final section concludes.

2. Inattention bias

The notion that research participants do not always attentively respond to surveys is not new. Decades ago, Nichols, Greene and Schmolck (1989) identified two types of inattention bias: content responsive faking and content nonresponsivity. The key difference is the influence of the survey content: content responsive faking assumes that inattention bias is not all encompassing, and instead, participants simply respond via a basic heuristic. The discrete choice literature has developed a rich literature in response to this concern via attribute non-attendance (ANA) modelling. ANA has been studied substantially in the health economics literature and other methods have been proposed to address ANA in discrete choice settings (Hole, 2011; Hole, Norman and Viney, 2016). In essence, ANA reveals that some people do not place any utility weight (or do not pay attention to) particular attributes (e.g. Campbell, Hensher and Scarpa, 2011; LaGarde, 2013; Scarpa et al., 2013). Stated another way, the ANA itself is a method whose intention is to impose an a priori behavioural restriction onto choice data. By contrast, the RRS is a simple measure whose intention is to provide quick, meaningful information regarding the issues surrounding measurement error due to inattention bias.

Although methodologically related, our approach also differs from the research on ANA as this article focuses on the second type of inattention bias: content nonresponsivity. Rather than seeking to systemically identify heuristics in the survey data, this concern is driven by ‘careless responding’ (Meade and Craig, 2012) or, for our purposes, ‘random responses’. Research in the psychology literature has suggested multiple methods for identifying careless respondents, although none has proven to be a panacea (Curran, 2016). For example, although response time is related to respondent inattention, it is difficult to identify an ‘optimal’ response time (Börger, 2016).

Recent research in the discrete choice literature has generally identified poor respondents via questions with obvious answers, or ‘trap questions’ (Gao, House, and Xie, 2016). Although these trap questions are unlikely to perfectly identify inattention bias, participants who incorrectly respond to the trap questions are more likely to provide questionable answers to choice questions (Jones, House and Gao, 2015). These inattentive responses have the potential to inflate policy recommendations and willingness-to-pay estimates (Malone and Lusk, 2018). As such, we compare the results from our RRS model with stated preference data where a trap question was used to identify inattention bias.

3. Validation with stated preference choice data and a trap question

We now move to an empirical application where we explore whether RRS varies with an alternative measure of inattention produced via a trap question. Data for the choice experiment come from one of the monthly editions of the Food Demand Survey, an online survey where participants are asked to choose between nine randomly ordered options (two beef, two pork, two chicken, two non-meat alternatives and a ‘none’ option) for nine choices. The choice experiment is a simple ‘branded’ choice experiment where the only attribute varying across options is product price (Lusk, 2017). After completing the choice experiment, subjects were asked the trap question shown in Figure 1, which is similar to that used in the broader literature on trap questions.² Participants were asked the trap question directly after the choice questions so as not to induce protest-like or other reactive behaviour when answering the choice questions (Oppenheimer, Meyvis and Davidenko, 2009). The actual instruction provided to the respondent was embedded in a paragraph-long survey question. Reading only the first and last few lines would direct participants to answer the question based on their mood, but fully reading the instructions in the question would notify attentive participants to select ‘none of the above’.

Of the 1,017 people who successfully completed the survey, 378 (37.2 per cent) did not correctly answer the trap question. Table 1 provides a comparison of parameter estimates for the latent class logit models when one class estimates the deterministic portion of the indirect utility function and the other class is restricted to be random. Likelihood ratio tests indicate that relative to the traditional multinomial models, the restricted latent class models improve model fit for all three specifications (χ²_Correct = 1,449.6, χ²_Incorrect = 489.3 and χ²_{Full data} = 2,261.2). Responses from the participants who correctly answered the trap question were as good as random 33.5 per cent of the time. However, the probability a choice is randomly made nearly doubles for the participants who incorrectly responded to the trap question (62.5 per cent). That is, the RRS measure is nearly double for people who missed the trap question, suggesting that the proposed method is a valid measure of inattention.³

A key concern for choice experimenters is that participant inattention might inflate willingness-to-pay (WTP) estimates. From the full dataset, Figure 2 displays participant WTP for the traditional multinomial logit (MNL) model versus the models with an estimated RRS. Results suggest that random responses inflated WTP estimates by approximately 23 per cent. For example, the traditional model suggests that consumer WTP for steak was $4.14; when we control for random responses via the RRS, consumer WTP decreased to $2.93. The only exception was WTP for pasta, which interestingly increased by 28 per cent when we control for participant inattention.

4. Conclusion

While there may be no clear threshold that might identify when measurement error due to participant inattention completely invalidates survey results, our results imply that a non-trivial fraction of survey takers is inattentive, with their choice behaviour being statistically indistinguishable from random answers. Inattentive survey takers can influence model estimates, biasing willingness-to-pay estimates and resulting policy recommendations. For example, the WTP estimates for correct respondents were not statistically different between the traditional MNL and the restricted latent class model. However, the restricted latent class model for incorrect participants indicates that severe participant inattention can skew WTP estimates.

Although the metric proposed here represents a useful contribution for discrete choice modellers, the RRS is not a panacea against inattention bias. It focuses on the percentage of individuals who ignored all attributes and is not intended to provide a measure of ANA. It is also possible that the probability of randomness is related to the number of choices, meaning that cross-study interpretation of the RRS should be cognisant of differences in design of the choice experiment. This article is also not recommending that the RRS approach be applied instead of the ANA or mixed logit approaches, as those models may often provide a better fit. Regardless, this study highlights the problem of inattentiveness in stated preference data and provides a simple way for practitioners to determine the problem’s severity for their own research.

Footnotes

References

Author notes

Review coordinated by Iain Fraser