An Absolute Test of Racial Prejudice

Abstract

1. Introduction

Racial and ethnic disparities persist in law enforcement, the criminal justice system, education, employment, health care, and housing (National Research Council 2004). Two sources of such disparities—statistical discrimination and prejudice—are central to the literature on the economics of discrimination. To illustrate each concept, consider the problem of a police officer who decides whether to search the vehicles of stopped drivers amid uncertainty about the drivers’ guilt.¹ Disparities are driven by prejudice if the officer has a utility motive for treating drivers differently, for example, because of their race (Becker 1957). Alternatively, disparities are driven by statistical discrimination if the officer has an informational motive (Arrow 1973; Phelps 1972). This might happen when the race of the driver correlates with unobserved guilt.

Distinguishing between prejudice and statistical discrimination matters. For the policymaker who seeks to mitigate existing disparities, prejudice requires different measures to address than statistical discrimination. For example, a temporary policy of affirmative action could resolve the issue of statistical discrimination (Coate and Loury 1993) but would not resolve the issue of prejudice. This underscores the value of identifying prejudice in the presence of statistical discrimination.

Empirically distinguishing between the sources of disparity is challenging. Continuing with the example of police officers deciding whether to conduct vehicle searches, search rates may vary by driver race even though police officers are not prejudiced. Action-based tests that explain (differences in) search decisions as a function of characteristics observed by the researcher are susceptible to omitted variables bias. Conversely, they may fail to find discrimination if officers enshroud their prejudice by conditioning decisions only on correlates of race. Such action-based methods also cannot distinguish between sources of disparity.

Alternatively, outcome-based tests, originating with Becker (1957), compare the guilt of searched drivers across race. In practice, outcome-based tests face the hurdle of inframarginality: even if officers are unprejudiced and apply the same standard of proof across all drivers, the average guilt of searched drivers may differ by driver race because of differences in the guilt distributions.² Put another way, prejudice is identified at the margin of search, but only the average, or inframarginal, search is observed. The inframarginality problem is first resolved by Knowles et al. (2001). They show that a comparison of average guilt, or hit rates, provides a robust test for prejudice in an equilibrium model where officers choose whether to search and drivers choose whether to carry contraband. Absent prejudice, hit rates are equalized across all drivers.³ In other words, equilibrium reasoning implies that every search is marginal, so that marginal and average searches (or rather, their expected outcomes) are equal.

However, Anwar and Fang (2006) provide convincing empirical evidence that the assumed equality between marginal and average outcomes may be violated in practice. In particular, they show that black, white, and Hispanic officers in their data exhibit markedly different search behavior from one another, and those who search more have lower hit rates. Leveraging this variation in search behavior across officers, they propose an alternative test of prejudice based on a rank order of search (or hit) rates. Suppose an officer, say a, searches minority drivers more than another officer b. Then officer a has a relatively lower threshold than officer b for searching minority drivers. If officers a and b have the opposite ranking of search rates among white drivers, then one of the officers must be prejudiced against one of the groups of drivers. Thus the test can find evidence of prejudice, but only in a relative sense: it cannot disentangle hypotheses of who is prejudiced against whom. In contrast, a test that can identify both the perpetrator and the victim of prejudice is henceforth referred to as absolute. In independent and contemporaneous work in the isomorphic context of judicial bail decisions, Arnold et al. (2018) introduce (partially) absolute tests using continuous variation in the judges’ decision threshold. Their insight is that popular instrumental variable (IV) techniques identify (an average of) local effects, which can in the limit be equated with (an average of) the search costs underlying prejudice.

This article introduces an absolute test for prejudice that requires only a minimal source of variation in a model that nests the above literature. The key intuition of the test is that each officer’s actions and outcomes generate a point on a concave “return possibility frontier,” whose slope at that point is equal to the officer’s search cost. Variation along a return possibility frontier (henceforth RPF) provides information about search costs, and discrepancies in search costs constitute prejudice. The test finds that an officer is prejudiced against, say, minority drivers relative to white drivers when the lower bound for the officer’s search cost on white drivers exceeds the upper bound for the officer’s search cost on minorities. In that case, the officer uses a higher standard of evidence when deciding whether to search white drivers.

The model of Knowles et al. (2001) is a special case where the RPF is linear in equilibrium; then the absolute test reduces to their hit rate test. The model of Anwar and Fang (2006) is a special case where the RPF is strictly concave, so that the search rate is a perfect proxy for the search cost. Even then, the absolute test is more powerful than their rank order test in two ways. First is statistical power: it can find evidence of prejudice where the rank order test does not. Second is explanatory power: it determines the direction and (bounds on) the magnitude of prejudice. This is because the absolute test leverages information on both actions and outcomes, whereas the rank order test is essentially an action-based test.⁴ The model of Arnold et al. (2018) is a special case where search decisions satisfy monotonicity, in the sense of Imbens and Angrist (1994). Combined with the decision rule, monotonicity implies a single RPF (by driver race), but the converse does not hold. The absolute test also relies on an IV, since it uses exogenous variation in the search rate in order to trace out the RPF. It converges to a version of the continuous instrument tests as the instrument becomes sufficiently local, while remaining consistent without further assumptions when the instrument is discrete, or even binary. This accommodates important sources of variation, such as policy interventions and the officer race instrument used by Anwar and Fang (2006). In fact, the absolute test only requires a first stage for some race of drivers. I formally elaborate the connections to each article in Subsection 4.3.

Concavity of the RPF also provides a new specification test of the model. This strengthens the testable implication, discussed in Anwar and Fang (2006) and extended in concurrent work by Gelbach (2021), that search rates and outcomes are negatively associated within each driver race.⁵ Namely, concavity of the RPF implies a negative association between search rates and expected outcomes, but not vice versa. Gelbach (2021) derives the additional implication that the average outcome among a group of searched drivers must be at least as high as the average search cost of the officers who search them. Empirically, he finds violating evidence in the application of Arnold et al. (2018). This suggests the potential usefulness of more general models.

In an extension to the basic model, I relax the assumption that all officers receive the same quality of information over drivers. For example, some officers may be more skilled, experienced, perceptive, or familiar with certain drivers than others, which lead to more effective search decisions. In short, I say that such officers are more discerning. Discernment is a serious concern for existing tests for prejudice. These hinge on the assumption that observed variation in search rates corresponds to variation in search costs alone, even though two officers with equal search costs but varying discernment could likely also exhibit different search rates. Furthermore, any assumption about discernment (or its absence) is logically separate from the standard IV assumptions of random assignment and exclusion. Discernment has an intuitive implication in my model: it expands the RPF. A version of the absolute test remains valid for officers that are assumed to be more discerning. Additionally, discernment has a simple testable implication: a more discerning officer who searches less has a higher hit rate. While my model is agnostic about the causes of discernment, varying discernment may itself represent a form of discrimination. Aigner and Cain (1977) suggest (exogenous) differences in signal precision across race as a source of labor market discrimination, and more recent work by Bartoš et al. (2016) studies attention—in other words, the choice of discernment—as an additional form of discrimination.

Empirically, I revisit the application of Anwar and Fang (2006). The striking feature of their dataset is that officers of different races and ethnicities have consistently different search patterns. White officers search drivers of every race more than Hispanic officers, who in turn search more than black officers. Thus, assuming that stops are randomly assigned, officer race effectively acts as a discrete IV. This dataset provides an ideal setting to apply my new methods because (a) the discrete instrument is not amenable to the continuous instrument tests of Arnold et al. (2018), and (b) a finding of prejudice would be the first in the dataset of Anwar and Fang (2006) and would stem from an improvement over their rank order test in practice.

In practice, my test uncovers the first suggestive⁶ evidence of prejudice in the Anwar and Fang (2006) dataset—specifically, black officers appear to exhibit prejudice toward Hispanic drivers. This illustrates the power of the test yet also raises important data-driven limitations. The test could not have found evidence of prejudice by white officers regardless of outcomes, not just in spite but because of the fact that they are uniformly the most stringent searchers; therefore it is always possible to rationalize white officers’ decisions by presuming that they have a low but equal threshold of search for drivers of every race.⁷ In that case, white officers would be strict, but not prejudiced. The data preclude evidence of prejudice by Hispanic officers for another reason: their relative behavior violates the basic model’s assumption that all officers are equally informed about the distribution of outcomes when deciding whether to search. Viewed through the lens of discernment, the data suggest that Hispanic officers are less discerning with white and black drivers, and more discerning with Hispanic drivers.

I examine the role of officer experience both as a potential explanation for the above violation and as a possible confounder for the other results. This is motivated by the facts that Hispanic officers in the data are on average less experienced than their white or black counterparts, and experience would correlate with discernment if officers (un)learn who to search. The evidence, if anything, points against the hypothesis of learning by experience. At the same time, the overall conclusions are relatively similar when disaggregated by officer experience. This confers some evidence of the absolute test’s robustness in practice.

2. Related Literature

The absolute test belongs to a large literature on testing for prejudice. Persico and Todd (2006) show that a version of the hit rate test of Knowles et al. (2001) remains valid when officers are heterogeneous but capacity-constrained in their searches. Dharmapala and Ross (2004) show that the hit rate test need not hold if drivers are not always observed by officers, or if the decision to commit crime is enriched beyond a yes-or-no decision to carry contraband.

In addition to the tests for prejudice already presented, Antonovics and Knight (2009) develop a parametric test for prejudice using variation in search behavior across officer race, similar in its source of variation to the previously discussed test of Anwar and Fang (2006). Simoiu et al. (2017) develop a parametric outcome test based on hierarchical Bayesian modeling. Alesina and La Ferrara (2014) develop a rank order test in the context of capital punishment decisions: if courts are unprejudiced, then the ranking of error rates within defendant race should be independent of the victim’s race. Anwar and Fang (2015) provide a model of prejudice in parole board release decisions that circumvents the inframarginality problem: prisoners are released as soon as their risk equals the parole board threshold, so that all released prisoners are marginal.

In the context of health care, Anwar and Fang (2012) provide a similar resolution to the inframarginality problem for emergency room diagnostic testing. When doctors optimally choose among a continuous set of diagnostic tests, the expected risk given a positive test is equalized across tested patients. Chandra and Staiger (2010) develop a parametric outcome-based test for prejudice that addresses the inframarginality problem by conditioning on an estimated propensity to receive treatment. More broadly, my distinction between search intensity and search discernment also relates to a literature on identifying sources of inefficiency in health care, which disentangles the analogous effects of overuse and expertise in a parametric selection framework (Abaluck et al. 2016; Chandra and Staiger 2017). Such a parametric approach is less satisfactory in the context of policing, where limited information about each traffic stop is observed by the researcher. Finally, distinguishing between statistical discrimination and prejudice is also a concern in the large literature on labor market discrimination; see Charles and Guryan (2011) for an overview.

In the context of policing, recent work by Goncalves and Mello (2021) and West (2018) tests for and finds evidence of differential leniency, in which police officers have discretion over punishments and enforce observed infractions with intensities that vary by driver race. Differential leniency is intuitively similar to prejudice in the sense that officers may incur different costs of leniency toward drivers of different races. Yet differential leniency is also fundamentally distinct because it arises with perfect information, which precludes the possibility of statistical discrimination. Thus the inframarginality problem at the heart of the literature on testing for prejudice does not arise.

The article also relates to and contributes to an extensive literature on identifying treatment effects using IVs.⁸ The estimators of slope of my RPF are, in the end, simple Wald estimators. These have a causal interpretation as local average treatment effects, or LATE, under the additional monotonicity assumption of Imbens and Angrist (1994). Yet monotonicity is not necessary for my results. In this sense, my work relates to de Chaisemartin (2017) and Frandsen et al. (2019), which extend the applicability and interpretation of IV methods beyond monotonicity.

However, my model is not weaker than the Imbens and Angrist (1994) model. Rather, I leverage the significant structure on decision-making inherent to the definition of prejudice: officers (decision-makers) search (treat) when the expected benefits exceed a cost, and differences in this cost across race constitute prejudice. This creates a natural separation between treatment decisions and the underlying information structure. I make assumptions on the latter, which are implied by but need not induce monotonicity. Such assumptions about information structure are without content in the Imbens and Angrist (1994) model, which is agnostic about how or why treatment decisions are made.

My approach also relates to a parallel literature on IV with selection models, which provides a more structural basis for treatment decisions relative to monotonicity (Heckman and Vytlacil 1999, 2005). In particular, the derivative of the RPF (i.e., the marginal return) coincides with the marginal treatment effect (MTE) central to that literature. Yet there are several differences between the objects. First, while the RPF is inherently grounded in a latent index representation—an officer searches if the expected returns exceed the cost—this representation may differ across officer types (i.e., the instrument). Varying discernment provides a case where the RPF, and thus the MTE curve, varies with the instrument. Still, a version of my results holds. This is because my model imposes additional structure in other ways. First, the return from not searching (i.e., the potential outcome without treatment) is zero. Second, marginal returns on the RPF are diminishing; equivalently, each officer’s MTE curve is nonincreasing. This structure is intuitively characterized in terms of the RPF: each frontier is concave and passes through the origin, and more discernment yields an expanded frontier.

The absolute test can draw conclusions about the existence, direction, and even magnitude of prejudice by bounding the possible search costs of each officer. The empirical application shows that this is true not just in theory but in practice. Thus the article fits into a large literature on partial identification (see Manski 2007; Tamer 2010 for overviews). Perhaps most closely related in this area is the work of Mogstad et al. (2018), who propose a computational framework for partial identification in the previously discussed latent index selection model. In fact, versions of the absolute test in my basic model of Section 3 could be computationally implemented via their framework. In the context of testing for prejudice in policing, Hernández-Murillo and Knowles (2004) also derive a test using bounds. However, their bounds stem from the researcher’s uncertainty over the fraction of discretionary searches.

Finally, the model assumes that the officers’ objective function is to maximize expected returns net a cost of search. Other objective functions may underlie decisions, in which case the test (or the underlying RPF intuition) need not apply. In the context of policing, Dominitz and Knowles (2006) and Manski (2006) consider problems where the objective is to minimize the crime rate and a social cost function, respectively.

3. Model

Consider a universe $[0,1]$ of traffic stops, on which uppercase letters denote random variables and lowercase letters denote their realizations. Suppose that, for all traffic stops, the researcher observes the search decision $D∈{0,1}$⁠, the race of the driver $R∈{m,w}$⁠, the finite return from search $Y∈R$⁠, and the type of the stopping officer $Z∈{a,b}$⁠. For example, the search return Y could be an indicator or an amount of recovered contraband, and the officer type could be an officer identifier or characteristic.

I assume the data are generated as follows. Each stop has a guilt type G, which is recovered during a stop if and only if the officer conducts a search: $Y=D·G$⁠. Before deciding whether to search, an officer of type z observes the race of the driver R and an additional type-dependent signal S(z) that may be correlated with guilt. When $Z≠z$⁠, the signal S(z) represents what an officer of type z would have observed if assigned to the stop. Thus, whereas the driver’s race and guilt are assumed to be immutable features of the stop itself, other information could vary with the stopping officer. This accommodates situations where officers acquire or perceive different information.

Officers of type z use the available information $(R,S(z))$ to make search decisions that maximize the return Y minus a search cost $τ(R,z)$⁠, which can depend on both the race of the driver and the type of the officer. Let $D*(r,s,z)$ denote the decision rule used by officer type z observing stop information (r, s). The decision rule coincides with the realized search decision when evaluated at the realized stop characteristics: $D*(R,S(Z),Z)=D$⁠. 

In turn, differences in search costs across driver race constitute prejudice: 

A prejudiced officer’s search decision rule depends on driver race even when the expected returns from search do not.

Using the available data, the researcher seeks to assess the existence and magnitude of prejudice among officers. The unobservability of the signals S(z) hampers identification of the decision rule and leads to the inframarginality problem: the researcher does not directly observe the expected return among marginally searched drivers. The problem of assessing prejudice is further complicated by the possibility of statistical discrimination: the search rate of unprejudiced officers may still depend on race.

Statistical discrimination occurs when the joint distribution of guilt and signal $(G,S(z))$ differs across driver race R.¹¹

To identify prejudice in the presence of the inframarginality problem and statistical discrimination, the researcher derives testable implications of prejudice as a function of the observed distribution of $(D,R,Y,Z)$⁠. The result is a test. Tests vary in the precision of the null hypotheses they can reject. A test for prejudice should at least be able to reject a null of no prejudice. Adopting the terminology of Anwar and Fang (2006), such a test is relative: 

A relative test can identify that prejudice exists, but it cannot disentangle finer hypotheses about which officer types are prejudiced against which driver groups. A more specific test is an absolute one: 

An absolute test is also relative, but a relative test need not be absolute. Thus absoluteness is a desirable property of a test.

The remaining assumptions ensure the existence of an absolute test. First is a standard instrument exogeneity assumption on officer type, imposed within driver race. 

Assumption 2 is satisfied if (a) stops are randomly assigned to officer types conditional on driver race and (b) drivers cannot condition their guilt on the type of the stopping officer. In Appendix A, I also derive the test under a weaker conditional independence assumption, which is invoked in the empirical application. Next, I assume that the joint distributions of guilt and observed signal are the same across officer types conditional on each driver race. 

Normalization (Signals). For each pair (r, z):

1. Conditional on R = r, the signal S(z) has a standard uniform distribution.

2. The expected benefit from search $E[G|R=r,S(z)=s]$ is weakly decreasing in the signal realization s.

The normalization is without loss of generality since, within each driver race R = r, one can first transform any signal S(z) into a continuous single-dimensional signal $S′(z)$ that is perfectly negatively related with the expected returns $E[G|R=r,S(z)=s]$ and then define $S″(z)$ to be the (r-conditional) quantile of $S′(z)$⁠. The signal $S″(z)$ then satisfies the desired normalization, which I adopt henceforth unless otherwise noted.

4. Results

Section 4 proceeds as follows. In Subsection 4.1, I introduce the unifying construct of the RPF and present three lemmas that modularize the logic of the test. In Subsection 4.2, I introduce and discuss the test (Theorem 1) and a related testable implication (Theorem 2). In Subsection 4.3, I relate the model and results to those of the existing literature. All proofs are relegated to Appendix B.

4.1 The RPF

The frontier exhibits diminishing returns to search because it is attained by searching the most promising signals first. In other words, the RPF is a concave function of the search rate q.

For each driver race r, Assumption 1 implies that each officer optimally chooses a search rate and total return on their RPF. The location depends on the officer’s search cost: higher costs lead to less search. The next lemma ties together the (observed) search rate and total return and the (unobserved) search cost as a supporting hyperplane of the (partially observed) RPF for each driver officer pair (r, z). 

A supporting hyperplane of a set is defined by two properties. First, it contains at least one point of the set, in this case, the pair of means $(δ(r,z),ψ(r,z))$⁠. Second, it contains the set in one of its half-spaces, in this case $h(q;r,z)≥ρ(q;r,z)$ for all $q∈[0,1]$⁠. Relatedly, in what follows it is without loss of generality to assume that all officer types use a signal cutoff rule: for all pairs (r, z), $D*(r,s,z)=1$ iff $s≤δ(r,z)$⁠. This can always be obtained by a rearrangement of the signal realizations.

Figure 1 provides graphical intuition for the search cost characterization of Lemma 1. The supporting hyperplane is a geometric expression of the first-order condition that equates marginal returns to search with marginal search costs. The marginal return corresponds to the slope of the RPF. The marginal cost, which is constant and equal to $τ(r,z)$ by assumption, corresponds to the slope of the hyperplane.

Figure 1.

Intuition for Search Cost Characterization and Bounds for a Fixed Driver Race r, with Numerical Example. Suppose the researcher is interested in the search cost $τ(r,a)$ of officer type a on drivers of race r. Lemma 1 characterizes this cost as the slope of a hyperplane $h(q;r,a)$ that supports the concave RPF $ρ(q;r,a)$ at the point defined by officer type a’s search rate and total return, $A=(δ(r,a),ψ(r,a))$⁠. This is a geometric expression of the optimality condition for the search decisions of officer type a, namely that the marginal benefit of search (slope of the return frontier) equals the marginal cost (slope of the hyperplane). If the RPF was fully (or even locally) observed, this condition would effectively identify the search cost. Yet partial information about the RPF $ρ(q;r,a)$ at different search rates q still restricts the possible slopes of the supporting cost hyperplane and thus imposes bounds on the possible values of the search cost $τ(r,a)$⁠. In this case, the origin point 0 is observed by definition of the RPF, namely $ρ(0;r,a)=0$⁠. The point A lies on the frontier $ρ(q;r,a)$ by Assumption 1 and is identified from the data by Assumption 2. Similarly, the point B lies on the frontier $ρ(q;r,b)$ by Assumption 1 and is identified from the data by Assumption 2. By Assumption 3, the frontiers $ρ(q;r,a)=ρ(q;r,b)$ are equal, so B is also a point on $ρ(q;r,a)$⁠. The slope of the line segment between points 0 and A, numerically $0.06/0.3=0.2$⁠, is the average return over stops searched by a. Since the expected return of searched stops weakly exceeds the search cost, this is an upper bound for the cost $τ(r,a)$⁠. The slope of the line segment between A and B, numerically $(0.09−0.06)/(0.9−0.3)=0.05$⁠, is the average return over stops not searched by a. This is a lower bound for the cost $τ(r,a)$⁠. Combining, we conclude that $τ(r,a)∈[0.05,0.2]$⁠. By analogous reasoning for the RPF $ρ(q;r,b)$ of officer type b, the slope between A and B, numerically 0.05, is an upper bound for $τ(r,b)$⁠. There is no lower bound because there is no higher variation, and so we can only conclude that $τ(r,b)≤0.05$⁠.

Open in new tabDownload slide

Flipping the logic, information about the RPF imposes restrictions on the possible hyperplanes—that is, it imposes bounds on the possible search costs. These bounds then underlie the subsequent absolute test for prejudice. Assumption 2 identifies the supporting point on the RPF for each driver-officer pair. 

Note that, by definition, the RPFs intersect for all officer types and a fixed driver race at the corner points $q∈{0,1}$ where an officer never (or always) searches: $ρ(0;r,z)=0$ and $ρ(1;r,z)=E[G|R=r]$⁠. Lemma 3 extends this to the domain $[0,1]$⁠. Lemma 3 follows immediately from equal informativeness (Assumption 3). However, I state it formally in order to compare with a result under weaker conditions in the extension of Subsection A.1.

4.2 The Absolute Test and Testable Implications

The main theoretical contribution of the article is an absolute test for prejudice. 

The proof proceeds in two steps: bounding the possible search costs, and using the bounds on search costs to derive violations of a test’s null hypothesis. I now provide additional intuition and discussion for each step. The relation to the existing literature is deferred until Subsection 4.3.

In the second step, an (m, w, a)-absolute test uses the bounds on search costs to potentially refute the null hypothesis that $τ(m,a)≥τ(w,a)$⁠. This happens when an upper bound for $τ(m,a)$ is less than a lower bound for $τ(w,a)$⁠. In order for such upper and lower bounds to exist, officers of type a must search at least some drivers of race m, and fewer drivers of race w than officers of type b (possibly none at all). These can be interpreted as necessary conditions on the search rates. In particular, the test cannot find evidence of prejudice against an officer type that searches drivers of each race more than the other officer types. Such an officer type’s search decisions can always be rationalized with the same sufficiently low search cost across driver races. This issue could be resolved with additional information or assumptions about the RPF beyond the observed means. For example, a lower bound on average guilt $E[G|R=r]$ among drivers of race r would suffice for establishing the necessary (though possibly weak) lower bound.

A fundamental feature of the model is that each RPF $ρ(q;r,z)$ is concave in q. Intuitively, this occurs because the return to search is diminishing in the search rate when the most promising signals are searched first. Concavity of the RPF is also the main testable implication of the model. In order to capture its full implications, I consider three officer types a, b, c. In the case where only two officer types are observed, the third type can be taken as the null search type with search rate and total return equal to zero. 

Next, I discuss how the absolute test (Theorem 1), the testable implication (Theorem 2), and the model relate to other approaches in the literature.

4.3 Relation to the Literature

4.3.1 Knowles et al. (2001)

The extension to an absolute test is immediate.

Thus my model and results recover the hit rate test. However, it is worth noting that implementing the test literally as equation (13) would require data on stops, whereas equation (9) only requires data on searches. Additionally, my model treats the RPF agnostically, although it may of course be generated in equilibrium.

Point identification (or even a lower bound) for $τ(r,b)$ is not possible because the data do not rule out that the RPF ceases to be linear for $q>δ(r,b)$⁠. Next, I relate my model and results to previous work in the setting where signals are informative about guilt and thus the RPF is nonlinear.

4.3.2 Anwar and Fang (2006)

Anwar and Fang (2006) (AF) propose a test of racial prejudice based on a comparison of rank orders of search (or hit) rates across driver race. In the AF model, officers of type $z=a,b$ make search decisions according to Assumption 1 after being randomly assigned to stops (Assumption 2). The central feature of the model is that officers observe informative signals about guilt beyond driver race. In addition to my signal normalization, the AF model assumes that the expected benefit from search $E[G|R=r,S(z)=s]$ is strictly decreasing in the signal realization s. I term this strong informativeness since it imposes more than just a violation of signal uninformativeness (equation 7). 

When equation (16) is rejected, equation (15) implies that $τ(w,a)>τ(w,b)$ and $τ(m,a)<τ(m,b)$⁠. It follows that at least one of the officer types is prejudiced against one of the driver races: if a is not prejudiced against m relative to w, then b must be prejudiced against w relative to m. The rank order test cannot disentangle the two alternative hypotheses. Thus it is, by definition, a relative test.

I refer to this as search–success consistency. Search–success consistency is a testable implication of the AF model. In addition, it implies that the rank order test (equation 16) can also be implemented in terms of average returns rather than search rates. Both points are developed in their paper.¹³

I now show how the absolute test of Theorem 1 and the testable implication of Theorem 2 improve on the power of the AF rank order test and search-success consistency, respectively, summarized in equations (16) and (17). For this purpose, the power of a test is defined as the probability of correctly rejecting the null hypothesis, and all results are provided for the asymptotic case where the distribution of $(D,R,Y,Z)$ is observed. Note that the assumptions of the following theorem include the AF model as special case. 

1. The test of Theorem 1 is more powerful than the rank order test.

2. The testable implication of Theorem 2 is equal to search–success consistency with two officer types, and more powerful with at least three officer types.

The rank order test identifies a sufficient condition on search rates for a finding of prejudice. In turn, a necessary condition for the rank order test to find prejudice is the existence of a first stage $δ(r,a)≠δ(r,b)$ for each $r=m,w$⁠. My test shows that neither the rank order condition nor the existence of both first stages is necessary to find prejudice. Rather, a test may find evidence of prejudice with a first stage for one driver race, and a positive search rate for the other. Of course, a positive search rate can itself be interpreted as a first stage relative to the trivially observed return at zero.

The new absolute test also improves on the relative rank order test in terms of explanatory power. Namely, it identifies not just the existence of prejudice, but its direction. Combining with the improvement in statistical power yields: 

4.3.3 Arnold et al. (2018)

In parallel work in the context of judicial bail decisions, Arnold et al. (2018) (ADY) propose a test based on identifying weighted averages of racial prejudice. In their model, officer (or judge) types $z∈(a,b)$ make search (bail) decisions according to Assumption 1 after being randomly assigned to stops (defendants; Assumption 2). A difference in their approach is that the search decisions satisfy monotonicity, in the sense of Imbens and Angrist (1994). 

In words, the slope of the RPF between the means of two observed officer types is equal to the expected benefit of the stops searched by the stricter officer type but not the more lenient one.

Another distinction relative to other approaches is that their test requires a sufficiently rich source of variation. 

Following the discussion of Heckman and Vytlacil (1999), the normalization that equates the search rate (i.e., the propensity score) with the officer type (i.e., the IV) is innocuous. This is also analogous to a two-stage least squares interpretation, in which the search rate summarizes the exogenous variation induced by the officer type. Substantively, Assumption 6 ensures that any neighborhood around an officer type’s search rate contains marginally stricter and more lenient officers.

This implication underlies the tests of racial prejudice introduced by ADY.

In this case, the weighting scheme is a function of the data. Conversely, the weighting scheme can be chosen by the researcher. A second weighting scheme, $λ(z)=1$⁠, yields an average level of prejudice across officer types.

Each test statistic of the form (22) identifies an average measure of prejudice across officer types. As a test (23), such a measure is absolute in the sense that it can identify the direction of prejudice among at least some officer types, yet relative in the sense that it does not identify which officer types are prejudiced. Of course, finer tests are also possible given identification of the search costs via equation (21). With continuous variation, the absolute test of Theorem 1 is such an example, which point identifies search costs for almost all officer types. In practice, however, estimates by officer type may be imprecise relative to an aggregate measure when the number of officer types is large, as discussed in ADY.¹⁴ Conversely, a sufficiently rich source of exogenous variation may not be available. Therefore, I see the approaches as complementary.

I conclude the discussion with a comparison of the models. For this purpose, I abstract from the empirical source of variation (Assumptions 2 and 6) and the signal normalization. Then the ADY model consists of the decision rule (Assumption 1) and monotonicity (Assumption 5); my model consists of the decision rule (Assumption 1) and equally informative signals (Assumption 3). The next result shows that my model recovers the ADY model as a special case. For simplicity, I derive the result for two officer types and a fixed driver race. 

5. Application

This section revisits the empirical application of Anwar and Fang (2006). Their dataset consists of 906,339 stops and 8976 searches conducted by troopers of the Florida State Highway Patrol from January 2000 to November 2001. Their empirical insight is that white and minority troopers (henceforth officers for internal consistency) exhibit systematically different search behavior in the data. This motivates using officer race and ethnicity (henceforth race for simplicity) to implement their rank order test of prejudice. I use the same strategy to implement my absolute test of prejudice.

Validity of both tests requires that traffic stops are assigned to officers independently of their race. However, Anwar and Fang (2006) observe that officers of different races are systematically assigned to patrol in different locations and at different times of the day. They address the issue by resampling a balanced number of officers across race within each troop.¹⁶ For the reasons discussed in Subsection A.3, I proceed instead by weighting each stop by the inverse probability of its officer race assignment given a full interaction of driver race, troop assignment, and a dummy variable indicating whether a stop occurred during daytime (defined as 6 a.m.–6 p.m.). In the process, I remove the 55,843 observations from Troop H (which has no Hispanic officers in the data) and an additional 160 observations that are missing data on the time of stop. This leaves 850,336 observations and 8642 searches in the final dataset. I henceforth refer to these raw observations as the unweighted sample. Weighting these observations as described generates a pseudosample—henceforth the weighted sample—in which drivers of each race have an equal probability of encountering an officer race within and across each troop-daytime cell.

Table 1 provides sample means and standard deviations (SDs) of observed driver characteristics for the unweighted and weighted samples. The table also provides the maximum absolute standardized mean difference across pairs of officer race for each observed characteristic. The standardized mean difference is a common metric for assessing the balance induced by inverse probability weighting (IPW; and other propensity score-based balancing methods); a standardized difference over 0.1 is often taken as evidence of significant imbalance (Austin and Stuart 2015). In the unweighted sample, the means across officer race indicate that each driver race is most likely to be stopped by an officer of the same race. Additionally, Hispanic officers are less likely to conduct stops during the daytime and on out-of-state drivers. The means in the weighted sample indicate an improvement in balance for each stop characteristic (although the cases of perfect balance are an artifact of the assignment model). The balance on observables is also reassuring because the assignment model is sparsely estimated on stop time and location, which are to some extent beyond an officer’s control. While it is impossible to rule out selection into stops based on unobservables, this provides reasonable evidence that stops in the weighted sample is randomly assigned to officers of different races (Assumption 7). I henceforth maintain this assumption.

Table 2 provides estimates and standard errors (SEs) for the search rates, average returns, and total returns by driver and officer race. These outcomes are identified in the weighted sample by Lemma 5. The search rates and average returns to search in Panels A and B of Table 2 correspond to the same Panels in Table 1 of Anwar and Fang (2006). In addition, Panel C provides the total returns to search, which are a central part of my approach. These are obtained by multiplying the search rates in Panel A with the average returns in Panel B. Finally, Table 3 provides estimates and SEs for the slopes of the RPFs between each pair of officer races, for each driver race. These estimates are subsequently used to assess both the testable implications of Theorem 2 and to implement the absolute test of prejudice of Theorem 1. All SEs (and the underlying variance–covariance matrix) are computed by repeating the estimation procedure in 1000 bootstrap samples.¹⁷

Figure 2 plots the search rates against the total returns for each combination of driver and officer race, with subfigures by driver race. This provides a graphical summary of the point estimates in Table 2 and slopes in Table 3. The figure crystallizes two points. First, search rates are consistently and significantly ordered by officer race across drivers: white officers search the most, and black officers search the least. Second, the point estimates are not all consistent with the basic model of Section 3, which implies that the points in each subfigure should lie on a concave curve, that is, a single RPF (Theorem 2).¹⁸

Figure 2.

Search Rates and Total Returns by Officer and Driver Race in the Dataset of Anwar and Fang (2006). Average returns are given by the quotient. Thus the figure provides a graphical summary of the means in Table 2 and the slopes in Table 3. The model implies that the points within each driver race should lie on a concave and nondecreasing RPF. However, the estimates among all drivers violate concavity, and the estimates among Hispanic drivers also violate monotonicity. None of the violations are statistically significant. Additionally, all violations involve Hispanic officers. Therefore the implementation of the absolute test proceeds by only leveraging the variation among black and white officers. In this case, upper and lower bounds on the search costs of black officers for each driver race are given by the slopes of line segments connecting $(0,b)$ and (b, w), respectively. These identify the expected guilt of signal realizations that are searched and not searched by black officers. Upper bounds on the search costs of white officers are given by the slopes of line segments (b, w), which identify the expected guilt of signal realizations that are searched by white but not black officers. Upper bounds on the search costs of Hispanic officers are given by the slopes of line segments $(0,h)$⁠, which identify the average returns of Hispanic officers.

Open in new tabDownload slide

In particular, white officers search black and white drivers more than Hispanic officers and obtain higher average returns; similarly, Hispanic officers search Hispanic drivers more than black officers yet obtain a higher average return. Finally, white officers search Hispanic drivers more than Hispanic officers but obtain lower total returns. This is inconsistent with the additional constraint that guilt is binary, hence nonnegative.¹⁹ However, none of the violations are statistically significant at the 95% level. The last violation is marginally significant; the one-sided z-test that the total return between Hispanic and white officers on Hispanic drivers is negative has a p-value of 0.064. This is computed using the coefficient and SE in Table 3. Correcting for multiple hypotheses would only weaken formal evidence against the model.

It is, however, noteworthy that the possible violations all involve Hispanic officers. In other words, the search rates and total returns among black and white officers are consistent with the basic model for all driver races. Additionally, Table 1 shows that black and white officers in the weighted sample are more similar on observed characteristics, in particular their average age and years of experience. Therefore I implement the absolute test using only the variation between black and white officers, and I subsequently explore the potential role of experience in explaining discrepancies.

Table 4 also provides a heuristic summary of the absolute test of prejudice. The test finds evidence of prejudice when the bounds within an officer race column do not intersect. In that case, the highest possible search cost for one driver race must be lower than the lowest possible search cost for another, suggesting prejudice toward the former. For example, the search costs of black officers on Hispanic drivers are bounded between $[6.7,18.3]$⁠, while the search costs of black officers on white drivers are bounded between $[19.2,37.6]$⁠. Taking these estimates at face value suggests that black officers are prejudiced against Hispanic drivers relative to white drivers. This provides the first (suggestive) evidence of prejudice in the dataset of Anwar and Fang (2006).²¹ Furthermore, this demonstrates an improvement over their rank-order test in practice, since the identically ranked search rates across officer races are consistent with that test’s null hypothesis of no prejudice. Finally, this finding can be sustained under weaker assumptions about signal informativeness. Namely, the finding relies only on the assumption that black officers are at least as discerning on white drivers as white officers. No assumption about relative signal informativeness is required within Hispanic drivers because the invoked upper bound relies only on black officers’ own average returns, rather than a comparison of their outcomes to those of other officers.

To incorporate uncertainty in the bounds, it is straightforward to conduct a one-sided test that the lower bound on the search cost for one driver race exceeds the upper bound on the search cost for another. Furthermore, note that one can only hope to statistically reject the null hypothesis if the bounds derived from the point estimates are suggestive of prejudice. Therefore it suffices to consider whether the upper bound on the search cost of black officers for Hispanic drivers is lower than the lower bound on the search cost of black officers for white drivers. The p-value of this z-test is 0.43. In this case, the test uncovers suggestive but statistically insignificant evidence of prejudice.

The bounds on search costs underlying the absolute test still provide useful descriptive information. For example, the suggestive evidence of prejudice by black officers stems partly from the fact that they are particularly effective when searching white drivers. This suggests a high standard of evidence for white drivers, which may reflect more subtle manifestations of racism. At the same time, the lower bound on search costs for black drivers is not significantly different, and in fact almost provides suggestive evidence of prejudice against Hispanic drivers relative to black drivers (the upper bound on search costs for the former is 18.3, while the lower bound on search costs for the latter is 16.6).

The fact that the absolute test only provides suggestive evidence of prejudice by black officers should not be interpreted as evidence that black officers are more prejudiced than others. Rather, it is largely an artifact (and limitation) of the test. The test cannot find evidence of prejudice by white officers because they search every driver race more than black or Hispanic officers. Consequently, even though white officers are only half as successful when searching Hispanic drivers as any other officer-driver pair, it is possible to rationalize their search behavior with an arbitrarily yet equally low search cost for each driver group. Similarly, the test cannot find evidence of prejudice by Hispanic officers because the equal informativeness assumption relative to white officers is empirically violated for every driver race. Therefore, the estimates for white officers do not provide a useful reference point for bounding Hispanic officer search costs, even though white officers search more often within every driver race. A potential remedy, in either case, would be to learn the unconditional probability that drivers carry contraband via randomized search. This would provide a lower bound on either officer’s search costs, although such a bound may be weak in practice. Another remedy would be to identify and leverage additional exogenous variation in officer search behavior.

Next, I examine the role of officer experience as an explanation for varying search behavior and discernment, and I evaluate its potential effects on the preceding results. To motivate the role of experience, Table 1 showed that Hispanic officers are younger and less experienced on average, and Table 2 provided suggestive evidence that they are not equally informed. This suggests that discernment may correlate with experience. For example, officers may learn who (not) to search on the job, or career considerations may alter the incentives for discernment across experience.

I proceed by splitting the set of stops by officer experience. A stop is defined as being conducted by a “new” officer if experience on the date of the stop is below the median experience across all stops, and the stop is conducted by an “experienced” officer otherwise. The median experience in the dataset is 11 years. In order to ensure that stops are randomly assigned across officer race and experience, I repeat the previous weighting procedure using the interaction of officer race and the experience indicator as a proxy for officer type. In the process, I eliminate an additional 64 observations with missing experience information, and 87,484 observations from Troops A and Q because they do not include each type of officer. This leaves a total of 762,788 stops and 7957 searches conducted by officers from the eight remaining troops.

Table 5 provides the search rates, average returns, and total returns by officer race and experience. Table 6 provides the differences in search rates and returns across experience, holding officer race fixed. Overall, experienced officers appear to search less. Two marginally significant exceptions to this are that experienced black officers search black drivers more, and experienced white officers search Hispanic drivers more. If experienced officers were at least as discerning, then Theorem 6 implies that those who search less should have higher average returns; a similar argument invoking nonnegativity of outcomes implies that those who search more should have higher total returns if the additional searches are ever successful. In contrast, experienced white officers search black and white drivers less but have (marginally significant) lower average returns. Experienced white officers search Hispanic drivers more without an increase in total returns. Similarly, experienced black officers have both lower search rates and average returns on Hispanic and white drivers, although the changes in average returns are noisily estimated and insignificant. As a whole, this collection of estimates provides suggestive evidence against a hypothesis that officers learn how to search.²² An exception is that experienced black officers search black drivers more with an insignificantly higher average return. Finally, experienced Hispanic officers search drivers of each race less and attain higher average but lower total returns, which is consistent with equal informativeness across experience.