Direct Effects

A 2018 National Academy of Sciences report on American policing begins its section on racial bias by noting the abundance of scholarship that records disparities in the criminal justice system. But shortly thereafter, the authors make a strange clarification: “In many cases there is little informative quantitative data on whether… policing is influenced by the racial or ethnic identity of citizens in a causal sense.”¹

On the one hand, troves of studies demonstrating racial disparities across a range of policing situations. On the other, a lack of data showing policing to be causally influenced by race. What accounts for the gap between evidence of racial disparities and proof of race as a causal influence on those disparities? The report presents a vignette that offers some clues:

[A] police officer may decide to stop and question or frisk a Black citizen but may decide not to question a White citizen, creating a racial disparity in stops… Based solely on measures of officer’s behavior, however, it is impossible to know whether this behavior was actually racially biased. If the Black and White pedestrians, for instance, acted differently as the officer approached (e.g., nervous versus calm), or if the officer encountered them in different surroundings (at night in an alley versus at noon in the park), or if the officer was searching for a suspect described as Black, an objective observer might conclude that the officer was simply responding to the situation at hand.²

The takeaway is that robust correlations in observational data offer satisfactory descriptive statistics, but they cannot, at least not on their own, meet the gold standard of scientific explanation—causal explanation. Only social scientists pursuing causal inference methods are equipped to answer sought-after “why” questions: Why are there racial disparities in policing? Is the disparity because of race or because of something else?

The experts behind the NAS report acknowledge that disparities are because of race in a certain, broad sense—they would not exist if a hierarchical race structure forged and fortified over centuries of racial exploitation, violence, and domination did not also exist. But the causal questions of quantitative social scientific interest are narrower, and seek to identify the effect of race on a particular decision-making process, policy, or outcome. In other words, such questions seek a shorter causal chain from race to some effect, often framed as the “direct causal effect” of race. In this framing, the “indirect causal effects” of race—which may variously pertain to the long histories of racial hierarchy encoded in residential segregation, wealth inequality, health outcomes, and so on—are intermediate outcomes between race and some final effect of interest, say, instances of police violence. These intermediate outcomes must be struck from causal analysis to isolate the direct effect of race. Failing to do so, the reasoning goes, would open the statistical floodgates and force us to incorporate the effect of all factors bearing any causal relationship to race. And surely that won’t do, when the task is to precisely measure the effect of race itself on a particular outcome.

In these formulations, the “direct effect” of race is supposed to capture something over and above backdrop racial disparities. Their inquiry is into how race causes some outcome, in excess of the broad racial stratification in housing, education, health, and so forth. In the literature, quantification of this “excess” yields an estimate of discrimination on the basis of race.³ Research that addresses this phenomenon typically proceeds by first identifying a particular racial inequality, then causally explaining it, and finally measuring the amount that is specifically “due” to racial discrimination. What is attractive about this methodology is that it appears to transform complex normative and theoretical questions—how does race structure our social world? what is racial discrimination?—into scientifically tractable efforts at hunting causes and effects.

But the dominant methodology rests on logic that does not work. No amount of meticulous experimental set-up and statistical practice can rescue quantitative social scientists from needing to make assumptions that amount to substantive claims about what race as a social category is and consequently what racial discrimination as a distinctive moral and legal wrong is. Any distinction between “direct” and “indirect” causal effects of race is inherently ambiguous, unless it is accompanied by answers to such basic theoretical and normative questions. This error of ambiguity lies at the core of causal interpretations of racial discrimination. Causal methods naturalize the distinction between direct (and thus discriminatory) effects of race and indirect (and thus legitimate) effects of race, creating what looks like a successful way to bootstrap formal statistical or experimental methods into substantive moral and political judgments. Showing how these approaches perform this remarkable sleight of hand will allow us to better see when social scientific methods are—and are not—well-suited to answer our most urgent questions about injustice.

In what follows, I discuss a variety of social scientific methods that attempt to compute direct effects of race. Among these, I pay special focus to approaches that make use of causal directed acyclic graphs (DAGs): diagrams that draw causal connections between two entities as arrows between nodes. This is not because I find causal DAGs notably more misleading or wrong-headed than the rest; instead, it is because I find the framework to provide the most transparent articulation of causal inference.

How to draw a DAG

Isolating a direct effect of race requires making a distinction between direct and indirect effects, but the distinction is so elusive that it verges on illusory. Any expert DAG drawer will tell you that (almost) any direct effect is really an indirect effect if you zoom further into the relevant causal mechanism. For instance, suppose we are interested in figuring out how someone’s wages are causally dependent on their race. In causal inference terms, we want to compute the direct causal effect of race on wages. One obvious way that race “causes” wages is via racially segmented labor markets. There are two options for rendering this causal fact in DAG form. One representation (see the figure labelled “Complex” below) draws the causal factor in explicit node form as, say, position in segmented labor market, which mediates an indirect effect of race on wages, marked by the black arrow. Alternatively, the causal factor can be embedded into the functional form of the structural equation—think of it as a formula that captures how the causal dependency works—that accompanies the direct effect of race on wages, shown as the blue arrow (see the figure labelled “Simple”). The underlying thought here is that, intuitively, the question of how race “causes” wages takes place against background conditions in which basic structural features of the labor market are assumed to be held fixed, and thus do not need explicit representation in node form. There is no more need to draw out a node representing structural labor market conditions than there is a need to amend the causal arrow match → fire by drawing out the node oxygen.

The Simple and Complex DAGs present two different representations of the causal relationship between race and wages. Note where their disagreements do and do not lie. Empirical facts and facts about causation do not determine whether the causal factor position in segmented labor market should be represented as a node (as in the Complex representation) or embedded within a structural equation (as in the Simple one). Adherents of both the Simple and Complex representations agree that racial segmentation of the labor market mediates some of the causal effect of race on wage earnings. Where they disagree is over which causal factors should be explicitly represented as nodes in the diagram and which should be assumed away in the background.

What are the stakes of drawing the line between foreground and background one way or another? The direct effects of race on wages in Simple and in Complex correspond to two different theoretical quantities. The direct effect of race on wages in Complex (the blue arrow) covers strictly fewer causal effects than its counterpart in Simple. In Simple, the indirect effects noted in Complex by the black arrows are instead lumped into the blue direct effect arrow. And so a choice between these two causal diagrams is also a choice between two different direct effects of race, and—if you subscribe to the direct effects view—two different assessments of the extent of racial discrimination on wages. No disagreement about causal facts is needed for controversy to ensue—just a disagreement over what effects should be taken for granted as a part of how race causally works as opposed to what effects should be considered as in principle distinct from race itself.

This simplified example illustrates a central component of my challenge to the causal inference approach to racial discrimination. The notion of a direct effect of race on some outcome is inherently ambiguous, because adding indirect pathways changes the “direct effect” quantity that is being measured. Furthermore, drawing additional indirect pathways need not show a commitment to the existence of more causal mechanisms. It may simply reflect an invisible shift in the line between foreground and background of a DAG, explicitly foregrounding and nodifying causal factors that had previously been relegated to the background, or vice versa. Model ambiguity is a genuine occupational hazard for the DAG-drawer.

At this point in the argument, an experienced quantitative social scientist may be frustrated. Haven’t I just made the trivial claim that adding more features into one’s model of how race causally affects some outcome can steal away some of the causal effect of race that “directly” affects the outcome? If so, why did I bring in these causal DAGs at all? I could have illustrated the point with causal interpretations of simple regression analyses, where adding more variables can depress the size of the coefficient for, and thus the causal effect of, race. This point, as a claim about statistics and modeling, is broadly accepted—even banal—among causal inference practitioners. But I want to suggest that the point is not just a matter of good statistical hygiene, of sorting out causal effects into ones due to race “directly” or due to race “indirectly” or due to possibly confounding “non-race” factors. Our social scientific statistical analysis is supposed to get at something in the really existing world. What is all this statistical finagling, disentangling the direct and indirect effect of race, trying to say about that world? On this, I think the causal methodologists have been less explicit.

A methodology that purifies out the “indirect effects” of race to get at the remaining nugget of discrimination given by the “direct effect” of race can be supported by two different conceptions of discrimination: a negative account and a positive account.⁴ Discrimination qua direct effect of race can be defined negatively as the remaining effect of race on some outcome after all of the indirect effects of race have been accounted for. This definition is, by construction, model-relative and depends on what I have called the foreground/background line that determines which causal factors are explicitly represented as distinct nodes and which are relegated to the background.

Alternatively, discrimination qua direct effect of race can be described positively as capturing a particular mechanism by which race directly “causes” certain outcomes—usually, people opt for the “mechanism” of racial animus, commonly taken by social scientists to be some kind of antipathy, prejudice, or generic “distaste” for members of some racial group.⁵ This definition does not suffer from model relativity, but if it can be computed using standard causal methods, it can only do so by fiat: the just-so assertion that the particular racial disparity that remains just is caused by racial animus.

Most causal treatments of racial discrimination want to have their cake and eat it too: they hope that, by sifting through all the indirect effects of race and attempting to eliminate non-race confounders, they will ultimately arrive at the discriminatory causal mechanism that is racial animus. The negative account and the positive account thereby happily coincide: the former’s methods can avoid the pitfalls of model relativity using the latter’s posited mechanism of racial animus to judge what should count as an indirect or direct effect of race. It is then no surprise that disagreements about discrimination cash out as a series of statistical skirmishes: what other factors must be controlled for such that all that remains that can cause a racial disparity is pure racial animus? Equivalently, for the DAG-drawer: how can the background/foreground line be drawn so that only non-animus factors are depicted as nodes in the foreground, thus leaving racial animus as the only effect transmitted along the direct effect arrow?

The positive account thus relies on the thought that, once all other factors have been controlled for, the only mechanism left to explain racial disparities is racial animus. But notice that still, it is only due to background knowledge about race and racial injustice that racial animus is hypothesized as the causal factor that accounts for the statistical gap.

Consider the famous résumé audit studies, which deliver identical fictional résumés to employers under differently raced names such as Jamal and Greg. (I’ve written about these previously here.) If all disparities that remain after instituting appropriate controls were due to animus, then, strictly speaking, résumé correspondence studies would reveal anti-people-named-Jamal animus rather than a general anti-Black animus. Furthermore, it isn’t the existence of the gap per se that identifies the animus mechanism—a parallel audit study of Katies and Claires showing a gap in the callback rates in favor of Katies would not proffer evidence of an animus against Claires. This is because we have no reason to think that there is a coherent group that is “Claires” which can properly be the target of animus or discrimination in the first place. In these studies, then, it isn’t the plain fact of a disparity that proves the existence of animus, but rather the underlying thought that there could be nothing else to cause such a disparity but the mechanism that is racial animus. But this is to return to the negative account of discrimination qua direct effect: the direct effect of race is defined as what remains after all other causal effects have been accounted for.

The failure of the positive account of discrimination suggests a reinterpretation of the audit study. If no experimental setup can ensure the sole operation of an “animus” mechanism, and the direct effect of race must be defined negatively, then the problem of model relativity rears its head once more: which causal factors should be foregrounded in the diagram, drawn out of the direct effect of race and represented as distinct causal factors conveying indirect effects of race? This can be understood as a very practical question about the setup of the audit study: which features should be equalized across Jamal and Greg to ensure that the direct effect of race can be correctly identified?

Left with a wholly negative conception of discrimination, we have little clue about what the direct effect of race really is. Without a sense of what the “direct effect” of race refers to in the world or why we have normative reason to care about it, it is unclear what the purification task of separating out indirect from direct effects is even trying to aim at. Without a guiding principle according to which we decide whether causal factors should be represented as explicit nodes in the foreground or embedded in the background, the exercise of distilling a direct effect of race out of indirect effects is ungrounded and arbitrary. The negative definition becomes an empty formalism—the direct effect of race is everything that isn’t indirect. And the indirect effects? Well, they’re just what should be purified out to get to the direct effect.

Many quantitative social scientists will reject the circularity: Jamal and Greg of the résumé audit study have been made identical in every respect but their racialized names, and so there are simply no more effects left to account for. If there is any solid purification process that can get at the direct effect, it’s this one.

But this is, of course, not strictly true. After all, the design of the audit study in fact shows that Greg and Jamal are not indiscriminately made identical. The audit study is not conducted with Gregs and Jamals applying to jobs noting that they both like the color green, have pet guinea pigs, listen to alternative rock, and play basketball on the weekends. They are rather made identical in one key respect: their résumé contents. This particular choice of equivalences belies a substantive moral point about how Jamals and Gregs ought to be treated by employers when they are identical in this particular respect. Further, the audit study’s silence on whether other features between Jamal and Greg are strictly identical suggests something further: even if Jamal and Greg had different favorite colors, that would still not be a reason to justify differential treatment in hiring. But neither of these are straightforward operationalizations of a formal prescription to “make Jamal and Greg equal.” They are substantive accounts of when Jamals and Gregs should be treated the same.

Audit study proponents are right that it is morally objectionable that Jamals are treated worse in the labor market even when they share identical résumés with Gregs. Where they are wrong is in suggesting that this difference in treatment corresponds to a direct effect of race negatively derived by following a formal rule that defines racial discrimination. Instead, they have filled out a substantive view about when job candidates ought to be treated equally, and interpreted the resulting racial disparity against Jamals as discriminatory. More broadly, they make a mistake in generalizing the correct finding that the audit study effect gives evidence of an instance of discrimination into the broader claim that the audit study is one of the cleanest experimental setups that manages to instantiate what discrimination is. As a standard of nondiscrimination, the racial disparities discovered in audit studies make for a low bar. That Greg and Jamal are identical in these respects and still Jamal is treated worse surely counts as evidence of racial discrimination, but the existence of this gap is by no means what discrimination is.

The negative account of the direct effect of race operationalizes the thought that nondiscrimination requires similarly situated people be treated similarly. Nondiscrimination on the basis of race says something further: that similar individuals (and groups) must be treated similarly despite the fact that they will be different in at least one significant respect: in their race. This presents an immediate challenge to the task of figuring who is similar to whom. And because of the kind of social category that race is, individuals who seem formally similar may not be substantively similar, and vice versa. So, we have to ask ourselves: how should the social fact of racial difference affect our task of coming to an appropriate sense of similarity?

One answer might be that it shouldn’t. We can carry on as before with a vague sense of “rational criteria” for employment, arrest, prosecution, and so on, and the demands of non-discrimination on the basis of race can be met if the police or the hiring committee apply whatever rules they find to be “rational” equally among Blacks and whites.⁶ Of course, to take this task is to translate an ethical notion of “similarity” into a purely instrumental notion of rationality. This is all fine and well and qualifies as an interpretation of the concept of discrimination. But it is important to underline that it is a substantive moral view about what it is for two individuals to be similar despite being differently raced, not a mathematical operationalization of a purely formal notion of “similarity.”

If discrimination is taken to be equivalent to some direct effect of race, differently raced individuals are similarly situated if they are similar across all “non-race” nodes. A DAG, then, wears its substantive conception of similarity on its sleeve. To draw a mediated effect of race is to draw a causal factor that is distinct from race itself and that carries an effect of race that isn’t the direct effect of race—and thus isn’t discrimination on the basis of race. In the case of the earlier DAG, Complex, it is to assert that facts about one’s position in the segmented labor market are not facts about race “proper.” Then, the effect of the racially segmented labor market on one’s hiring outcomes should not be considered a part of the “direct effect” of race on wages. In the case of the policing vignette given in the National Academy of Sciences report quoted at the top of this essay, to draw police perception of suspect activity as a distinct node is to assert that such “perceptions” can be distinguished from perceptions of race itself. To draw surveillance of public housing residents as a distinct node is to assert that choosing to surveil public housing projects is not a choice about race itself.

The pattern should now be clear: as more nodes crop up to mediate effects causally linked to race, more of the direct effect of race is sapped out, with each additional indirect effect of race explaining away some of the direct effect. Casual talk about discrimination is rife with claims like these: discrimination is not caused by race but by family criminal history, by neighborhood poverty levels, by occupational segregation. The disparity isn’t because of race but because of differences in proximity to crime. The racial wage gap isn’t due to discrimination on the basis of race; it’s due to the skewed racial composition of various labor market segments. These are the implicit assumptions that justify the statistical choices to condition on, say, the neighborhood of a stop and frisk action or to condition on, say, the identification of “suspect” activity. Here, statistical claims are always substantive claims about what race as a social category is and what it is to discriminate on the basis of it. At the extreme, drawing more and more mediated effects of race vacates out the direct effect of race until poof—it vanishes entirely, and along with it all reasonable complaint of racial discrimination.

Proponents of causal methods for identifying discrimination often point to the law as validating their approach. Indeed, such statistical analyses often form the evidentiary backbone of legal battles over discrimination. Analysts for each side spar over the validity of a few basic statistical statements, which often take the form:

P(Outcome | X, Race = Black) ≠ P(Outcome | X, Race = white)

This is the statistical gold mine (or land mine) of a discrimination case, to show that outcomes among individuals raced Black and individuals raced white are equal or not equal—once everything “relevant” (variable X) has been accounted for.

For the most part, warring experts are well-qualified to crank the statistical machinery, and interestingly, they mostly agree on how, as a matter of good statistical practice, to compute this estimand. Where they split is in their choices about what should and should not be included in the model or what should or should not be distinguished from the direct effect of race, in other words what features should or should not count toward this ultimate statistical comparison, given racial differences. They disagree on how to resolve model ambiguity, on how to equalize Greg and Jamal—on how to fill in the content of the statistical similarity test.

This task is, I have argued, a substantive one, and yet statistical skirmishes are rarely framed as such. Read through pages and pages of expert briefs and you will quickly find yourself inundated in the methodological nitty-gritty: adjusting for confounding, charges of omitted variable bias, race coefficients disappearing when additional covariates are accounted for, concern about conditioning on a collider, and so on. Courts accept methodologies that seek to purify out the causal effects of all “non-race” factors to get at the true causal effect of race itself to provide evidence of discrimination or its absence.⁷ In doing so, they suggest that proving disparate treatment simply amounts to showing that the direct causal influence of the protected category attribute in question is nonzero.

So the statistical parley goes on and on, while the reasoning behind why we are doing any of this, what theoretical construct we are trying to get at, and, most importantly, what any of this mathematical maneuvering has to do with the substantive question of whether some person or institution has committed the particular moral and/or legal wrong of discrimination on the basis of race is left opaque.⁸

To draw a DAG?

Much of the above can be taken in the spirit of a guide for the cautious DAG user: When you draw a causal diagram that looks to illuminate the causal effects of any social category that functions as an axis of privilege and subordination, you can’t claim to only be using statistical methods to expose social scientific causes and effects. From the start, you are engaging in good old social and political analysis, with all its attendant normativity. Any reasons you have to adopt one causal structure rather than another, or one set of structural equation functional forms over another, are grounded in judgments about how the social world works. These reasons include substantive normative judgments about how race figures in our world and concern for a set of practical political projects regarding race that your theoretical explorations might be in service of. This is precisely what makes defining racial discrimination as the direct effect of race so deceptive. It casts a normative dispute about what kinds of actions should be deemed discriminatory as an empirical dispute about causes. And figuring causes? Well that’s the gold standard of scientific inquiry. We know how to do that.

The last challenge I will press concerns this basic claim. I worry that we do not know how social categories constituted by unjust social relations act causally in the world—at least not well enough to be able to render them in familiar causal representations. My critique arises from a sense that race “causes” wages in a wholly different way from how striking a match “causes” fire or how smoking “causes” lung cancer. Methods largely built to investigate these latter causal phenomena might simply be unsuited to investigate the former.

Mostly, though, I question whether it makes sense to depict race as a category that produces various causal effects—each independent of the others and none constitutive of what the category of “race” is in our world. For example, claims such as “X% of Black people killed at the hands of police are explained by the racial wealth gap, not race itself,” or “Y% of the racial disparity would remain in a counterfactual situation in which Black people share the same income distribution as whites” suggest that race produces effects in the world in a modular fashion.⁹ On the one hand, race “causes” police violence, and on the other, it “causes” a poor-skewed income distribution, such that the two can be mixed and matched: turn off the race-wealth link and you can measure the direct race-police violence and wealth-police violence links.

It’s true that your average statistician or econometrician won’t bat an eye at claims like these. So allow me: why would we expect the way race “causes” police killings to stay the same in a world in which Black people and white people had the exact same income or wealth distribution? Why would we expect any social causal dynamics in that counterfactual world to be the same as those in our world? Why would we even think that what Blackness is in that world is the same as what Blackness is in ours? If they aren’t the same, why do we care about these quantities at all?

In the end—and this is where I think causal inference hard-liners such as Judea Pearl are completely on target—the objects of our inquiry do not exist in the world of statistics. We are interested in whether there is, more broadly, racial discrimination, unfairness, injustice in our world and whether, more narrowly, these observational data of racial inequality are a manifestation of it. These are not inquiries about numbers or diagrams—we do not care that race is a causal factor in your diagram. They are questions about the actually existing social world: what happens in it, and what we think should happen in it. Diagrams and models of that world will only be as good as our theories of what’s in it and hopes for what should be.