↳ Causality

March 2nd, 2020

↳ Causality

Honeysuckle

CLEAR MEANS

Evaluating evidence-based policy

Over the past two decades, "evidence-based policy" has come to define the common sense of research and policymakers around the world. But while attempts have been made to create formalization schemes for the ranking of evidence for policy, a gulf remains between rhetoric about evidence-based policy and applied theories for its development.

In a 2011 paper, philosophers of science NANCY CARTWRIGHT and JACOB STEGENGA lay out a "theory of evidence for use," discussing the role of causal counterfactuals, INUS conditions, and mechanisms in producing evidence—and how all this matters for its evaluators.

From the paper:

"Truth is a good thing. But it doesn’t take one very far. Suppose we have at our disposal the entire encyclopaedia of unified science containing all the true claims there are. Which facts from the encyclopaedia do we bring to the table for policy deliberation? Among all the true facts, we want on the table as evidence only those that are relevant to the policy. And given a collection of relevant true facts we want to know how to assess whether the policy will be effective in light of them. How are we supposed to make these decisions? That is the problem from the user’s point of view and that is the problem of focus here.

We propose three principles. First, policy effectiveness claims are really causal counterfactuals and the proper evaluation of a causal counterfactual requires a causal model that (i) lays out the causes that will operate and (ii) tells what they produce in combination. Second, causes are INUS conditions, so it is important to review both the different causal complexes that will affect the result (the different pies) and the different components (slices) that are necessary to act together within each complex (or pie) if the targeted result is to be achieved. Third, a good answer to the question ‘How will the policy variable produce the effect’ can help elicit the set of auxiliary factors that must be in place along with the policy variable if the policy variable is to operate successfully."

Link to the paper.

  • Cartwright has written extensively on evidence and its uses. See: her 2012 book Evidence Based Policy: A Practical Guide to Doing it Better; her 2011 paper in The Lancet on RCTs and effectiveness; and her 2016 co-authored monograph on child safety, featuring applications of the above reasoning.
  • For further introduction to the philosophical underpinnings of Cartwright's applied work, and the relationship between theories of causality and evidence, see her 2015 paper "Single Case Causes: What is Evidence and Why." Link. And also: "Causal claims: warranting them and using them." Link.
  • Obliquely related, see this illuminating discussion of causality in the context of reasoning about discrimination in machine learning and the law, by JFI fellow and Harvard PhD Candidate Lily Hu and Yale Law School Professor Issa Kohler-Hausmann: "What's Sex Got To Do With Machine Learning?" Link.
  • A 2017 paper by Abhijit Banerjee et al: "A Theory of Experimenters," which models "experimenters as ambiguity-averse decision-makers, who make trade-offs between subjective expected performance and robustness. This framework accounts for experimenters' preference for randomization, and clarifies the circumstances in which randomization is optimal: when the available sample size is large enough or robustness is an important concern." Link.
⤷ Full Article

February 3rd, 2020

Winter Night

VARIABLE DEPEDENCE

Debating the merits of large- and small-N studies

Sample size does more than determine the sort of methodology appropriate for a given study; theorists of social science have long pointed out that the number of case studies considered determines the sorts of questions researchers can analyze and the structure of their causal claims.

A 2003 paper by PETER HALL takes these debates further. In the context of comparative political science, Hall argues that the sort of methods researchers use should be consistent with their beliefs about the nature of historical development. From the paper:

"Ontology is crucial to methodology because the appropriateness of a particular set of methods for a given problem turns on assumptions about the nature of the causal relations they are meant to discover. It makes little sense to apply methods designed to establish the presence of functional relationships, for instance, if we confront a world in which causal relationships are not functional. To be valid, the methodologies used in a field must be congruent with its prevailing ontologies. There has been a postwar trend in comparative politics toward statistical methods, based preeminently on the standard regression model. Over the same period, the ontologies of the field have moved in a different direction: toward theories, such as those based on path dependence or strategic interaction, whose conceptions of the causal structures underlying outcomes are at odds with the assumptions required for standard regression techniques.

The types of regression analyses commonly used to study comparative politics provide valid support for causal inferences only if the causal relations they are examining meet a rigorous set of assumptions. In general, this method assumes unit homogeneity, which is to say that, other things being equal, a change in the value of a causal variable x will produce a corresponding change in the value of the outcome variable y of the same magnitude across all the cases. It assumes no systematic correlation between the causal variables included in the analysis and other causal variables. And most regression analyses assume that there is no reciprocal causation, that is, that the causal variables are unaffected by the dependent variable. The problem is that the world may not have this causal structure.

Small-N comparison is therefore far more useful for assessing causal theories than conventional understandings of the 'comparative method' imply. Precisely because such research designs cover small numbers of cases, the researcher can investigate causal processes in each of them in detail, thereby assessing the relevant theories against especially diverse kinds of observations. Reconceptualized in these terms, the comparative method emerges not as a poor substitute for statistical analysis, but as a distinctive approach that offers a much richer set of observations, especially about causal processes, than statistical analyses normally allow."

Link to the piece.

  • "Except for probabilistic situations that approach 1 or 0 (in other words, those that are almost deterministic), studies based on a small number of cases have difficulty in evaluating probabilistic theories." Stanley Lieberson's 1991 overview of the causal assumptions inherent to small-N studies. Link.
  • Theda Skocpol and Margaret Somers on "The Uses of Comparative History in Macrosocial Inquiry." Link.
  • Jean Lachapelle, Lucan A. Way, and Steven Levitsky use small-N process tracing to "examine the role of the coercive apparatus in responding to crises triggered by mass anti-regime protest in Iran and Egypt." Link. Andrey V. Korotayev, Leonid M. Issaev, Sergey Yu. Malkov and Alisa R. Shishkina present a quantitative analysis of destabilization factors in 19 countries during the Arab Spring. Link.
⤷ Full Article

September 9th, 2019

Original & Forgery

MULTIPLY EFFECT

The difficulties of causal reasoning and race

While the thorny ethical questions dogging the development and implementation of algorithmic decision systems touch on all manner of social phenomena, arguably the most widely discussed is that of racial discrimination. The watershed moment for the algorithmic ethics conversation was ProPublica's 2016 article on the COMPAS risk-scoring algorithm, and a huge number of ensuing papers in computer science, law, and related disciplines attempt to grapple with the question of algorithmic fairness by thinking through the role of race and discrimination in decision systems.

In a paper from earlier this year, ISSA KOHLER-HAUSMAN of Yale Law School examines the way that race and racial discrimination are conceived of in law and the social sciences. Challenging the premises of an array of research across disciplines, Kolher-Hausmann argues for both a reassessment of the basis of reasoning about discrimination, and a new approach grounded in a social constructivist view of race.

From the paper:

"This Article argues that animating the most common approaches to detecting discrimination in both law and social science is a model of discrimination that is, well, wrong. I term this model the 'counterfactual causal model' of race discrimination. Discrimination, on this account, is detected by measuring the 'treatment effect of race,' where treatment is conceptualized as manipulating the raced status of otherwise identical units (e.g., a person, a neighborhood, a school). Discrimination is present when an adverse outcome occurs in the world in which a unit is 'treated' by being raced—for example, black—and not in the world in which the otherwise identical unit is 'treated' by being, for example, raced white. The counterfactual model has the allure of precision and the security of seemingly obvious divisions or natural facts.

Currently, many courts, experts, and commentators approach detecting discrimination as an exercise measuring the counterfactual causal effect of race-qua-treatment, looking for complex methods to strip away confounding variables to get at a solid state of race and race alone. But what we are arguing about when we argue about whether or not statistical evidence provides proof of discrimination is precisely what we mean by the concept DISCRIMINATION."

Link to the article. And stay tuned for a forthcoming post on the Phenomenal World by JFI fellow Lily Hu that grapples with these themes.

  • For an example of the logic Kohler-Hausmann is writing against, see Edmund S. Phelps' 1972 paper "The Statistical Theory of Racism and Sexism." Link.
  • A recent paper deals with the issue of causal reasoning in an epidemiological study: "If causation must be defined by intervention, and interventions on race and the whole of SeS are vague or impractical, how is one to frame discussions of causation as they relate to this and other vital issues?" Link.
  • From Kohler-Hausmann's footnotes, two excellent works informing her approach: first, the canonical book Racecraft by Karen Fields and Barbara Fields; second, a 2000 article by Tukufu Zuberi, "Decracializing Social Statistics: Problems in the Quantification of Race." Link to the first, link to the second.
⤷ Full Article

June 2nd, 2018

Even With Closed Eyes

ARTIFICIAL INFERENCE

Causal reasoning and machine learning 

In a recent paper titled "The Seven Pillars of Causal Reasoning with Reflections on Machine Learning", JUDEA PEARL, professor of computer science at UCLA and author of Causality popup: yes, writes:

“Current machine learning systems operate, almost exclusively, in a statistical or model-free mode, which entails severe theoretical limits on their power and performance. Such systems cannot reason about interventions and retrospection and, therefore, cannot serve as the basis for strong AI. To achieve human level intelligence, learning machines need the guidance of a model of reality, similar to the ones used in causal inference tasks. To demonstrate the essential role of such models, I will present a summary of seven tasks which are beyond reach of current machine learning systems and which have been accomplished using the tools of causal modeling." 

The tasks include work on counterfactuals, and new approaches to handling incomplete data. Link popup: yes to the paper. A vivid expression of the issue: "Unlike the rules of geometry, mechanics, optics or probabilities, the rules of cause and effect have been denied the benefits of mathematical analysis. To appreciate the extent of this denial, readers would be stunned to know that only a few decades ago scientists were unable to write down a mathematical equation for the obvious fact that 'mud does not cause rain.' Even today, only the top echelon of the scientific community can write such an equation and formally distinguish 'mud causes rain' from 'rain causes mud.'”

Pearl also has a new book out, co-authored by DANA MCKENZIE, in which he argues for the importance of determining cause and effect in the machine learning context. From an interview in Quanta magazine about his work and the new book:

"As much as I look into what’s being done with deep learning, I see they’re all stuck there on the level of associations. Curve fitting. That sounds like sacrilege, to say that all the impressive achievements of deep learning amount to just fitting a curve to data. If we want machines to reason about interventions ('What if we ban cigarettes?') and introspection ('What if I had finished high school?'), we must invoke causal models. Associations are not enough—and this is a mathematical fact, not opinion.

We have to equip machines with a model of the environment. If a machine does not have a model of reality, you cannot expect the machine to behave intelligently in that reality. The first step, one that will take place in maybe 10 years, is that conceptual models of reality will be programmed by humans."

Link popup: yes to the interview. (And link popup: yes to the book page.) 

⤷ Full Article