↳ Metaresearch

March 28th, 2019

↳ Metaresearch

Experiments for Policy Choice

Randomized experiments have become part of the standard toolkit for policy evaluation, and are usually designed to give precise estimates of causal effects. But, in practice, their actual goal is to pick good policies. These two goals are not the same.

Is this the best way to go about things? Can we maybe make better policy choices, with smaller experimental budgets, by doing things a little differently? This is the question that Anja Sautmann and I address in our new work on “Adaptive experiments for policy choice.” If we wish to pick good policies, we should run experiments adaptively, shifting toward better policies over time. This gives us the highest chance to pick the best policy after the experiment has concluded.

⤷ Full Article

March 19th, 2019

Ideology in AP Economics

When the media talks about ideological indoctrination in education, it is usually assumed to refer to liberal arts professors pushing their liberal agenda. Less discussed is the very different strain of ideology found in economics. The normative import is harder to spot here, as economics presents itself as a science: it provides an empirical study of the economy, just as mechanical engineering provides an empirical study of certain physical structures. When economists offer advice on matters of policy, it’s taken to be normatively neutral expert testimony, on a par with the advice of engineers on bridge construction. However, tools from the philosophy of explanation, in particular the work of Alan Garfinkel, show how explanations that appear purely empirical can in fact carry significant normative assumptions.1 With this, we will uncover the ideology embedded in economics.

More specifically, we’ll look at the ideology embedded in the foundations of traditional economics—as found in a typical introductory micro-economics class. Economics as a whole is diverse and sprawling, such that no single ideology could possibly be attributed to the entire discipline, and many specialized fields avoid many of the criticisms I make here. Despite this, if there are ideological assumptions in standard introductory course, this is of great significance.

⤷ Full Article

January 26th, 2019

Bone Mobile


Learning about long-term effects of interventions, and designing interventions to facilitate the long view

A new paper from the Center for Effective Global Action at Berkeley surveys a topic important to our researchers here at JFI: the question of long-run effects of interventions. In our literature review of cash transfer studies, we identified the need for more work beyond the bounds of a short-term randomized controlled trial. This is especially crucial for basic income, which is among those policies intended to be permanent.

The authors of the new Berkeley report, Adrien Bouguen, Yue Huang, Michael Kremer, and Edward Miguel, note that it’s a particularly apt moment for this kind of work: “Given the large numbers of RCTs launched in the 2000’s, every year that goes by means that more and more RCT studies are ‘aging into’ a phase where the assessment of long-run impacts becomes possible.”

The report includes a summary of what we know about long-run impacts so far:

"Section 2 summarizes and evaluates the growing body of evidence from RCTs on the long-term impacts of international development interventions, and find most (though not all) provide evidence for positive and meaningful effects on individual economic productivity and living standards. Most of these studies examine existing cash transfer, child health, or education interventions, and shed light on important theoretical questions such as the existence of poverty traps (Bandiera et al., 2018) and returns to human capital investments in the long term."

Also notable is the last section, which contains considerations for study design, "lessons from our experience in conducting long-term tracking studies, as well as innovative data approaches." Link to the full paper.

  • In his paper "When are Cash Transfers Transformative?," Bruce Wydick also notes the need for long-run analysis: "Whether or not these positive impacts have long-term transformative effects—and under what conditions—is a question that is less settled and remains an active subject of research." The rest of the paper is of interest as well, including Wydick's five factors that tend to signal that a cash transfer will be transformative. Link.
  • For more on the rising popularity of RCTs, a 2016 paper by major RCT influencers Banerjee, Duflo, and Kremer quantifies that growth and discusses the impact of RCTs. Link. Here’s the PowerPoint version of that paper. David McKenzie at the World Bank responds to the paper, disputing some of its claims. Link.
⤷ Full Article

January 12th, 2019



Another kind of cybersecurity risk: the destruction of common knowledge

In a report for the Berkman Klein center, Henry Farrell and Bruce Schneier identify a gap in current approaches to cybersecurity. National cybersecurity officials still base their thinking on Cold War-type threats, where technologists focus on hackers. Combining both approaches, Farrell and Schneier make a wider argument about collective knowledge in democratic systems—and the dangers of its diminishment.

From the abstract:

"We demonstrate systematic differences between how autocracies and democracies work as information systems, because they rely on different mixes of common and contested political knowledge. Stable autocracies will have common knowledge over who is in charge and their associated ideological or policy goals, but will generate contested knowledge over who the various political actors in society are, and how they might form coalitions and gain public support, so as to make it more difficult for coalitions to displace the regime. Stable democracies will have contested knowledge over who is in charge, but common knowledge over who the political actors are, and how they may form coalitions and gain public support... democracies are vulnerable to measures that 'flood' public debate and disrupt shared decentralized understandings of actors and coalitions, in ways that autocracies are not."

One compelling metaresearch point from the paper is that autocratic governments receive analysis of information trade-offs, while democratic governments do not:

"There is existing research literature on the informational trade-offs or 'dictators' dilemmas' that autocrats face, in seeking to balance between their own need for useful information and economic growth, and the risk that others can use available information to undermine their rule. There is no corresponding literature on the informational trade-offs that democracies face between desiderata like availability and stability."

Full paper available on SSRN here.

  • Farrell summarizes the work on Crooked Timber: "In other words, the same fake news techniques that benefit autocracies by making everyone unsure about political alternatives undermine democracies by making people question the common political systems that bind their society." Many substantive comments follow. Link.
  • Jeremy Wallace, an expert on authoritarianism, weighs in on Twitter: "Insiders, inevitably, have even more information about the contours of these debates. On the other hand, there's a lot that dictators don't know--about their own regimes, the threats that they are facing, etc." Link to Wallace's work on the topic.
  • Related reading recommended by Wallace, from Daniel Little, a 2016 paper on propaganda: "Surprisingly, the government tends to pick a high level of propaganda precisely when it is ineffective." Link.
⤷ Full Article

January 5th, 2019

Aunt Eliza


Bringing evidence to bear on policy

Happy 2019. We’re beginning with a report from Evidence in Practice, a project from the Yale School of Management. The report focuses on how to integrate rigorously researched evidence with policy and practice, with an emphasis on international development. The needs numerous stakeholders involved in research and policymaking are enumerated, along with their own needs and priorities: funders, researchers, intermediaries, policymakers, and implementers each receive consideration. One of the strengths of the report is its quotations from dozens of interviews across these groups, which give a sense of the messy, at times frustrating, always collaborative business of effecting change in the world. As to the question of what works:

⤷ Full Article

December 15th, 2018

Space Dance


A new book examines the economic and social impacts of R&D

Last May, we highlighted a report on workforce training and technological competitiveness which outlined trends in research and development investment. The report found that despite "total U.S. R&D funding reaching an all-time high in 2015," it's shifted dramatically to the private sector: "federal funding for R&D, which goes overwhelmingly to basic scientific research, has declined steadily and is now at the lowest level since the early 1950s." This week, we take a look at the returns to these investments and discuss how best to measure and trace the ways research spending affects economic activity and policy.

In the most recent Issues in Science and Technology, IRWIN FELLER reviewsMeasuring the Economic Value of Research, a technical monograph that discusses how best to measure the impact and value of research on policy objectives. Notably, the book highlights UMETRICS, a unified dataset from a consortium of universities "that can be used to better inform decisions relating to the level, apportionment, human capital needs, and physical facility requirements of public investments in R&D and the returns of these investments." While it represents a big data approach to program evaluation, Feller notes that UMETRICS' strength is in the "small data, theory-driven, and exacting construction of its constituent datasets," all of which offer insight into the importance of human capital in successful R&D:

"The book’s characterization of the ways in which scientific ideas are transmitted to and constitute value to the broader economy encompasses publications and patents, but most importantly includes the employment of people trained in food safety research. This emphasis on human capital reflects a core proposition of UMETRICS, namely the 'importance of people—students, principal investigators, postdoctoral researchers, and research staff—who conduct research, create new knowledge, and transmit that knowledge into the broader economy.'

In particular, the chapters on workforce dynamics relating to employment, earnings, occupations, and early careers highlight the nuanced, disaggregated, and policy-relevant information made possible by UMETRICS. These data provide much-needed reinforcement to the historic proposition advanced by research-oriented universities that their major contribution to societal well-being—economic and beyond—is through the joint production of research and graduate education, more than patents or other metrics of technology transfer or firm formation."

The UMETRICS dataset traces the social and economic returns of research universities and allows for a larger examination of universities as sociopolitical anchors and scientific infrastructure.

⤷ Full Article

November 17th, 2018

Poetry Machine



The inefficiency of lobbying

A few weeks ago, we spotlighted work by Elliott Ash et. al. on the startling influence of the Manne economics seminars in shaping judicial decision-making. This week we’re looking at an industry that, conversely, seems extremely influential, but is frequently ineffectual: lobbying.

In a 2009 book, "Lobbying and Policy Change: Who Wins, Who Loses, and Why," FRANK R. BAUMGARTNER et. al. take an unprecedentedly thorough look at lobbying in Washington, scrutinizing "ninety-eight randomly selected policy issues in which interest groups were involved and then followed those issues across two Congresses." What they find is complexity and gridlock:

"Since we followed our issues for four years, we know a lot about what eventually occurred (if anything did). In fact, as we outline in the chapters to come, for the majority of our issues, little happened.

If what they are supposed to be doing is producing change, interest groups are a surprisingly ineffectual lot. A focus by the media and many academics on explaining political change or sensational examples of lobbying success obscures the fact that lobbyists often toil with little success in gaining attention to their causes or they meet such opposition to their efforts that the resulting battle leads to a stalemate.

Of course, many lobbyists are active because their organizations benefit from the status quo and they want to make sure that it stays in place. We will show that one of the best single predictors of success in the lobbying game is not how much money an organization has on its side, but simply whether it is attempting to protect the policy that is already in place."

Preview on Google Books here.

⤷ Full Article

November 3rd, 2018

Absorbs The Shape



How do we meaningfully compare regime change?

In last week’s newsletter, we spotlighted work by Elliott Ash, Daniel Chen, and Suresh Naidu that provided quantitative analysis of the judicial effects of the law and economics movement. More generally, the paper examined how small-scale intellectual projects—like a series of economic seminars by the Manne Economics Institute—carry significant judicial and ideological outcomes. This week, we examine ideological diffusion on the macroscale and explore the role of external and international influences on democratic uprisings throughout history.

Democratic uprisings (beginning with the American Revolution and culminating in the Arab Spring) mark the last two hundred years of global democratization. The specific regional, historical, and economic circumstances surrounding these turbulent and diverse democratic revolutions make meaningful and effective comparisons hard to achieve.

In a recent paper, SEVA GUNITSKY offers a conceptual framework for better comparing democratic regime shifts over time and identifies the recurring mechanisms that catalyze and shape democratic uprisings. The work approaches regime changes as clusterings or cascades and so organizes thirteen democratic 'waves' into four typologies along two central dimensions: the origins of external influences (horizontal or vertical) and the role of those influences in timing the democratic wave (contagion or emulation). With this framework, Gunitsky claims that global interactions spark democratic changes:

"The looming presence of waves suggests that studies of democratization cannot focus only on the local drivers of revolts from below or elite concessions from above. Episodes of mass political contention were often embedded in broader transnational processes that involved regional cross-border ties and global hegemonic rivalries. More generally, examining the causes of democratic waves is a reminder that global democratization is more than the sum of its parts. The spread of democracy embodies multiple facets of a systemic phenomenon, driven by cross-border linkages that cannot be reduced to their individual components. Examining how democracy spreads can offer fundamental insight into the nature of democracy itself."

Gunitsky delves further into democratic regime changes—and those that resulted in fascism and communism—offering prescient insight into the election of Jair Bolsonaro in Brazil:

"Sudden shifts in the structure of hegemonic power have produced some of the most consequential regime cascades in modern history. In some ways, the twentieth century can be imagined as a series of hegemonic shocks and institutional waves. Yet the links between systemic shifts and institutional waves were not limited to democracy: German economic recovery in the 1930s led to the diffusion of fascist ideas and institutions, and the Soviet victory in World War II prompted a global communist wave that spread through both force and admiration. Future hegemonic transitions, including the decline of American dominance, are likely to produce similar anti-democratic cascades, particularly in case of a sudden U.S. decline."

Link to the paper, and link to Gunitsky's book on the same topic.

  • In a 2013 paper, Nathan Nunn and Paola Giuliano examine the importance of local-level democracy—a tradition of electing a local leader through consensus rather than appointment—and its relationship to state and national level democratic institutions. Link.
  • James Kloppenberg's Toward Democracy probes the Atlantic Democratic Wave and provides a major synthesis of Western intellectual thought. Link.
⤷ Full Article

October 27th, 2018

The Seasons


Applying quantitative methods to examine the spread of ideology in judicial opinion

In a recent paper, co-authors ELLIOTT ASH, DANIEL L. CHEN, and SURESH NAIDU provide a quantitative analysis of the judicial effects of the law and economics movement. Comparing attendance at seminars run by the Manne Economics Institute for Federal Judges from 1976 to 1999 against 380,000 circuit court cases and one million criminal sentencing decisions in district courts, the authors identify both the effects on judicial decision-making and the dispersion of economic language and reasoning throughout the federal judiciary.

“Economics-trained judges significantly impact U.S. judicial outcomes. They render conservative votes and verdicts, are against regulation and criminal appeals, and mete harsher criminal sentences and deterrence reasoning. When ideas move from economics into law, ideas have consequences. Economics likely changed how judges perceived the consequences of their decisions. If you teach judges that markets work, they deregulate government. If you teach judges that deterrence works, they become harsher to criminal defendants. Economics training focusing on efficiency may have crowded out other constitutional theories of interpretation. Economics training accounts for a substantial portion of the conservative shift in the federal judiciary since 1976.”

Link to the paper.

  • Henry Farrell at Crooked Timber picks out some additional highlights. Link.
  • A Washington Post article from January 1980 provides some contemporaneous context on the Manne seminars. Link.
  • In a relevant 2015 paper, Pedro Bordalo, Nicola Gennaioli, and Andrei Shleifer apply salience theory to model judicial decision-making: "The context of the judicial decision, which is comparative by nature, shapes which aspects of the case stand out and draw the judge’s attention. By focusing judicial attention on such salient aspects of the case, legally irrelevant information can affect judicial decisions." Link.
⤷ Full Article

October 20th, 2018

Action Plan


Competing definitons of the term have vast policy implications

The formal definition of family is “blood, marriage, or adoption,” but that leaves out many possible arrangements, including families of unmarried people, foster children, co-ops, and, until 2015, gay partnerships. In the 1970s, family law became more open to “functional families” outside the formal definition, while zoning law kept to the strictly formal. Legal historian KATE REDBURN writes, “These contradictions leave critical family law doctrines unstable in thirty-two states.”

In a recent working paper, Redburn examines how these changes came to be, and looks more generally at how legal regimes exist within connected networks and influence each other despite traditional boundaries of scale (local, state, etc.) and subject (family law, zoning law):

“Viewed through a broader lens, this story might suggest lessons for law and social movements. While progressives oriented their campaigns at the state level, homeowners imbued local governance with conservative social politics in defense of their prejudices and property values. Neither movement, nor the judges adjudicating their case, nor the legislators revising state and local statutes, paid adequate attention to the interlocking nature of legal doctrines, rendering their movements less successful than they have previously appeared. Though we tend to think of legal fields as distinct regimes, ignoring the multifaceted ways that doctrines overlap, connect, and contradict each other can have perilous consequences. Their blind spot has has grown to encompass millions of Americans.”

Redburn’s case study provides ample evidence that micro-level legal conflicts can uphold and alter legal understandings:

“Motivated constituencies of voters and their elected representatives can produce legal change quite out of sync with social trends. Such was the case in the zoning definition of family in the late 1960s and early 1970s. Despite social change resulting in more functional families, protective homeowners and the conservative movement successfully shifted zoning law away from the functional family approach.”

⤷ Full Article