Michael Giberson

William Easterly writes, “Few people outside academia realize how badly Randomized Evaluation has polarized academic development economists for and against.”

That claim seems reasonable enough. I’d bet few people outside academia know what randomized evaluation is. Frankly, I’d bet you could survey economists on the floor of the upcoming American Economic Association meetings in Atlanta and, for non-development specialist, find that fewer than 50 percent “realize how badly Randomized Evaluation has polarized academic development economists.”

Easterly raises the point as a way to introduce a conference and now edited book volume — he helped organize the conference and edit the book — which brought together the fors and againsts for dialog.

Michael Giberson

Bloomberg Markets Magazine has a feature on economist John List and his $10 million research project on education. Along the way we get an introduction to List’s work on field experiments in economics, a splash of lab-based economics back story, and the reaction of education specialists who think List’s project is wholly off target.

List, along with collaborators Steven Levitt and Roland Fryer, has obtained a $10 grant for a program which randomly assigned 3-5 year old students to one of three groups: (1) free all-day preschool, (2) “parenting academy” for the student’s parent or guardian, or (3) a control group with neither intervention. The program intends for follow the students into adulthood in order to assess the long-term effects of the intervention.

List says he doesn’t know much about education theory, so he enlisted specialists to consult on the preschool curriculum. One such consultant, Clancy Blair, a New York University professor of applied psychology, says he was astonished by the size of the project and by how it focuses on financial incentives without looking at such variables as how the parents interact with their children.

“That’s a crazy idea,” says Blair, who studies how young children learn. “It’s not based on any prior research. This isn’t the incremental process of science. It’s ‘I have a crazy idea and I convinced someone to give me $10 million.’”

List says too many decisions in fields from education to business to philanthropy are made without any scientific basis. Without experimenting, you can’t evaluate whether a program is effective, he says.

“We need hundreds of experiments going on at once all over the country,” he says. “Then we can understand what works and what doesn’t.” …

“What educators need to know are what are the best ways to educate kids, and this is trying to short-circuit that,” Blair says. “We have fundamental problems in education, and this is sort of a distraction.”

List says he understands the objections. “If I was in the field, I’d hate me, too,” List says in November while driving to his sons’ indoor baseball practice in one of Chicago’s south suburbs. “There should be skeptics.”

Lynne Kiesling

Think consciously about a decision you contemplated recently. As you were weighing your options, how much did you really know that you could bring to bear, definitively, on your decision? Was the outcome pre-determined, or was it unknown to you? For most of the decision-making situations we confront regularly, we don’t have full information about all of the inputs, causal factors, and consequent outcomes. Whether it’s due to costly information, imperfect foresight, the substantial role of tacit knowledge, the inability to predict the actions of others, or other cognitive or environmental factors, our empirical knowledge has significant limits. And yet we make decisions ranging from the color of socks to wear today to whether or not to bail out Bear Stearns or Lehman Brothers. But we do so despite these significant limits of our empirical knowledge.

We build, test, and apply models to try to reduce this knowledge constraint. Models hypothesize causal relationships, and in social science we test those models largely using quantitative data and statistical tests. But when we build formal models, we make simplifying assumptions to make sure that the model is mathematically tractable, and we test those models for causality using incomplete data because we can’t capture or quantify all potentially causal factors. Sometimes these simplifying assumptions and omitted variables are innocuous, but then how useful will such models be in helping us to understand and predict outcomes in complex systems? Complex systems are characterized by interdependence and interaction among decisions of agents in ways that are non-deterministic, and specific outcomes in complex systems are typically not predictable (although analyses of complex phenomena like networks can reveal patterns of interactions or patterns of outcomes).

One person who’s been thinking carefully through these questions is Jim Manzi, whose new book Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society is generating a lot of discussion (and is on my summer reading list). On EconTalk this week he and Russ Roberts talked about the ideas in the book, and their implications for “business, politics, and society”. Russ summarizes the books focus as

Manzi argues that unlike science, which can produce useful results using controlled experiments, social science typically involves complex systems where system-wide experiments are rare and statistical tools are limited in their ability to isolate causal relations. Because of the complexity of social environments, even narrow experiments are unlikely to have the wide application that can be found in the laws uncovered by experiments in the physical sciences. Manzi advocates a trial-and-error approach using randomized field trials to verify the usefulness of many policy proposals. And he argues for humility and lowered expectations when it comes to understanding causal effects in social settings related to public policy.

Experimentation in complex social environments is a theme on which I am writing this summer, with application to competition and deregulation in retail electricity markets. Manzi’s ideas certainly flesh out the argument for experimentation as an approach to implementing institutional change that can identify unintended consequences and head costly design choices off at the pass before they become costly or disruptive. I made similar arguments in an article in Electricity Journal in 2005 for using economic experiments to test electricity policy institutional designs, and Mike and I discussed those issues here and here. In broad brushstroke, traditional cost-based economic regulation typically stifles experimentation, because to implement it the regulator has to define the characteristics of the product, define the boundaries of the market, and erect a legal entry barrier to create a monopoly in that market. Experimentation occurs predominantly through entry, by product differentiation that consequently changes the market boundaries. To the extent that experimentation does occur in regulated industries, it’s very project-based, with preferred vendor partners and strict limits on what the regulated firm can and cannot do. So even when regulation doesn’t stifle experimentation, it does narrow and truncate it.

Recently Manzi wrote some guest posts at Megan McArdle’s blog at The Atlantic, including this one summarizing his book and providing an interesting case study to illustrate it. His summary of the book’s ideas is relevant and worth considering:

1. Nonexperimental social science currently is not capable of making useful, reliable, and nonobvious predictions for the effects of most proposed policy interventions.
2. Social science very likely can improve its practical utility by conducting many more experiments, and should do so.
3. Even with such improvement, it will not be able to adjudicate most important policy debates.
4. Recognition of this uncertainty calls for a heavy reliance on unstructured trial-and-error progress.
5. The limits to the use of trial and error are established predominantly by the need for strategy and long-term vision.

That post is rich with ideas, and I suspect Mike and I will want to pursue them here as we delve into the book.

Some of the most illuminating work in market design lately has been in payment for ecosystem services (PES). These projects provide examples of the continuing relevance of institutions to economic and policy outcomes, and the importance of Elinor Ostrom’s work on the diversity of governance institutions in common-pool resources; despite criticisms leveled at synthetic markets, they can lead to better outcomes compared to the realistic alternative, which is usually a regulatory response.

In 2017 my former colleague Seema Jayachandran and co-authors published (in Science) a thorough analysis of a randomized controlled trial they performed in Uganda with groups of private forest owners in 121 randomly-selected towns. The randomly-selected treatment group received payments if they chose not to cut down trees, and half of the villages had the experiment run for two years. These two dimensions of the experiment allowed the authors to test whether payment induced some people to leave trees standing that they would otherwise cut down, compared to the rate of cutting in the control group, and to test an income and duration effect across the villages. They also performed a benefit-cost analysis of the impact of the delayed carbon emissions using the EPA’s median estimate of the social cost of carbon from 2012. As summarized in the paper’s abstract:

We evaluated a program of payments for ecosystem services in Uganda that offered forest-owning households annual payments of 70,000 Ugandan shillings per hectare if they conserved their forest. The program was implemented as a randomized controlled trial in 121 villages, 60 of which received the program for 2 years. The primary outcome was the change in land area covered by trees, measured by classifying high-resolution satellite imagery. We found that tree cover declined by 4.2% during the study period in treatment villages, compared to 9.1% in control villages. We found no evidence that enrollees shifted their deforestation to nearby land. We valued the delayed carbon dioxide emissions and found that this program benefit is 2.4 times as large as the program costs.

This innovative paper combined economics, experimental methods, GIS mapping, and environmental science to study this important question (and I recommend Brad Plumer’s New York Times analysis of the research and its importance).

Another recent PES analysis from Krister Andersson and co-authors (in Nature Sustainability) uses a “framed field experiment” to simulate the effects of conservation payments on short-term decisions, and on the duration of those effects; it also tested the effects of communal versus individual decision-making:

To shed light on the debate, Andersson and his colleagues traveled to 54 villages near tropical forests in Bolivia, Indonesia, Peru, Tanzania and Uganda.

There, they staged a half-day table-top simulation game in which local forest users were divided into groups of eight and asked to make decisions about how many trees they would harvest from a shared forest.

They had the opportunity to earn more than a full day’s pay based upon their decisions.

In the first stage, they were not allowed to communicate with others in their group and made individual decisions based on their own needs and values. In the second stage, they were offered money to cut down fewer trees (to mimic a PES), asked to discuss for five minutes and decide as group, or both. In the third stage, they went back to making decisions alone with no cash incentive.

Participants who got cash in the second stage cut down 19 percent fewer trees. Those who got cash and were encouraged to communicate in their decisions cut down 48 percent fewer trees.

Even after payments stopped, those groups that had been paid continued to conserve, with the group that got cash and worked together maintaining a 23 percent reduction (compared to pre-payment) in the number of trees cut down.

Those who had indicated in surveys prior to the game that they trusted their other community members conserved the most, cutting down 35 percent fewer trees in the game post-payment than prior to payment.

“Our experimental results suggest that payments, especially when they are conditional on group cooperation, can help people realize the value of cooperation and that lasting cooperation can lead to better forest conditions,” said Andersson.

An important aspect of the underlying economics in both studies is the crucial role that institutions play in structuring social interactions and shaping incentives — the Ostrom point. The Andersson et. al. paper analyzes that dimension of the question more explicitly than the Jayachandran et. al. paper does (the bibliography of the Andersson et. al. paper reflects the Bloomington School influence and is worth exploring if you are interested in these questions). Both analyses use synthetic markets to achieve specific policy objectives, and involve careful institutional design to craft market rules.

Synthetic markets and deliberate institutional design are prone to criticisms, one of which is epistemic. Synthetic market designers engage in deliberate institutional design to create a market that did not exist before, in contrast to a more organic process of market emergence. Emergent processes, with distributed trial and error learning, more effectively capture and reflect the private and often tacit knowledge embedded in the subjective preferences and opportunity costs of the individual participants, and market designers ex ante do not have access to that knowledge when they are making design decisions unless they go out and try to learn about it.

Synthetic market designers are also teleological — they have a goal in mind, typically a policy goal, and design rules to shape incentives with the aim of achieving that goal — in contrast to the more open-ended nature of markets where the people who have the goals are the individual participants, and the market institutions evolve and change over time to better enable individuals to coordinate and to meet their individual goals mutually through markets. By designing to achieve a shared goal, synthetic market design prioritizes that goal without knowing the preferences of the participants and whether they value using resources toward that goal relative to the other ways they can use their resources and create new resources.

In cases where property rights are never going to be well-defined, though, synthetic markets can yield better outcomes than the realistic feasible political alternatives. In many environmental contexts the realistic alternative is command-and-control regulation rather than the utopian ideal of well-defined property rights and markets with low entry barriers.

I am concerned about these epistemic aspects of synthetic market design. Designing market rules to achieve specific outcomes can involve overlooking the wide range of unknowable incentives facing market participants, so they may not make choices in the way that the designers expect. The designers also may not pay enough attention to the emergent, organic nature of market processes, and may be overly prescriptive in their rules. And, to quote the great philosopher Yogi Berra, prediction is hard, especially about the future, so the designers may come up with an institutional framework that suits a specific context and goal, but then as economic and technological change happen, how well will that framework adapt to unknown and changing conditions? And will that framework foreclose choices that might be more beneficial/value-creating/cost-minimizing/innovation-inducing? Synthetic market design is really difficult, for all of these reasons.

These epistemic issues illustrate why testing is an essential aspect of synthetic market design (a point I argued in a paper in the Electricity Journal). Field experiments such as the two papers highlighted here provide estimates of how people respond to the incentives in a particular institutional framework, the magnitude of the responses, whether they yield any unintended consequences, and some estimate of whether or not the design process is worth it (which inescapably involves normative assumptions and value judgments). Doing comparative institutional analysis, where the treatments are different institutional arrangements, can deepen those insights and help minimize the costs associated with deliberate institutional design in an organic, complex system.