On the history of impact evaluation

Good evaluative research has had a huge impact on the lives of people. Careful observational studies have long been important tools for such research. (Chapter 6 of EOP provides a non-technical review of the methods found in practice.)

An example is the use of border discontinuities—an instance of what is often called these days a “regression discontinuity design.” Here the key assumption is that the counterfactual in the absence of a “treatment” of interest is continuous across some geographic borderline; all that differs across that border is the presence of the treatment. Under that assumption, we can infer the impact of the treatment by comparing outcomes either side of the border. The assumption is important, and one should look for any confounding factors in each application—any reason why the counterfactual outcomes might differ across the border.

A famous example of this type of evaluation method is found in the work of David Card and Alan Kruger, in their book, Myth and Measurement. Card and Kruger were interested in the effect of a higher minimum wage rate on employment. They compared employment in the fast-food industry either side of the border between two US states. The minimum wage rate had risen on one side of the border, but not the other. They found that the higher minimum wage increased incomes but did not reduce employment. (There has been a continuing debate on this issue, though the best evidence so far that I know of, from work by Arindrajit Dube and colleagues using comparisons of contiguous counties between US states with different minimum wages, comes to a similar conclusion to Card and Kruger.)

If you had asked me yesterday for the first example of the use of border discontinuity in impact evaluation, I would probably have said Card and Kruger. I just found out that I would have been wrong by 100 years! In an 1893 volume (available here in a 1895 English translation), the German microbiologist and medical doctor Robert Koch used this method to identify the effect of public water filtration on cholera in Germany. In case you don’t know, cholera is a truly awful bacterial disease. If untreated, the dehydration brought on by cholera will kill even a healthy person in just a few hours.

Famously, John Snow had mapped the incidence of deaths in a severe and geographically concentrated outbreak of cholera in 1854 London. The map suggested that the incidence of deaths due to cholera was associated with a specific drinking water supplier (the Broad Street pump in Soho) that had been contaminated by sewerage. Prior to this, the prevailing view was that cholera was an air-borne disease, not water borne. Snow’s map was essentially a descriptive tool, albeit a powerful one. It did not clinch the case for believing that dirty water was the culprit, though it had much influence on the debate.

In the 1892 cholera outbreak in Hamburg, Robert Koch conducted an observational study that was more suggestive of a causal impact of contaminated water. Koch observed that along the border between Hamburg (with no public water filtration) and neighboring Altona (with water filtration), there was cholera on the Hamburg side but not the Altona side. Koch (p.25) argued that (in today’s terminology) the counterfactual was continuous across this border:

“On both sides of the frontier, the state of the soil, the buildings, the sewerage, the population, in short all of the conditions that are of importance in this connection, are perfectly similar, and yet the cholera in Hamburg spread only to the frontier of Altona, and stopped there.” (p.25).

Interestingly, Koch goes on to provide a further test. It turns out that there was a group of residences on the Hamburg side that got their water from Altona. Yes, you guessed it: little or no cholera in that group of houses.

This was influential in persuading European cities to filter their water. In his path-breaking history of the world viewed through the lens of infectious diseases, Plagues and Peoples, William McNeill writes that, based on Koch’s observational study, “Doubters were silenced and cholera … never returned to European cities.” (p.278).

There can be no doubt that this empirical demonstration of the health benefits of clean drinking water has saved countless lives, and made life much better for countless others who did not die from diseases such as cholera, but suffered directly from the illness, or through the harm to their loved ones.  

If you know of other early examples of the use of observational studies in impact evaluation I would like to hear about them. Tell me about it at mr1185@georgetown.edu.

Martin Ravallion