The Limits of Evidence-Based Policymaking

There’s been a lot of commentary on the consequences of the Rogoff-Reinhart error, which is seriously undermining the empirical basis for the austerity program that has swept the US, Canada and Europe. I think this episode highlights the limits of evidence-based policy-making.

One clue that there is deep limits to the concept is that I’ve never actually heard a policy-maker claim that they are making policy without evidence. It seems that everyone justifies their policy preferences with some sort of evidence; it’s just that there is always a debate about what constitutes evidence. I also think that many liberal (and here I mean, small-l liberals with a strong commitment to equating justice with procedure, rather than outcome) scholars and citizens adhere to an ideal of evidence-based policymaking. I think it’s because they feel that somehow the nastiness of politics can be minimized in public policy making if we can all just agree on the facts and procedures for adjudicating facts.

Here I’ll make two claims: evidence can only inform, it can never resolve, political debate. And evidence-based policy-making will always be influenced, if not, determined by pre-existing values, rather than the other way around.

If evidence really could influence policy to the degree that the adherents of evidence-based policy-making claim it can, then we would be seeing a much quicker reversal of economic policy given that one of the essential pieces of evidence for austerity has been shown to be, well, wrong. Aside from this in Italy, austerity remains the dominant economic policy.
Continue reading

It’s true it is early days and this development may ultimately shift global economic policy, but, the longer it takes, the more it proves my point. Evidence cannot be decisive for policy-making; instead, it interacts with deeper values which are of greater import. In terms of austerity, it wasn’t R-R that told policy-makers to adopt austerity; I assert that it fit with widespread, deeply-held values that a good life is one that is lived modestly with an emphasis on earning and saving money.

I write this post only partially motivated by the fall-out from R+R. But this episode reinforces some of the conclusions that I’ve drawn out of my ongoing research into the politics of the regulation of BPA in Canada and the US. Clearly, evidence mattered in that case. Both sides – those in favour of and those opposed to stricter regulations – marshalled as much scientific evidence as possible to support their claims. And both sides are guilty of calling the other “anti-scientific”. The problem is that there is no agreement on what constitutes evidence. Those who assert there is a great threat posed by exposure to BPA often rely on studies which often do show adverse effects from exposure to BPA, but base this conclusion on studies that only administered one or two different doses of BPA. When these small studies show a positive relationship between exposure and effect, environmental activists seize on them because they have already made value commitments that we are constantly being poisoned by toxins and that modern society is killing us.

It has to be emphasized: there are powerful structural forces that govern the scientific endeavour that leads to the publication of these kinds of exciting findings based on small sample sizes and dose ranges. Put yourself in the shoes of a new, untenured, natural scientist. You need to get grant money and you need to get publications. In tough economic times and the competitive scientific job market these pressures can be excruciating. But null findings just won’t cut it. Journals that are owned by a small oligopoly of media companies want citations to papers that they publish and tend not to attract a lot of notice. Findings that show that a substance does have an effect is likely to get cited.

The vast majority of positive findings are published when they show effects that are “statistically significant at the .05 level”. This means that there is only a 5 per cent chance that that published finding will have occurred by chance alone. But what that means is that even if scientists are entirely objective, then 1 in 20 scientific studies will be a false positive. However, scientists are not entirely objective. Too often, null results from studies are shelved and only the statistically significant studies are published. But, if we are never told how many null findings the scientist received before getting a statistically significant finding, then we have no way of determining whether or not the finding could be random noise or a real finding. Ben Goldacre has pointed out that this badly plagues pharmaceutical research, but Ioannides has found similar problems in medical research.

There are two wonderful images from a presentation to an expert working group on BPA from June 2012 that prove this point better than I ever could. This is around 50 random values ranging from 80 to about 145 plotted on the y-axis.
Screen shot 2013-04-30 at 2.15.49 PM
Would you say that there is any pattern evident? Of course not.

Now, would you say the same about the following four values?

Screen shot 2013-04-30 at 2.15.54 PM

That looks to me like BPA/coffee/smoking/cosmetics causes cancer/obesity/heart attacks!

Except the values are drawn from the same randomly selected 20 values in the first figure. Look closely, I can see at least two sequences of 4 randomly generated values that increase monotonically creating the impression of a linear, cause-and-effect relationship.

This is what gets published, even if it’s the product of random noise.

There are ways to guard against this, but they essentially involve doing large-scale, multi-generation rodent studies with a wide range of doses. But this costs a lot of money and sadly,the only people who have this kind of money to do research (which often results in null findings) are corporations who obviously have a strong interest in the publication of such findings when substances they produce are under scrutiny.

Just because it is funded by corporations does not mean the “evidence” is invalid according to strict scientific standards, but we’re not dealing in science: we’re dealing in politics. And this fact certainly excites activist groups, politicians, journalists, rival scientists and voters who see these kinds of studies not as “evidence” of no findings, but of “evidence” of collusion and corruption.

Into this vortex of institutional, economic and value-based judgements, politicians and regulators are called on to make a decision. They have developed some tools to improve the role of “evidence” in policy making, primarily through the use of systematic literature reviews and tools to evaluate the merits of different type of evidence.

But as I stated before, this always interacts with pre-existing values. An excellent example of this in the Canadian Environmental Protection Act. There, the Ministers of Health and the Environment are required to apply both the precautionary principle and the weight-of-evidence approach when publishing risk assessments of potentially toxic substances. The former, in one form, requires (some might say allows) regulatory action even based on inconclusive “evidence”. The latter requires regulators to rationally weight the merits of “evidence”, discounting weak “evidence” such as studies with small dose ranges, and according greater weight to large-scale, more credible, more reliable studies.

With the requirement to apply both principles entrenched in Canadian legislation, you can see how “evidence” can never answer the regulator’s question of what type of policy she should adopt. There is always a debate about what constitutes “evidence”. Those who are convinced that inconclusive “evidence” is actually sufficient “evidence” for action, will simply demand that the state act, those who disagree that the “evidence” is conclusive will simply state so. There actually is no “evidence” that can reconcile these two positions, because this is actually never about “evidence”; politics is about values and interests.

That’s why the R+R findings made such a splash in the first place, why their refutation won’t matter much, why BPA was deemed “toxic” even though the scrreening assessment explicitly states that it is safe at current levels of exposure and, frankly, why evidence-based policy-making is not so much impossible, just illusory. It can’t exist as the kind of decisive resolution to political challenges people want it to be.

Public Opinion, Precaution and Toxicology

There is news today that a coroner in New Zealand formally ruled that the excessive consumption of Coca Cola led to the untimely death of Natasha Harris three years ago. Harris, it seems, developed a nasty habit of drinking up to 10 litres a day. “I find that when all the available evidence is considered, were it not for the consumption of very large quantities of Coke by Natasha Harris, it is unlikely that she would have died when she died and how she died” is what the coroner determined.

So what does this have to do with a blog about Canadian, politics and public opinion? Because this tragic incident illustrates a bedrock principle of toxicology that citizens, journalists and politicians often forget when our attention turns to issues of concern with possibly toxic substances. That principle is “the dose makes the poison.” It was formulated in the 16th century by the Swiss-German chemist Paracelsus and it holds that all substances (Coca-Cola included) are toxic in some quantity. Even something as seemingly innocuous as water can be toxic when enough has been consumed. This poor soul died from the excessive consumption of water during a brutal hazing ritual in New York. From this observation, the traditional role of toxicology has developed which is to ascertain at what level of exposure any given substance causes adverse effects.

With the rise of concerns about toxic substances in the post-World War II era, scientists have discovered some substances that deviate from this principle (see tamoxifen, an important drug in chemotherapy). But these effects are rare, difficult to demonstrate and the scientific debates that surround them are heated and complex. But when issues shift into the public and media spheres, the complex science behind these assertions, sadly, usually gets lost in the shuffle.
Continue reading

There are multiple reasons for why this is. One reason is our own inability to perceive risks rationally. There is so much literature on the psychology of human risk perception that it is difficult to know where to start, but I might recommend this thispaper or this book. The long and short of the field of risk perception is that humans get very frightful about lots of things that actually present very little danger. After September 11th, Americans turned away from planes for their long trips and hopped into their cars, leading to a sharp increase in traffic fatalities in the months following.

Chemicals are one of those things that people fear a great deal. Ask people if they think the colourless, odourless chemical dihydrogen monoxide (the ingestion of which is the cause of thousands of deaths each year) should be banned and you’d be surprised (and a little concerned) at the number of people who would agree.

So when the public and the mass media turn their attention to issues of possible harm from exposure to chemicals, fears can be stoked, perspective can be lost and can lead, ultimately, to over- and unnecessary regulation of products and activity. For example, I was able to show in this paper that within-state newspaper coverage of Bisphenol A was positively linked to the chance that a US state legislature would introduce or adopt legislation banning products made with BPA the following year. It showed (again) the important role that increased media salience can have on policy changes. This, even though, scientific information suggests that exposure to BPA is hundres, if not thousands of times lower than the level at which there is a consensus that it causes adverse effects in rats (see here).

This research area is rife for investigation and it’s become a long-term project I’m working on, so I’m going to have a lot more to say about this in the future. I thought the news about Ms. Harris’ tragic death due to the excessive consumption of Coca-Cola was a useful way to set the stage for more thoughts and research to come.