There’s been a lot of commentary on the consequences of the Rogoff-Reinhart error, which is seriously undermining the empirical basis for the austerity program that has swept the US, Canada and Europe. I think this episode highlights the limits of evidence-based policy-making.
One clue that there is deep limits to the concept is that I’ve never actually heard a policy-maker claim that they are making policy without evidence. It seems that everyone justifies their policy preferences with some sort of evidence; it’s just that there is always a debate about what constitutes evidence. I also think that many liberal (and here I mean, small-l liberals with a strong commitment to equating justice with procedure, rather than outcome) scholars and citizens adhere to an ideal of evidence-based policymaking. I think it’s because they feel that somehow the nastiness of politics can be minimized in public policy making if we can all just agree on the facts and procedures for adjudicating facts.
Here I’ll make two claims: evidence can only inform, it can never resolve, political debate. And evidence-based policy-making will always be influenced, if not, determined by pre-existing values, rather than the other way around.
If evidence really could influence policy to the degree that the adherents of evidence-based policy-making claim it can, then we would be seeing a much quicker reversal of economic policy given that one of the essential pieces of evidence for austerity has been shown to be, well, wrong. Aside from this in Italy, austerity remains the dominant economic policy.
Continue reading
I write this post only partially motivated by the fall-out from R+R. But this episode reinforces some of the conclusions that I’ve drawn out of my ongoing research into the politics of the regulation of BPA in Canada and the US. Clearly, evidence mattered in that case. Both sides – those in favour of and those opposed to stricter regulations – marshalled as much scientific evidence as possible to support their claims. And both sides are guilty of calling the other “anti-scientific”. The problem is that there is no agreement on what constitutes evidence. Those who assert there is a great threat posed by exposure to BPA often rely on studies which often do show adverse effects from exposure to BPA, but base this conclusion on studies that only administered one or two different doses of BPA. When these small studies show a positive relationship between exposure and effect, environmental activists seize on them because they have already made value commitments that we are constantly being poisoned by toxins and that modern society is killing us.
It has to be emphasized: there are powerful structural forces that govern the scientific endeavour that leads to the publication of these kinds of exciting findings based on small sample sizes and dose ranges. Put yourself in the shoes of a new, untenured, natural scientist. You need to get grant money and you need to get publications. In tough economic times and the competitive scientific job market these pressures can be excruciating. But null findings just won’t cut it. Journals that are owned by a small oligopoly of media companies want citations to papers that they publish and tend not to attract a lot of notice. Findings that show that a substance does have an effect is likely to get cited.
The vast majority of positive findings are published when they show effects that are “statistically significant at the .05 level”. This means that there is only a 5 per cent chance that that published finding will have occurred by chance alone. But what that means is that even if scientists are entirely objective, then 1 in 20 scientific studies will be a false positive. However, scientists are not entirely objective. Too often, null results from studies are shelved and only the statistically significant studies are published. But, if we are never told how many null findings the scientist received before getting a statistically significant finding, then we have no way of determining whether or not the finding could be random noise or a real finding. Ben Goldacre has pointed out that this badly plagues pharmaceutical research, but Ioannides has found similar problems in medical research.
There are two wonderful images from a presentation to an expert working group on BPA from June 2012 that prove this point better than I ever could. This is around 50 random values ranging from 80 to about 145 plotted on the y-axis.
Would you say that there is any pattern evident? Of course not.
Now, would you say the same about the following four values?
That looks to me like BPA/coffee/smoking/cosmetics causes cancer/obesity/heart attacks!
Except the values are drawn from the same randomly selected 20 values in the first figure. Look closely, I can see at least two sequences of 4 randomly generated values that increase monotonically creating the impression of a linear, cause-and-effect relationship.
This is what gets published, even if it’s the product of random noise.
There are ways to guard against this, but they essentially involve doing large-scale, multi-generation rodent studies with a wide range of doses. But this costs a lot of money and sadly,the only people who have this kind of money to do research (which often results in null findings) are corporations who obviously have a strong interest in the publication of such findings when substances they produce are under scrutiny.
Just because it is funded by corporations does not mean the “evidence” is invalid according to strict scientific standards, but we’re not dealing in science: we’re dealing in politics. And this fact certainly excites activist groups, politicians, journalists, rival scientists and voters who see these kinds of studies not as “evidence” of no findings, but of “evidence” of collusion and corruption.
Into this vortex of institutional, economic and value-based judgements, politicians and regulators are called on to make a decision. They have developed some tools to improve the role of “evidence” in policy making, primarily through the use of systematic literature reviews and tools to evaluate the merits of different type of evidence.
But as I stated before, this always interacts with pre-existing values. An excellent example of this in the Canadian Environmental Protection Act. There, the Ministers of Health and the Environment are required to apply both the precautionary principle and the weight-of-evidence approach when publishing risk assessments of potentially toxic substances. The former, in one form, requires (some might say allows) regulatory action even based on inconclusive “evidence”. The latter requires regulators to rationally weight the merits of “evidence”, discounting weak “evidence” such as studies with small dose ranges, and according greater weight to large-scale, more credible, more reliable studies.
With the requirement to apply both principles entrenched in Canadian legislation, you can see how “evidence” can never answer the regulator’s question of what type of policy she should adopt. There is always a debate about what constitutes “evidence”. Those who are convinced that inconclusive “evidence” is actually sufficient “evidence” for action, will simply demand that the state act, those who disagree that the “evidence” is conclusive will simply state so. There actually is no “evidence” that can reconcile these two positions, because this is actually never about “evidence”; politics is about values and interests.
That’s why the R+R findings made such a splash in the first place, why their refutation won’t matter much, why BPA was deemed “toxic” even though the scrreening assessment explicitly states that it is safe at current levels of exposure and, frankly, why evidence-based policy-making is not so much impossible, just illusory. It can’t exist as the kind of decisive resolution to political challenges people want it to be.
