Limits of evidence-based policy

There is something paradoxical about Michael Gove's recent speech calling for government to be "rigorous and fearless in its evaluation of policy and projects." It's that his praise for evidence-based policy has come in a year when we've seen that policy should sometimes not be based on rigorous evidence.

The best time to have imposed the lockdown was as soon as possible after a few cases had been discovered. But this would have been a hard sell. Not just the grifter media but the public would have asked: why are we losing our freedoms to forestall so small a threat? But of course, by the time there was strong evidence that a lockdown was necessary, it was too late.

Tens of thousands of us face the problem of having to act on insufficient evidence. The equity investor who waits for strong evidence that a share is under-priced will buy when it no longer is. Loads of hedge funds exemplify Gove's ideal: they employ lots of data scientists working with strong incentives and clear metrics. But their results are generally mediocre. Similarly, the entrepreneur who waits for proof that there's a market for a product will find that his rivals have captured it. The factory manager who waits for good evidence that a machine needs replacing will see it blow up. And so on.

The same applies to macroeconomic policy. The Bank of England did not cut Bank rate to 05% and introduce Qe until March 2009 - 12 months after the recession had begun. Waiting for strong evidence of a recession before loosening policy means loosening too late. This is why I favour strong automatic stabilizers.

In these cases, the demand for rigorous evidence is an example of what Jon Elster has called hyper-rationality:

Sometimes people ignore the costs of decision-making. They search for the solution that would have been best if found instantaneously and costlessly, ignoring the fact that the search itself has costs that may detract from optimality. (Solomonic Judgements, p260)

There are other problems. It's sometimes hard to extrapolate the results of the RCTs lauded by Gove. They "prove", for example, that parachutes don't save lives. As Cartwright and Deaton say:

Demonstrating that a treatment works in one situation is exceedingly weak evidence that it will work in the same way elsewhere; this is the ‘transportation’ problem.

And where they do yield results, these can be hard to interpret. The average treatment effect hides important heterogeneity of effects on individuals, for example.

There's also the problem that past evidence is no guide to the future. We have evidence from the 50s and 60s that very high top tax rates are entirely consistent with strong GDP growth. But is this evidence useful? Mr Gove and his friends would probably say not. And perhaps for a good reason: high taxes are sustainable if those paying them regard them as a price worth paying to live in a civilized society, but resist them if they regard them as infringements of their rights. Policies effects depend upon beliefs. And these change.

Indeed, they sometimes change because of policy. One effect of the expansion of universities in the 90s has been to create a large cohort of metropolitan liberals. How do we factor this into cost-benefit analyses?

There are two very important categories of cases where past evidence doesn't help us predict the future.

One is the problem of radical uncertainty: we can never be 100% confident that past statistical relationships will continue to hold. The other is that of reflexivity. Beliefs can change the world. For example, McLean and Pontiff and Cotter & McGeever have shown that when strong evidence emerges of stock market anomalies, these get competed away. In economic policy, the counterpart to this is Goodhart's law: "any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." You can't solve these problems with big data: the datasets used to find stock market anomalies are huge, but their findings less so .

All of which leads us to three paradoxes.

Paradox one is that, as Abby Innes points out, there is a precedent for Gove's belief that data science will deliver better government. The old Soviet Union had a "fantasy of ‘optimal government'" in which "cyberneticians would depict the governmental system as an object of technical control, with inputs, outputs, and feedback loops: the language of machines." This is evoked in Francis Spufford's wonderful Red Plenty. In this sense, Gove's role model is not so much FDR as Leonid Kantorovich. This project, however, failed in part for the reason Hayek pointed out - that governments just cannot know enough about complex systems.

Paradox two, as Phil points out, is that Gove's vision is unconservative. It is one in which:

the centrality of government is overweening and checks on the executive's remit - other state institutions, watchdog quangos, parliamentary scrutiny and media accountability - are brushed aside by authoritarian, action-oriented government.

Which leads to paradox three. One the question: is data-driven effective government likely, it is the conservative Gove here who is taking the side of the old USSR and this Marxist who is one the side of Hayek. 

Comments

[ltr]

July 4, 2020

Coronavirus

UK

Cases   ( 284,900)
Deaths   ( 44,198)

Would evidence-based policy explain this disastrous response to the coronavirus or the intolerable 15.5% deaths to confirmed cases ratio.

Should we commend Boris Johnson?

[ltr]

That the British media, nor Labour nor academics with the exception of Simon Wren-Lewis are willing to criticize Conservatives for the disastrous response to the coronavirus is astonishing and disheartening to me. This speaks decidedly poorly of British democracy and proper valuing and concern even for British health.

[Jim]

Rigorous and fearless - as if. Imagine some project is proposed, consultants are called in, eventually a kickoff meeting with a civil servant. Any sane consultant will find out what they want the answer to be, no good having to re-fudge your report and evidence to get your invoice past the paymaster. Ask 'and what is the view of the Minister'.

Very rarely the question will be a genuine question - what is this or that all about. Much more often some 'line' has been decided, your job is to make sure your report backs that up. Conflicts frequently arise, find out who holds the purse strings. So much for evidence based, life is too short to discover 'the truth' when it comes to politics.

Which brings me to Gove. Remember the Free Schools saga. What is not so remembered are the teachers who were in Gove's way and got pushed out. So they went to tribunal and usually won (excellent Ofsted results etc). Big payouts, long faces at local authorities and NDAs all round. Gove sailed blithly on and we don't ask whether the Free Schools turned out to be much better.

Consult any lawyer and you will find he who controls the evidence controls the trial. Therein lies the rub with government jobs. As if sorting fact from fiction were not enough, having bosses with an agenda and control of what is an acceptable answer makes 'evidence and data based' decision making the stuff of fairyland. Expect little from Gove and Dommy.

[Dipper]

Oh look people criticising the Government's performance on Covid, like they are the masters of Covid, like they have this virus fully understood, know it's every feature, and have an ability to model public policy and understand its effects and collateral consequences in infinite detail.

Cases ( 284,900)

What is a 'case'? Is someone who has been exposed but generated no symptoms or antibodies and doesn't now have the virus a case? Do they need antibodies to be a case? And if we are widely testing and 'actual' incidence is low, then aren't we just picking up false positives?

Alastair Campbell is all over this virus, expertly using his degree in Modern Languages to navigate the intricacies of virus biology and the complexities of the immune system like he's James Frigging Watson. Yeah lets just do what he says.

And where's that second wave all you critics have been promising me every time someone pops out to walk the dog?

[ltr]

July 4, 2020

Coronavirus (Deaths per million)

Belgium ( 843)
UK ( 651)
Spain ( 607)
Italy ( 576)

Sweden ( 537)
France ( 458)
US ( 400)
Netherlands ( 357)

Ireland ( 353)
Canada ( 230)
Switzerland ( 227)
Luxembourg ( 176)

Portugal ( 157)
Germany ( 108)
Denmark ( 105)
Austria ( 78)

Finland ( 59)
Norway ( 46)
Greece ( 18)

[ I suppose that evidence-based coronavirus policy for Boris Johnson and Conservatives means all those other British dying. ]

[ltr]

Conservative policy on protecting the British against the coronavirus has been disastrous, but the routine failure of writers beyond Simon Wren-Lewis to point this out tells me there is an important weakness in British democracy or the British understanding and appreciation of human rights. We really did need to be properly protected against the spread of the coronavirus, but that was evidently unclear to our governors.

When then is the criticism?

[ltr]

As for my criticism of Conservative coronavirus policy, I have been frightened for my family and myself through this period and I really do not care to be frightened because I have no sense the government cares a fig about my family or me. Actually I have no sense now that Labour cares either.

A chilling sense of British government now.

[Dipper]

@ ltr

yes I get you are worried about this. I'm not that enthusiastic about getting it myself. But its not as though Johnson stood for election on a promise to introduce a pandemic if he got elected.

Neither you nor I know what it is that determines how much, or how rapidly, this spreads and what determines whether people hardly notice it or die form it. The stats don't mean much, as we cannot trust the comparisons as they aren't done on a like for like basis, and countries are not identical.

What would you have the government do? Lockdown until there's a vaccine? We could still be here in fifty years time waiting for a vaccine whilst society has collapsed around us.

Covid shows us the difficulty of evidence based policy. Science works from performing experiments under controlled conditions, or conditions that can be precisely characterised and differences understood. It works through having reliable measurement techniques for well understood features, which are precisely defined. We have hardly any of that right now for Covid

Science is really hard and I'm not claiming to have been particularly good at it. But these clowns calling the odds on Covid are just making themselves look stupid at best and malicious at worst.

[ltr]

Dipper,

I do appreciate the thoughtful and kind response, and am much more satisfied taking the perspective you offer. The need I realize has been to feel as though decisions made for us were "personally" made.

I am grateful for and comfortable with your "kind" response.

[Dipper]

ltr - thanks for your comment - appreciated. Hope this virus avoids you and your family and that somehow we can get through this quickly and get back to 'normal'.

[ltr]

Hope this virus avoids you and your family and that somehow we can get through this quickly and get back to 'normal'.

[ Perfect and the same, and as for "normal" there will be changes for us collectively and personally and I am quite unsure what either will be but am thinking.

Chris Dillow should help us think the coming normal through.

Be careful and well, and remind those about you as I will. ]

[John Street]

I like your post - excellent food for thought.

[Robert Mitchell]

Hedging does not need evidence. You can bet on a stock moving up and also bet it will move down, and get out of both trades while one of them gives you a greater profit than the other cost. Butterflies may have mediocre returns but when you're leveraged, even 10 basis points on a trade can be worthwhile.

[D]

Typos.

2 * "one" instead of "on" in last para

[aragon]

Re:Original Post.

Are the systems that complex in aggregate?

What mental model do the three (Boris, Dom and Micheal (Gove)) have? (aka UK Government)

Action orientated but no coherent sense of direction, lazier fare, and it will be alright on the night (blind faith)?

Polices depend on beliefs, but Boris does not have them (either of them).

https://www.economist.com/free-exchange/2009/04/29/in-defense-of-the-gaussian-copula

"You can have the most complicated and complete model in the world to explain asset correlation, but if you calibrate it assuming housing prices won't fall on a national level, the model cannot hedge you against that happening."

"housing prices won't fall on a national level" - Who would have thunk it?

"but that banks began to hold mortgage assets on their balance sheets"

No Banks sold CDO's to investors...

"it screams incompetence."

No comment required.

"But like many useful innovations, when it is exploited by the wrong people it can cause great harm."

Like people with the wrong mental models or world view...

Finance is not the place to find successful data driven outcomes.

Black-Scholes-Merton (BSM) turned out well?
Long Term Capital Management - turned out to be pretty short term.

Of course Michael Gove should be scanning the horizon not examining in detail his own feet. Especially the one in his mouth.

All that remains of the UK economy is an empty husk, all that is left, is for the parasite known as Finance to die, having killed it's host.

It's the 'vision thing' and Singapore is not it. Sorry to obtuse but it's early

[Lee]

The people are sovereign.
Not Johnston
Not the Tories
Not the Govt
Not Londinium
But the people.

The living will of the people is violated everytime these autocratic clowns are allowed to fester.

The people have no means of removing deadbeats.

The people should decide every detail by vote.