By now you’ll all have seen Robert Yeh’s paper showing that there is no evidence that parachutes save the lives of people jumping from a plane. This raises a question for all social scientists: what counts as evidence? I suspect we tend to overweight some kinds of evidence, and underweight others.
Yeh’s paper is a lovely illustration of a general problem with randomized control trials – that they tell us how a treatment worked under particular circumstances, but are silent about its effects in other circumstances. They can lack external validity Yeh shows that parachutes are useless for someone jumping from a plane when it is on the ground. But this tells us nothing about their value when the plane is in the air – which is an important omission.
We should place this problem with RCTs alongside two other Big Facts in the social sciences. One is the replicability crisis. This does not afflict only psychology but other disciplines too: Campbell Harvey has said that most findings in financial economics are “likely false”. The other (related) is the fetishization of statistical significance despite the fact that, as Deirdre McCloskey has said (pdf), it “has little to do with a defensible notion of scientific inference, error analysis, or rational decision making” and “is neither necessary nor sufficient for proving discovery of a scientific or commercially relevant result.”
If we take all this together, it suggests that a lot of conventional evidence isn’t as compelling as it seems. Which suggests that maybe the converse is true. Perhaps there are some types of evidence we under-appreciate. For example:
- Non-quantitative observation. Some economists are sniffy about “casual empiricism”. But there needn’t be anything casual about it. In sociology and anthropology ethnography is an acceptable, laudable, tradition. And it used to be in economics. Adam Smith’s work was based upon decades of observation of real people. And perhaps Coase’s The Nature of the Firm (pdf) was founded on ethnographic evidence.
- Personal experience. Just before Christmas Betsey Stevenson and Robin Hanson argued about sexual harassment. A big part of the issue was what counts as evidence: Betsey emphasised the personal experience of victims, whilst Robin emphasized thought experiments and surveys. For me, Betsey has a point. One thing I find irritating about some “as if” modelling is that it ignores personal experience: Greg Clark’s story (pdf) of the rise of the factory system is one example, and the idea of unemployment as arising from a taste for leisure is another.
- Memory. The Easterlin paradox – that economic growth doesn’t make rich societies happier - has been well challenged in statistical terms. But what lends it credence for me is not so much the statistical evidence as memory. UK GDP per head is more than 50% higher than it was 30 years ago. But are we really happier than we were then? My memory suggests not. Of course, this doesn’t prove there’s zero link between GDP and happiness, but for me it’s evidence of a weak one.
- Music. Bruce Springsteen’s album Born in the USA – which is now a third of a century old – told us that deindustrialization was linked to threats to male identity and a yearning for the past. He was years ahead of political scientists in highlighting the issues that contributed to Trump’s election. Similarly, the rise of punk in the 70s betokened harsher economic and social times; the recent popularity of drill music alerts us to the fact that inner city youth have violent lives; and the ubiquity of homesickness in old folk songs suggest urbanization was accompanied by a sense of loss. And there are countless individual songs that tell us something. To take just one example, the Pistol Annies’ Got My Name Changed Back is gloriously vivid corroboration of Andrew Clark’s finding that we adapt to traumatic events. The Boss was exaggerating when he sang “We learned more from a three-minute record, baby, than we ever learned in school”, but he had a point.
- Literature. Eugenio Proto and colleagues have used machine reading of thousands of books to measure historic happiness. But we can of course use them less quantitatively. Surely Jane Eyre or Tess of the D’Urbervilles tell us something about the lives of real women in the 19th century.
You might object that these informal methods don’t generate proof. True. But do we need it? In finance, if you wait for proof that a strategy works before adopting it, you will buy at the top of the market. It’s entirely reasonable to act on probabilities. For example, if there’s a small chance of a disaster than can be cheaply averted, it’s wise to do so without waiting for more evidence.
In making this Feyerabendian argument for empirical diversity I am not discounting conventional scientific methods. Very often, these are vital. For example, I find the most convincing evidence that markets are prone to irrational bubbles to come from laboratory experiments (pdf) which show them to be commonplace: “real-world” mispricings can be explained in other ways. We should ask: what would count as evidence here? The answer will vary from context to context. I fear this trivial thought is under-appreciated.
Evidence for some is something to be carefully controlled, framed and measured out in very limited quantities. One can include the techniques of shredding, misfiling, denying, economy and twisting. Then of course you can go in for proposal first, evidence later. This is not so different from the scientist's hypothesis/look around/experiment game but with ample opportunities to ignore the bits that don't fit.
Personally I am in favour of having a wide ranging look at evidence, good and suspect, a peering into dark corners and a turning over of stones. But too often evidence gets shut up in boxes of a physical and administrative and mental kind, bad decisions quickly follow.
Posted by: rogerh | January 11, 2019 at 10:34 AM
Thank you Chris. We can agree on many things. I did not know about Yeh's paper. Interesting read for me. I was thinking about your conclusion arriving at Feyerbend's work.
Could you not have also driven towards MacIntyre's engagement with the social science too? For example, it is a key of his engagement with the 'social sciences' that these 'human' theories have to 'live' with a well known 'counter examples' with their own models, methodologies and contrasting results. MacIntyre gives an example for patterns of crime in large cities. Your 'what would count as evidence here?' also made me think of the debate between Basil Bernstein and Labov about the ablity of working class kids purported 'restricted language codes'. Who you agree with depends it seems to me on how compelling you find Labov's 'counter' evidence.
The following is more of an open question. Is it not the case that Economics as a discipline has been 'reinvented' in modernity, so that the practioners at university and we the citizenship are no longer conscious of the features of the pluralist real,'social science's you allude to above? That politically, this is Economics great merit for the Right: as a 'proper' 'hard-science' discipline. That the Feyerbend like 'playful', 'anything goes' physicist pushing the boundries, breaking the rules of his/her scientific paradigm has been subverted into a pretence that Economic 'hard' science requires a rigid mathmatical,deductive structure that if deviated from, renders the sinner: a non-economist conducting non-science.
Posted by: MikeW | January 11, 2019 at 12:36 PM
I think economics has never really come to terms with the notion of a social science. This is the root of many ills and leads me to agree strongly with your post. It astonishes me that economists get seemingly no exposure to issues in the philosophy of science in university instruction.
The older I get, the more grateful I am to my own teachers some 30 years ago, who introduced me to all that stuff, as well as writers like Elster, Hischmann, Marglin, Glyn, Kalecki and so on, as well as the whole range of theoretical perspectives beyond the mainstream. I genuinely feel sorry for modern day students of the subject.
Posted by: rjw | January 15, 2019 at 10:09 PM