The journalist's fallacy

Tim objects to this:

A generation of youngsters may be developing a skewed view of sex from pornography, a court has heard, after a 12-year-old schoolboy raped and sexually assaulted a younger girl after copying a hardcore film he watched on the internet.

He's right to object. This is an example of what we might call the "journalist's fallacy" - though in this case it's perpetrated by someone from a profession even more ignorant of statistics than journalists. I mean by this the tendency to draw strong inferences from one or two observations, without asking: are these observations a representative sample? In this example, we have a good reason to suspect not; 12-year-old rapists are rare, whilst internet porn is ubiquitous.

I call this the journalist's fallacy (though it's related to the availability heuristic) because journalists are especially prone to it - perhaps because they know that a human interest story or lively image makes for "better writing" than statistical evidence, and they mistake good writing for good thinking*.

Take just three examples.

In the Times, Anushka Asthana writes of "women friends who say they felt they had to slip their engagement rings into their pockets before job interviews". But she doesn't say that, among people of fianceable age (22-29), women actually earn more than men, suggesting either than such behaviour is unnecessary or that employers are easily fooled.

Zoe Williams writes: "My son and his friends are constantly worrying about sugar, how it makes you fat, how you can find it even in places you don't expect." But she doesn't tell us whether her children are representative or not.

Joan Smith writes:

The problem is how many adults ignore health advice altogether. Alcoholism is a huge social problem and so is obesity. A couple of days ago, I walked past a shop where a hugely overweight assistant had slipped outside for a quick cigarette, and I couldn't believe that someone who already had a life-shortening condition was blithely risking lung cancer as well.

She fails to see that one lardy doesn't provide evidence of "many".

I mention these not because they are especially bad examples - I'm sure you could find much worse - but simply because they are recent ones, taken from three different papers today and yesterday. That they are so easy to find might be a sign of how common the error is.

Not only is the error (I suspect) common, it might also be costly, in three related ways:

- In relying so much upon personal experience and anecdote, other information gets downgraded. That information is the hard-found evidence provided by serious statistical research. The journalist's fallacy can easily lead to an anti-intellectualism in which personal, biased and limited experience is prioritized over proper social science. One example of this could be Mary Ann Sieghart's assertion that grammar schools would increase social mobility, oblivious to the fact that the hard research on this issue is, ahem, more ambiguous.

- The ignorance of serious social science breeds overconfidence. The message of lots of research is that facts are hard to come by, exceptions are common, and evidence ambiguous or missing. Such warnings rarely surround anecdotal evidence.

- The combination of overconfidence and generalization from what are often vivid and extreme cases (our 12-year-old alleged rapist) can lead to demands that "something must be done". It might be no accident that the increased influence of columnists upon politicians (if not the public!) under the last government coincided by legislative hyper-activism.

* It does not, of course, follow that bad writing is a mark of good thinking; this is the sociologist's fallacy.

Comments

[pablopatito]

I have a degree in Economics from a good University and all those three years really taught me is that the economy is really complex and I haven't a clue about it. The more you learn the less you know..

Yet I have mates who have never studied the subject in their lives telling me they think Paul Krugman is an idiot.

And most journalists are just "your mates down the pub", with English degrees and self-confidence.

[98economics.blogspot.com]

Call it the moralist's fallacy...

(@pablo: if the more your learn, the less you know, then your mates down the pub do indeed know better than PK ;-)

[Chris]

I call it a 'survey of one'. I'd noticed that journalists tend to fall into this trap. Never let the truth stand in the way of a good headline!

[Phil]

Correlation really doesn't imply causation, and most actual correlations aren't that strong statistically. Most sociologist, most of the time, are providing negative evidence at best. But that's a really depressing thought, innit?

[Keith]

"Yet I have mates who have never studied the subject in their lives telling me they think Paul Krugman is an idiot."

May be you need some new mates? Prof.Krugman seems like a very nice person, although I have not had the pleasure of an introduction, and very bright. Unlike your friends.

[Will Davies]

I can stomach the statisticians vs journalists debate, as long as the former also accept that they have a political and cultural history. The notion that there is an objective view of society dates back to the French Revolution, but many of the techniques available for analysing it only emerged in the late 19th century and then matured in the 1920s and '30s (sampling techniques and econometrics).

With apologies for sounding like a post-modernist, I think there are good reasons why an understanding of statistical normality is beneficial to public debate, but very few statisticians ever articulate them. In a sort of Dawkins/Goldacre fashion, they are presented as unarguable and obvious. I agree that statistical sophistication is supportive of an enlightend public sphere, whereas knee-jerk op-ed-ism is far less so. But there are justifications for this, which can and should be articulated. It's irritating when quant people just assume that they're dealing in reality, and everyone else is just partaking in an expanded version of a Stoke Newington dinner party.

[Blissex]

«I call it a 'survey of one'. I'd noticed that journalists tend to fall into this trap. Never let the truth stand in the way of a good headline!»

It is not a trap: it is their job. Journalists get paid to entertain their readers, that is to spin stories that excite them and confirm their prejudices.

Blame the readers for buying that. The journalists are just responding to a market for news-as-entertainment, or even news-as-propaganda.

The day that newspapers increase their sales (or their proprietors cover higher losses) by reporting realistic, sober stories, you will get them.

[BenSix]

But you must admit that the comic value of the implication that Zoe Williams feels she and her experiences are representative of the general public's is great indeed.

[Peterson]

Is it down to ignorance or laziness? Perhaps Asthana, Williams and Smith are all well aware that their own experiences and those of their friends are not necessarily representative of anything - but searching for real evidence would have required them to do some actual work. And since, as Blissex notes, the role of columnists is to entertain and provoke rather than inform, they wouldn't necessarily have had a better article (from that point of view) at the end of it.

If I was running a newspaper I would not employ any regular columnists, because after a while they always get lazy and start either to repeat themselves, to feed off other columnists or to deal in personal anecdotes as is the case above.

And then, finally, when they really can't be bothered at all any more, they resort to the final refuge of the desperate British newspaper pundit: the article bashing cyclists for running red lights.

[Ralph]

Please could you explain "availability heuristic"?

[Frances Coppola]

Rick calls it "I Know A Man Who....". I believe the correct term is "generalising from the particular". And we all do it, a lot of the time (and I have just given a fine example of it!)

[Andrew Curry]

As the saying goes: the singular of 'data' isn't 'anecdote'.