On confined expertise

"Why is Richard Dawkins such a jerk? asks Matthew Francis.Some recent laboratory experiments help shed light upon this.

Researchers at Microsoft first looked at professional pundits' predictions for the NBA playoffs - which are best-of-seven matches - conditional upon seeing the results of previous games in the series. They found that the professionals' forecasts were more internally consistent than amateurs' forecasts, suggesting that the experts are more rational than the amateurs. However, when the pundits were then asked analogous questions about the probability of seeing red and black balls drawn from an urn, this superior internal consistency disappeared. They conclude:

Expertise fades in the lab even when the lab explicitly mimics the field.

This is consistent with some earlier research (pdf) by Steve Levitt and colleagues, who have found that professional poker players' ability to use minimax strategies deserts them when they are asked to play abstract games in the lab.

This corroborates the claim made by Dan Davies in a classic post: there is no such thing as a general purpose expert. As Richard Feynman said, "a scientist looking at nonscientific problems is just as dumb as the next guy.”

Instead, as George Loewenstein has said (pdf), "all forms of thinking and problem solving are context-dependent." When they are taken out of their context, experts lose their expertise.

Richard Dawkins is the poster-boy for this. He's a brilliant popular scientist, but can be a prat in other contexts. But of course he is not alone. Think of Niall Ferguson or James Watson or Tim Hunt or Steve Levitt, or many businessmen in politics. Peter Spence's advice seems, therefore, reasonable:

It'd be great if we all treated the opinions of Nobel prize winners outside of their field of expertise as basically irrelevant.

For me, there's something sad here. It seems that the scientific approach - such as the question "what counts as evidence here?" - is hard to apply outside of one's own field. In this sense, perhaps science and religion are closer than Professor Dawkins would like to think: just as many Christians forget their Christian principles when they are outside the church, so scientists forget their scientific principles when they are outside the lab.

Comments

[Matt M]

Matthew Francis is a new age liberal more consummated by feewings than rationality or the truth about reality. His article attacking a solid study explaining the lack of women in STEM certifies this. He should stick to being a calculator.

As regards experts speaking out of their own area not having expertise consult:

http://press.princeton.edu/titles/7959.html

People versed in basic probability, hypothesis testing and data understanding make better judgement than home field experts without those skills.

Dawkins might be occasionally wrong about this or that issue, but usually more correct than most religious nuts and social justice warriors.

[Shuggy]

Absolutely. Always think this whenever Dawkins rails against theology departments in universities, despite the fact that they're almost certainly converting more people to liberal versions of religion, agnosticism or atheism than he ever will. If he can't be bothered to find out what's going on where he actually works, his opinions on non-scientific stuff aren't really worth listening to.

[Magpie]

"In this sense, perhaps science and religion are closer than Professor Dawkins would like to think: just as many Christians forget their Christian principles when they are outside the church, so scientists forget their scientific principles when they are outside the lab."

Excellent and excellently put.

[Adam]

I wonder if the same would apply to philosophers? Let's get them in the lab next time.

[e]

Interesting post; yet one that I would once have thought was silly. Didn’t we used to ‘know’ this stuff instinctively?

[chairman]

Man, that Slate article was really exceptionally bad.

[chris]

@ Matt - I'm surprised you're bringing Tetlock into it. You could read him as corroborating my point - that experts' knowledge is confined, because it doesn't extend to the future of their discipline. I'd argue that there's no such thing as an expert on the future.
I agree that basic stats can do better than futurology - and say so often in my day job. But they only allow us to know the odds; they don't give us anything like certainty.

[reason]

What is Niall Ferguson doing in that list. He is only good at polemic. .

[PDV]

The important point with superforecasters is that it is a learnable skill; it is possible to be a expert in domain-general prediction.

[Magpie]

@ Adam,

"I wonder if the same would apply to philosophers? Let's get them in the lab next time."

Believe me: I share the feeling. You could add economists to those who enjoy pontificating way beyond their area of expertise.

By those things in life, a few days ago I myself wrote precisely about this.

[www.greenworld-nature.com]

Nice share,,

[bob]

really? thought you were too intelligent to be bothered by the supposedly controversial utterances of hunt, watson and dawkins...