Responsible policy-makers should consult widely before reaching their decisions. This is trivial. It is also wrong, according to new research. A neat experiment at Nottingham University shows how consultation can be counter-productive.
They got subjects to say whether a couple of paintings were by Paul Klee or Wassily Kandinsky. The subjects were split into two groups. One group comprised individuals making their own decisions. The other comprised individuals who were assigned to teams of six and allowed to consult team members.
And members of the teams did worse. Whereas only 29% of individuals got both paintings wrong, a whopping 51% of team members did so - twice as many as you‘d expect by chance.
There was, however, no significant difference in the proportions getting both questions right: 38% of individuals versus 36% of team members.
Consultation, then, increases the chances of a bad decision, without improving the chances of a good one. What’s more, people don’t realize this; most said that they found the consultation process helpful.
The reason for this is that people are misled into giving wrong answers by team members who are irrationally over-confident, because these send out more (false) competence cues. “Individuals who know little are swayed by those who know less” say the researchers. (This is more true for women than men, but let’s not go there.)
There is, though, a crucial thing about this result.
It all hinges upon whether the answer to a question is demonstrable or not. If it is demonstrable, then an expert who knows the answer can prove that he does so by using logic or evidence, and non-experts will defer to him. Consultation will then work, simply by virtue of bringing expertise into play.
But some knowledge is non-demonstrable. The expert might be able to distinguish between Klee and Kandinsky, but he’ll find it harder to demonstrate his expertise to laymen than, say, a mathematician will be able to demonstrate that he knows the solution to an equation. And where knowledge is non-demonstrable, people might follow false experts.
This, I fear, might have widespread implications. Juries, for example, are asked to find a non-demonstrable answer.
It suggests that public consultation exercises must be carefully designed - or least they should, insofar as the object of them is to reach the correct decision, which might not always be the case.
'Consultation' covers so many sins that equating it with groups of people trying to work out who produced a particular painting is somewhat meaningless. To take an example dear to my heart, consultations on planning applications involve asking people for information they have near-direct access to; i.e. the expected impact of the given development on their residential amenity. The information that goes into informing that expectation can be 'led' in the manner that you describe, but given that most consultation responses are via the post it's not clear that that's really a factor.
Posted by: Adam Bell | September 16, 2011 at 05:09 PM
Of course, "consultation" means many things. The point of this experiment is to show that there are circumstances when it doesn't work - such as eliciting opinion rather than genuine knowledge.
Posted by: chris | September 16, 2011 at 06:15 PM
Experimental design fail, as the youth are fond of saying these days.
Firstly, the paintings are clearly of a different style. I could recognise each artist's style without recognising the specific works. And in fact, this is a "demonstrably" correct judgement - the very property the authors were trying to get away from.
That, along with the implication that one work is going to be by Klee and one by Kandinsky, should pull answers from the "one right" category to the "both wrong" and "both right" categories, even for groups with zero exposure to either artist.
It also blows all their p values out of the water, since they are no longer dealing with statistically independent outcomes.
A moment's intelligent thought would have at least produced a design with an entirely different choice for a second work.
Posted by: Andrew | September 16, 2011 at 06:53 PM
Second major fail is that, as they discuss, the answers of people in the "team" treatments were not statistically independent of each other. This does not matter for assessing the p values of team treatment influencing accuracy (ignoring my first point). However, it remains to be shown (and I can't be bothered to calculate) whether the tendency to wrong answers was statistically significant when analysed per team units. (There were only 1/6 the "n" for teams as the individuals in teams.)
In other words, although the team treated individuals had a statistically significant tendency to underperforance relative to the individually treated, all it shows is that they had a significant tendency to align with team performance.
Whether team performance itself was sub-par with statistical significance has not been shown.
Made up analogy for this point: England supporters tended to bet the wrong way in the crucial match with Argentina compared to indifferent. Therefore team supporters perform worse in match prediction (p<0.00000001). But no. Although the individual performance of this group was of course not statically independent (look at my p value!), the performance of the group may have random, or even good over the long run. Too few trials (only one of the group) to tell.
They should address this before speculating about over confident ignoramuses (who after all, should perform randomly, not worse than randomly).
(Reminds me of the tale told me by a psychology professor, who had a brilliant medical student who hated the psychology course, and protested by getting negative 100% in the true/false multiple choice exam.)
To conclude, sorry about the snarky tone, but I am vaguely irritated a poorly thought-out experiment. It's not even my job to know about this. It's their job though, isn't it? They actually get paid for it. They've wasted money, and at least 342 people's time.
Posted by: Andrew | September 16, 2011 at 06:54 PM
Chris you have obviously nerver sat on a jury; I have several times. The idea you can lead 12 people from off the street in a given direction they do not wish ( in a criminal trial ) to follow is mere ignorance. When the Liberty of the subject is in issue my experience is that Jurors take their duty very seriously indeed. John Lilburne was essentially correct. Better a Jury than a Judge. I would not try to be overconfident or bossey in the Jury room if you ever serve they will mince you.
Posted by: Keith | September 16, 2011 at 09:28 PM
Of course, one of the reasons to consult isn't to make a better decision in some Platonic sense. It's so that the opinions of other people are represented.
It may well be the case that one less ratgasm overall is worth some subgroup not getting screwed.
Posted by: Alex | September 18, 2011 at 03:32 PM
I've have let this stew for a week now and it remains a crap experiment, not a "neat" one.
You can't have two trials that are statistically dependent on one another and then go around analysing results as if they were two coin tosses.
Is this one Coke or Pepsi? How about the other one, is that Coke or Pepsi? Weird! I would expect half of you to only get one right!
I am alleging fundamental professional incompetence here. I would welcome someone to show me how I'm wrong.
Posted by: Andrew | September 23, 2011 at 07:18 PM
teavesti Is this one Coke or Pepsi? How about the other one, is that Coke or Pepsi? Weird! I would expect half of you to only get one right!
Posted by: travesti bursa | September 25, 2011 at 05:57 PM