Outcome bias & cargo cults

Is it time for a Nobel prize for managers? asks Andrew Hill. The Platonic essence of a QTWTAIN, says John.
He’s right. The reason for this is that our judgement of managers is clouded by the outcome bias.
When management theorists talk about bosses they very often don’t take one early in his career at a firm, and say: “This is his strategy. It’s good because x, y, z. He‘s a good manager. His firm will succeed.” Instead, they use hindsight. They take a firm that’s succeeded and infer from this that the CEO and the strategy was good. Or they see a firm struggling and infer the CEO is bad.
This is the outcome bias; we judge managers not by the quality of their work, but by their results.  In this sense, management differs - to least some degree - from most of the disciplines which get Nobel prizes. We can assess the work of, say, Chris Sims, independently of anything it has led to. But we rarely assess managers independently of results. This is wrong because it ignores the role of luck. A poor boss could get lucky and succeed and  a good one could get unlucky and fail.
Duncan Watts’ Everything is Obvious - which I strongly recommend - gives an example of how tricky it is to slough off outcome bias. He compares the iPod and Betamax video tapes:

The iPod was exactly the kind of strategic play that the lesson of Betamax…should have taught them would fail. The iPod was large and expensive.  It was based on closed architecture that Apple refused to license, ran on proprietary software, and was actively resisted by the major content providers. Nevertheless, it was a smashing success. So, in what sense was Apple’s strategy better than Sony’s? Yes, Apple had made a great product, but so had Sony. Yes, they looked ahead and did their best to see which way the technological winds were blowing, but so did Sony. And yes, once they made their choices, they stuck to them and executed them brilliantly; but that’s exactly what Sony did as well. The only important difference…was that Sony’s choices happened to be wrong while Apple’s happened to be right.

This sort of thing is, of course common in sport. I’ve described how Arsene Wenger is the victim of it now. And a new paper shows how NFL coaches are prone to the bias. The authors show that coaches tend not to change their teams’ offensive strategies after victories, even if these were narrow, but they often change them after defeats, even if the defeat was very narrow or if it were due to defensive errors rather than attacking ones. They say:

One can easily misinterpret a favorable outcome as justification for a given strategy, overriding more subtle evidence to the contrary.

All of this raises a question. Could it be that management is, to some extent, like a cargo cult? It performs certain peculiar rituals, after which nice things sometimes happen and sometimes not. However its members infer from this not that the rituals are (largely? partly?) irrelevant to the outcome, but instead that some of their performers have skill whilst others do not. And in this way, irrational practices persist, and their luckier practitioners get rich.

Comments

[Luis Enrique]

I'm a bit confused - do you think it is meaningful to say football managers are good (or bad)?

I don't see the basis for questioning that there is something meaningful called "management" that may vary in quality, with real effects.

http://www.stanford.edu/~nbloom/index_files/Page371.htm

[CMcM]

I share your scepticism about management as worthy of a Nobel - but I'm not so certain about the arguments you have deployed in arguing for this position.

I suppose what I demur from is your contrast with economics: it may well be true that the work of Chris Sims and others of your tribe can be judged 'independently of anything it has led to', but that is hardly a sufficient judgement per se, at least not in the long run. Unless you want economics to be seen as a hermetically sealed discipline, akin to theology perhaps, surely there must be some reference not just to the explanatory powers and internal rigour of any particular economic approach, but also to to its predictive powers?

[WH]

I know it's just an illustrative example, but surely it doesn't make sense to compare Beetamax and ipods without bringing in VHS and the dogs-dinner of non-ipod mp3 players? The ipod succeeded because it was easier than its competitors. But I guess this is the same point: looking in retrospect at which factors were _really_ important.

[Churm Rincewind]

Just in passing, Duncan Watts' analysis of the comparative failure of the Betamax format is completely erroneous. What Sony failed to realise was consumers wanted to buy content, not technology. So initially Sony did not invest in content; VHS did, on a massive scale. That's why they won out.

Just thought I'd mention it...

[David G]

Isn't this issue the same as for all Nobel's however? They aren't given out to scientists or economists who investigate a certain area in the most effective way only to prove nothing; they are given to scientists who, by luck, judgement or perseverence, stumble onto great new things. (and sometimes to geniuses who make staggering breakthroughs through a leap in understanding, admittedly)

When was the peace prize last given to someone who took the optimum possible strategy, avoiding worst case scenarios but who was in such a position that war or stalemate was absolutely impossible to avoid?

In Obama's case for instance, the award was given *before* he even delivered.

In which case, a Nobel for management seems entirely plausible.

[Keith]

David G is probably right. All prizes are really results based and that depends on luck and a judgement about what is important and constitutes success. The economics "Nobel" is not real as it forms no part of the will of Nobel. He did not consider economics a Science and considering the failure of a lot of economic theorising to be useful at all, it is open to doubt if Economics is worthy of a prize named after Nobel but set up by Bankers. All that money is not enough they wanted a Nobel prize like real Scientists! Poor darlings! Actual Scientists can prove that their theories work and are based on evidence and can make things like space rockets that fly to to the moon thanks to Newton etc. Cooks can bake cakes that taste nice following their recipes. If only Economics and "management" were that useful. Namely worked reliably without being biased by political prejudice. As Albert Einstein put it " there is no such thing as Jewish Science or Aryan Science, there is knowledge and there is ignorance."

[Neil]

"Or they see a firm struggling and infer the CEO is bad."

Of course, this CEO could still win the prize provided he condemns 'red tape' vociferously enough.

[Tim Almond]

Duncan Watts should do some more detailed research into both products.

1) The iPod was large and expensive? Large compared to what? I've recently re-read the slashdot article about the iPod being released and the iPod certainly wasn't large. And it was expensive at version 1, but by v3 wasn't much more expensive than the Creative players.
2) The iPod was based on closed architecture? Is he talking about the transfer mechanism to the device, because I think all players at that time were closed (certainly Creative players were). If he's talking about formats, Apple do have their own encrypted flavour of AAC, but they also supported MP3. In fact, one of the reasons why Sony music players never took off is that they were even more closed, supporting ATRAC and not MP3.
3) If the iPod was "actively resisted" by content providers, why was iTunes Music Store the first major music store? It was Apple that convinced the music industry that this could work.
4) Sony had a great product in Betamax? That's a questionable claim. The problem with early Betamax machines was that they only recorded for 1 hour because they used smaller tapes. VHS were bigger and could hold 2 hours. Betamax later solved this problem, but by then, VHS was on a roll (so video shops had limited betamax tapes to rent etc.).

Apple just made very good products. And did so when they had Steve Jobs at the helm. He's the man who on day 1 of his return told the board that Apple's direction was going to be towards industrial design.

I'm really curious how you arrived at your view about management? What manufacturing, retail or service companies have you worked in?