On incompetence

Should ineliminable incompetence play a bigger role in economic and political thinking?

My trigger for asking is a piece in the Times by Oliver Kay on the appalling mismanagement of football clubs such as Blackburn Rovers, Charlton, Leeds and Blackpool. But of course, incompetence is much wider than that. Trains are late and overcrowded; building projects run over time and budget; utilities, banks and broadband providers often have poor quality service that can’t wholly be explained by profit-maximizing; and you all have examples of bad management in your own workplace.

Which brings me to a paradox. Whereas our own eyes tell us that incompetence is ubiquitous, standard economic theory regards it as merely a temporary deviation. It thinks that agents are incentivized to optimize; that badly managed assets will be bought cheaply by people better equipped to run them; and that competition will drive incompetent firms out of business.

But this doesn’t happen – at least not fully. Even the best incentives can’t put in what God left out: I have an incentive to become a Premier League footballer or Nobel laureate, but you’d be ill-advised to bet on me becoming either. Credit constraints, among other things, prevent assets flowing to their best managers. And Bloom and Van Reneen have shown (pdf) that, in all countries, there is a “long tail of badly managed firms”, which tells us that competition doesn’t fully weed out idiots.

The failure here, though, isn’t merely one of Econ 101. Real people make the same mistake.

We know that stock-pickers pay too much for growth (pdf) stocks and ones on the verge (pdf) of collapse. One reason for this might be that they underestimate the ubiquity of incompetence – the fact that managers can’t grow firms or turn them around as easily as thought.

And in politics neoliberals overstate the extent to which incompetence can be removed by competition and/or well-paid management whilst the left exaggerates the benefits of nationalization. Not only do we over-estimate state capacity, but we also over-estimate management capacity*.  Perhaps we underestimate the extent to which success is a matter of luck and then habit plus marginal gains.

Even the English language misleads us here. Think of the synonyms and near-synonyms for incompetence: mistakes, errors, ineptness, ineffectual, bungling, incapability and so on. All of these suggest a shortfall from a norm of competence. But perhaps the opposite is the case – that it is incompetence that is the norm. As Paul Ormerod said (pdf):

Failure is all around us. Failure is pervasive. Failure is everywhere, across time, across place, and across different aspects of life…Yet the existence of failure is one of the great unmentionables.

Here, we – or at least we economists – perhaps need a paradigm shift. It’s a cliché that economics has a powerful theory of optimization. But we also have a powerful theory of incompetence. The cognitive biases programme explains how it persists. For example, the planning fallacy predicts that complex projects will over-run budget; the overconfidence effect tells us that assets will sometimes be bought not by those best able to run them but by those who over-estimate their ability to do so, such as the Venkys at Blackburn; Bayesian conservatism predicts that bosses will under-react to feedback; and so on. I might add that the optimism bias causes us to under-rate these mechanisms.

My point here is perhaps an old conservative one – that our institutions, public and private, are deeply imperfect and might well remain so, that there is, as Adam Smith said, “a great deal of ruin in a nation.” Remembering this might not be good politics: there’s a danger it’ll reinforce the status quo bias. But it would be good for our mental health.

* One thing I found irritating about much of the coverage of the collapse of BHS was the lack of distinction between the failure of the company, which was forgivable and perhaps unavoidable, and the plundering of the pension fund, which was neither.

Comments

[technicalities]

I would change "Bayesian conservatism" to "non-Bayesian conservatism" there: epistemic conservatism is a deviation from the Bayesian norm, after all.

(Long-time admirer, first-time nitpicker.)

[Blissex]

«the lack of distinction between the failure of the company, which was forgivable and perhaps unavoidable, and the plundering of the pension fund, which was neither»

But asset stripping is a core human right -- also where would be the UK economy today without asset stripping in the City and elsewhere? :-)

[PDV]

There is a problem with at least one of your examples of mismanagement. Building projects running over time and budget is not mismanagement. It's misrepresentation, which all parties involved, managers and purchasers, are incentivized to support. Especially when it's a government doing the purchasing, since infrastructure is far more valuable than the sticker price but the true value would be politically unpalatable.

I expect similar things are going on with the other problems. The management is fine; the incentives are wrong.

[Christopher]

A few observations:

I've worked in software development for 20 years and experience tells me "on time, on budget, to specifications; choose any two".

Now on a management level we should follow the money (incentives) and understand that in IT in particular, those that can't, manage. These managers are oblivious to their shortcomings (Dunning-Kruger effect), and the heirarchical and hereditary nature of management ensures failure is baked in. Failure is only a problem if lessons are not learned - but incentives rarely reward the necessary reflection.

[Dave Timoney]

I'm not sure incompetence is the right word here. That would suggest an innate inability that might only be obscured by luck. Most trains aren't late, showing that a competent service is possible, whereas you're not going to make it as a top-flight footballer no matter how many you fluke in off your shin.

I think what you're really addressing is efficiency. Given the universe's inexorable entropy, efficiency is simply more exceptional than we are prepared to admit. Where standard economics goes wrong is assuming that a system can tend towards efficiency because of inherent incentives. In reality, entropy (decay, dissolution, the path of least resistance) is a stronger force.

I suppose one way of thinking about the vogue for stagnation in economic debate is an overdue recognition of entropy.

[reason]

Trains don't run on time and complex projects run over time because there is a ratchet effect, that says if one thing goes wrong in a complex process, there is no chance that things going better than expected elsewhere can make up for it. You don't need incompetence, just limited information, uncertainty and variability (note uncertainty is ambiguous here so I included variability to cover both meanings of uncertainty).

[Deviation From The Mean]

"and you all have examples of bad management in your own workplace"

I am at the point where I think bad management is the cultural norm. Managers have an obsession with projects or tasks being completed quickly but do not seem to have the same obsession with projects delivering exactly what they need to. Managers seem to want their workers constantly busy, never coming up for breath.

This results in workers not having time to sit back, asses what they do and things of ways to improve themselves and their procedures.

The problem is that managers view workers with a combination of suspicion and a sense that workers are mere machines who must perform tasks. They have no concept of workers as human beings who can actually think.

I think failure is a fundamental part of any process, in that you can choose a perfectly sensible and rational option but it ends up not turning out as expected. From these 'failures' great lessons are learned. The trick I guess is to know when something is a dead end or is an opportunity to open up something new.

Capitalism has trouble with this inherent failure because it categorizes and must categorize, for ideological reasons, people as winners and losers because of its inherent inequality. So under capitalism failure can make the difference between starvation and luxury. Because of this dead ends can persist, Zombies can flourish and progress is a uphill and rocky path. And everyone is where they deserve to be.

[Deviation From The Mean]

Entropy is completely beside the point here because entropy concerns the physical but incompetence is a conceptual construct. So entropy has not stopped humans developing ever more productive methods, and capitalism clearly applied science to all aspects of human life more than any previous system. Entropy did not factor into any of this.

I think entropy has a place in economics on some level, which I seem to remember Paul Cockshott developing an equilibrium theory along these lines.

[Blissex]

Yes, as some other commenters say some of the examples above are just obvious deliberate "optimism" as to what can be achieved.

Also, today I found a nice quote in "The end of parliamentary socialism" about competence and J Corbyn, about the 1983 Labour electoral campaign:

«It was related to the general sense, felt very strongly by the Labour new left activists themselves, that the election campaign had shown that there was, as newly elected Labour new left MP, Jeremy Corbyn, put it, "great incompetence in the party machine".»

That incompetence, which persists today, is probably due to:

* In areas of traditional Labour vote the party machine atrophied (and often became corrupt).

* At the national level the leaders campaigned more through the Murdoch papers read by the "conservatory-building classes" than through the party machine.

Competence and even more so effectiveness must be cultivated, and a bit of zest helps too.

[Dave Timoney]

@DFTM,

Entropy doesn't stop things, it slows them down. Most productivity increases follow a common profile in which a rapid improvement is followed by a slow but steady decline. This is why productivity is best approached as a continuous series of refinements rather than periodic leaps. Capitalism encodes this through commoditisation and inbuilt obsolescence.

[Avraam Jack Dectis]

.
Incompetence?

Or the inevitable result of imperfection?

The first you can pillory.

The second you can expect.

The first you can sanction.

The second you should only work to minimize.

The viewpoint may say as much about the viewer as the view
.

[Fred List]

American economists are certainly incompetent. For some reason the fantastic successes of East Asia under mercantilism are ignored; free trade is continuously justified by our economists in spite of overwhelming empirical evidence from economic history. Economic history itself is largely not taught or mis-taught. The puzzle of how US economists can be so provincial, even though they seemingly do travel (any comparison of a US city and an East Asian city will reveal the former as a poverty-stricken, backward shell) and yet nothing about the East Asian experience has been integrated into core US economic models or discourse. How many more decades? Does all of East Asia need to look like Star Trek and all of the US like Burkina Faso before any US academic economist notices?

[Blissex]

«all of the US like Burkina Faso before any US academic economist notices?»

Suppose that the elites that "sponsor" the USA economists (or politicians) have applied the Bain Consulting Matrix to countries, or region of countries, and classified most western countries or regions of the USA and UK as "dogs" or "cash cows", and are asset stripping them accordingly, making a lot of money. Would the USA economists (or politicians) want to notice that their "sponsors" are doing that?
The principle seems to be "The farm is run for the benefit of the farmer, not that of the cattle".

And from the point of view of the elites, what's wrong with Burkina-Faso or Mexico? It is a source of cheap servants.

Maybve the USA elites have a brazilian model in mind, and the UK elites have Dubai as a model.

Some people want to go back the 50s: the 1850s or the 1750s.

[Deviation From The Mean]

"Entropy doesn't stop things, it slows them down"

But productivity slows down, then increases, then slows down again, then increases, then increases again, then increases again, then slows down, slows down again, then increases etc etc.

In fact looking at the US productivity chart over the last 60 years the graph shows a steadily increasing line!

"This is why productivity is best approached as a continuous series of refinements rather than periodic leaps. "

WFT? History tells us the opposite. have you heard of the industrial revolution?

[Graeme]

Ah, the curious meme of the "plundering of the BHS pension fund" emerges...

The pension scheme deficit as follows at the following dates (August year end), and less than this if reported net of deferred tax:

2014: -£139m (-£111m net of deferred tax)

2013: -£137m

2012: -£95m

2011: -£105m

2010: -£162m

Information taken from Companies Hosue (note 22 of the August 2014 accounts):

https://beta.companieshouse.gov.uk/company/00229606/filing-history?page=1

As we know, the pension fund was just in surplus in 2008 before the crash.

The looting has surely been down by the BoE's interest rate policy rather than any plundering of the fund.

[David]

Graeme.... more Basel 2, Solvency 2 and FRS17 I think..... forcing pension funds to link assets to liabilities.... so forcing them out of equities........ in which, with quantitative "easing" flowing in billions they would have performed considerably better.

[Graeme]

David, I think you are agreeing with me :)

[Cliff Tolputt]

The economics of football are a source of wonder. The top UK clubs have just received yet another financial boost from their share of the TV derived income. What are they doing with it - record transfer fees, eye-wateringly large salaries for their players, and the agents incentivised to create mischief. Incompetence is the kindest of words from this perspective.

[gastro george]

I wonder if this piece would have been written in, say, Germany or Switzerland. If not, then doesn't it say more about institutional failure in the UK than anything else?

[Kallan Greybe]

I'm not sure you're giving nationalisation a fair shake in this. While I think it's better that certain public goods are managed by the state, this isn't because states are in principle better managers, it's because states are capable of changing direction due to political argument. In principle this means that citizens will often have more control over how resources get used. My go-to example here is to contrast the Stafford Hospital Scandal with our ongoing problems with British energy companies.

In the case of the NHS, a purely political campaign was more than capable of getting all sorts of changes to be implemented. By contrast, despite years of complaints due to their operating a cartel so obvious even the Daily Mail noticed, the energy companies were able to completely ignore calls to change how they were pricing energy for years. When Ofgem eventually went in and investigated, all they could say was "the market is working as expected", and my worry is that they are probably right, these sorts of inefficiencies and injustices are what we can expect from at least certain markets working as expected. A market system in energy therefore cannot deliver anything like the kind of energy infrastructure we need. What we need is for decisions in this space to be shaped by political demands like the demand that we don't allow old people in a supposedly developed nation to die of exposure over the winter, or that we recognise that it is insane for more than half of the average Briton's income to be paid out to rentier landlords and energy cartels.