Diversity trumps ability

Jon Lansman wants Labour MPs to be "ordinary people who have held normal jobs" rather than career politicians. There's a powerful piece of thinking on his side - the diversity trumps ability theorem. This is an extension of James Surowiecki's wisdom of crowds theory, but it has been mathematically formalized by Lu Hong and Scott Page, who summarise it thus:

When selecting a problem-solving team from a diverse population of intelligent agents, a team of randomly selected agents outperforms a team comprised of the best-performing agents.

I'll spare you the maths, but give you the gist. Let's suppose that we want to find the best possible policy, according to some objective criteria - that is, we are in the domain of epistemic (pdf) democracy (pdf). Suppose too that there is bounded rationality and limited knowledge and that each individual selects the best option using his own information set and decision rule.

In these conditions, each individual, if s/he is moderately competent, will find a local maxmimum - the best option, given his/her information and decision rule.

But local maxima aren't necessarily global maxima. And Hong and Page show that even experts might well not find that global maximum because their decision rules and information sets might not be wide enough to encompass the best option: this might be because of deformation professionnelle, or groupthink or simply because their Bayesian priors limit the number of options they search for.

Instead, widening the population of searchers increases our chances of finding that global maximum, because doing so brings more decision rules and information sets to bear on the problem. Cognitive diversity - in the sense of different ways of thinking - can therefore beat experts. It increases our chances of finding the best option. This might be why diversity within companies is associated with more innovation.

Note that this requires effective deliberative democracy, so that lesser options can be discarded in favour of better ones; simple "speak your branes" direct democracy is not sufficient.

Jon's call for an end to Labour's meritocratic preference for career politicians should be seen as an endosement of the diversity trumps ability theorem.

Now, I'm not saying this theorem is universally valid. The virtue of fomalizing it as Hong and Page have done is that it allows us to see more clearly when it is and when it isn't, and John Weymark shows that there are conditions in which it doesn't hold. My hunch, though, is that it might be sufficiently applicable to the Labour party to endorse Jon's call.

Herein, though, lies what some might see as a paradox. The diversity trumps ability theorem is the strongest part of the case for free markets. It is by increasing the diversity of firms that we increase our chances of finding good new products and more efficient processes. The fact that so much (pdf) productivity growth comes from entry and exit rather than from the growth of existing firms can be seen as corroroboration of our theorem.

In this sense, the case for free markets and the case against decision-making by tiny homogenous elites have a common root. 

Comments

[Rafael Barbieri]

I'd agree if by a "free market" you mean one that can be accessed by a wide range of individuals/entities.

Since whatever society deems as money (in the case of the US money is predominantly electronic bank deposit balances)determines one's ability to access the "free" market, if financial resources are concentrated then market diversity declines.

In a sense does ensuring that all citizens have at least a basic degree of access to the markets enable at least a basic level of market diversity?

[hellblazer]

It seems some mathematicians are not convinced by some of the claims made for Hong and Page's work: see http://www.ams.org/notices/201409/rnoti-p1024.pdf

[Stevenclarkesblog.wordpress.com]

Caution is needed in what type of diversity the Labour Party seeks.

There is a huge drive to achieve gender equality in representation (no bad thing) and ethnic representation.

There is the risk that this just selects more career politicians of the right sex and skin tones, and pushes out those whose background and personality type would add real cognitive diversity.

[guthrie]

You might like to define what sort of merit the labour party actually selects for, and what being a career politician actually means.

Because right now it seems to me, and I think many others, that being a good politician in today's environment is orthogonal to actually doing anything good and useful for your constituents/ the wider public.

[Dave Timoney]

Lansman is arguing for strict representation (daily reporting, reselection, recall) and an ostentatious hairshirt (capped salaries, no pay for councillors). This shows a fear of sophistication and a contempt for the idea that politics requires particular skills.

It is ironic to see someone on the left arguing against decent salaries for politicians given the importance of this to the development of democracy in the UK. Poorly-paid representatives would be even more vulnerable to corruption.

Logically, sortition (like jury service) would provide the widest range of experience for our MPs, but selection by lot rather than election is unlikely to be considered democratic by most people, despite its ancient pedigree, and the gain in terms of breadth will be (by definition) marginal.

Lansman is echoing the received wisdom of our day, that "People hate politicians of all parties because they see them as self-serving careerists not ordinary people with principles they share".

But that image has been carefully cultivated by specific interests in society, both overt (the media denigration) and covert (the neoliberal insistence that everyone should be a "self-serving careerist"). Perhaps we hate MPs because they are more like us than we care to admit.

[Stevenclarkesblog.wordpress.com]

@FATE I wouldn't advocate it for the Commons, but I wouldn't mind an element of sortocracy in the Lords (20%, say, of the Lords, for a term of 1 Parliament).

[Dave Timoney]

All the House of Lords requires is abolition. No reform, no tinkering, no replacement.

[Igor Belanov]

I agree that Lansman's ideas are problematic and confuse representation of views with representation of social/occupational groups. I don't think I would suggest that the opposite position is 'meritocratic' though. MPs are educationally much more highly qualified than the general population, but the impression they give is that of pretending to be stupider than they are in an attempt to 'relate' to their constituents. Politicians from similar backgrounds in the past seem to have been more reluctant to hide their abilities. I think it would be better to ask what in society provokes MPs to act as they do, rather than suggesting a conspiracy.

[Deviation From The Mean]

“The case for free markets and the case against decision-making by tiny homogenous elites have a common root”

Except that one leads to the other!

The case is for actually abolishing the so called, never actually existing free market and replacing it with democratic enterprises and a democratic organisation.
The market is an out of touch other, something alien that says it delivers what we want but actually delivers something very different. If we want society to produce what we want we have to get rid of the so called, never actually existing free market (if that isn’t a paradox).

“It is ironic to see someone on the left arguing against decent salaries for politicians given the importance of this to the development of democracy in the UK”

Not in the context of lack of diversity, high representation among private/Oxbridge types and lack of working class representation!!!

What is ironic is the left defending such a state of affairs.

[Dave Timoney]

@DFTM, the diversity of MPs is a product of institutional factors, such as the selection process, de facto career paths (unions, SPADs etc), and networking (including nepotism). It isn't the result of pay rates.

The introduction of MPs pay in 1911 (by a Liberal government dependent on Labour support) was a direct result of the Osborne judgement of 1909 that blocked union subsidies to Labour MPs, and was thus an attack on working-class representation. Prior to this, MPs were essentially men of independent means or those who could combine the morning-averse House with a career in the City or law courts.

Paying MPs on an informal basis dates back to the Medieval era. It went out of fashion with the Glorious Revolution, which left many MPs in the 18th century dependent on rich sponsors (and not just those in rotten boroughs). It is interesting to note how the views of a contemporary, Samuel Pepys, compare and contrast with Lansman's critique:

"... the bane of the Parliament hath been the leaving off of the old custom of the places allowing wages to those that served them in Parliament, by which they chose men that understood their business and would attend it, and they could expect an account from, which now they cannot."

[Deviation From The Mean]

"It isn't the result of pay rates."

If MP's were paid, say, the average wage then I may be inclined to have a degree of sympathy with this view but they are not and therefore I don't.

Of course the other factors you mentioned are important and point to something, that the system itself is rotten and that pay rates won't really solve the problem. And the problem surely has to be addressed?

[Dave Timoney]

The idea that MPs should be on a median wage reflects a desire that they be representative of their constituents (i.e. having the same financial constraints) as much as it is an assessment of their social worth.

The problem with this is that a truly representative chamber would have MPs on widely differing pay rates, reflecting actual income inequality, possibly including disabled members on £2 an hour.

My point is that if we want MPs to be representative, their pay is probably one of the lesser issues to worry about.

[Stevenclarkesblog.wordpress.com]

Let's assume that the parties increase the diversity of their candidates - and there's a much bigger ideological spread (skeptics, libertarians and Marxists), and greater representation of jobs and class backgrounds.

You've only fixed one part of the system.

The Overton window which so limits political possibilities is not all down to the politicians.

You'll still face a press that is obsessed with immigrants/Europe/public sector cuts and takes no interest in automation, secular stagnation or any other big issues that we face.

You'll still face a public that gives little serious thought to politics.

You'll still face powerful vested interests who would threaten bad consequences if certain policies were pursued.

I would strongly suspect that politics would return to being as unsatisfactory as it is already, despite the parties' efforts, because it is emergent from the entire political-media system, rather than from the failings of politicians (real though they are.)

[Neil Harding]

This confirms my theory that electoral democracy has failed. Elections select from narrow elite groups and are easily bought by the rich and powerful. The jury service model is the best way to make informed decisions. Select 1000 people and put them in the 2 Westminster chambers. Guarantees a proper representative sample of public and avoids wasting money on party funding and elections. In fact lets randomly select our judges and civil service as aell. Come to think of it, we could do this for all the top jobs. At least everyone has a fair chance, unlike now.

[Adam Dutton]

Without empirical data I have no interest in this. Surely this is precisely the sort of work that can be done in a lab.