"We'll have to look at the data"

Here are some things I've seen this week:

 - Describing the decline of universities, Marina Warner says: "not everything that is valuable can be measured."

 - Some US newspapers have signed up to Blendle, allowing readers to pay to read individual articles.

 - Dave Tickner says that Peter Moores' response to England's being knocked out of the World Cup - "We'll have to look at the data" - "sums up everything's that's wrong" with his management.

 - South Yorkshire police diverted funds from sex abuse inquiries towards tackling car crime, which the Force was targeting.

 - Mike Goodman says that much of Mesut Ozil's contribution to Arsenal doesn't show up in statistics.

These apparently disparate things all bear upon the same question: can information be fully quantified, codified and therefore centralized, or were Hayek and Polanyi right to claim that some knowledge is inherently fragmentary, tacit and dispersed?

This is of course not merely a matter of epistemology. It bears directly upon how organizations should be structured. If everything can be measured by a central authority then hierarchy is feasible. If not, then we might need more decentralized forms of organization. It's no accident that the increased use of metrics in universities has coincided with the rise of vice-chancellors' salaries. Claims about knowledge are also claims to power.

As with most questions in the social sciences, the answer here is: to some extent. Given the ubiquity of cognitive biases, data can tell us what really works; this is the message of Moneyball. For example, stats disprove the claim that Ozil is lazy. And targets and quantification can be used to identify lazy academics and policemen.

However, good ideas can be pushed too far, with counterproductive consequences: revenge effects are common. There are at least four different mechanisms through which this can happen:

 - Gaming. David Boyle has claimed that school league tables led teacher to focusing excessively upon D-grade students at the expense of others, because converting D to C grades improved schools' performance. Similarly, South Yorkshire police were told to target car crime, they did just that to the detriment of tackling what we now know to have been more serious crime. This was an example of the drawback of centralized over dispersed knowledge; the belief that sex abuse was a serious problem was a hunch of a few junior officers, whereas knowledge of car crime was more quantified and centralized.

 - Excessive investment in measured outcomes. This happens when academics are encouraged to write grant applications rather than teach students. And it'll happen if journalists are paid per click: we all know that the way to attract eyeballs is to write about celebs or shrill partisan pieces.

 - Some stats are just bad and can give a mere illusion of knowledge. For example, in 2007-08 banks' risk models were based on data which over-sampled low volatility and under-sampled high. The upshot was that the crisis came as a shock. David Viniar, Goldman’s chief financial officer, famously said: "We were seeing things that were 25-standard deviation moves, several days in a row.” But in fact, a better inference would have been that risk was mismeasured.

- Adaptive markets. Companies or sports teams might get a competitive advantage from using statistical methods but they cannot retain it, simply because others will emulate them. This too is a message of Moneyball.

Herein lies my problem. I fear managers are underestimating these drawbacks. This might be simply because of deformation professionnelle - the tendency for one's professional background to warp one's perspective. Or it might instead be an example of Upton Sinclair's famous saying: "It is difficult to get a man to understand something, when his salary depends upon his not understanding it."

Comments

[Dan Goodman]

I think there are things that aren't measurable in a meaningful way, but that the more pertinent problem is that even when something is measurable it can take decades to centuries to find the right measure. This is, in some sense, the history of science. Our mistake is to think that the first naive formula we come up with will be good enough to base our actions on.

[Luke]

" - Gaming. David Boyle has claimed that school league tables led teacher to focusing excessively upon D-grade students at the expense of others, because converting D to C grades improved schools' performance. "

There's obvious problems with gaming, but in this particular case, might there be something to be said for concentrating on the D students? Their employability might go up disproportionately. And the benefits of moving from "poor" (D?) to "reasonable" (C?) literacy/numeracy might be greater than moving from "very poor" to "poor" or "reasonable" to "moderately good."

If the targets/measurements are well-designed (or lucky), gaming might not be too bad. Which doesn't really contradict you.

[Luis Enrique]

It doesn't have to lead to centralisation. In development "results based aid" is seen as a way of disrupting centralised control. Instead the idea is to specify what you want and leave people free to figure out how to achieve it without being managed from the centre

Of course concerns about gaming, measures ng the wrong thing etc. Still apply, so can't be used in every context

[Timlagor]

I would contend that pretty much everything is at least theoretically measurable but whether its feasible or worthwhile in practice is an entirely different question.
The other humongous obstacle is that having measured things makes it far easier for people who don't understand the data or its limitations to make and enforce bad decisions.

UK Conservative Education and Health Policy over the last few years are classic examples of people at the top deciding they know best empowered by 'data'.

Focusing on Grade D->C: I would contend* that the top end of the spectrum yields much bigger returns in economic terms at least**. I consider fairness to be a major consideration however.
* completely without the support of any data :)
** the level level of critical thinking where people actually understand what they are voting for on election day might just be a mid-point worth focussing on (though apparently that's actually a high bar)

[An Alien Visitor]

I agree that some things are harder to measure. For example, take the quick striker who appears useless but in fact stretches the opposition team by forcing them to defend deeper. This is hard, though not imposible to measure.

I think Hayek's argument was that central planning was a disaster because it couldn't react quickly enough to 'very local in real time events'.

The problem with this is neither can the market!

My belief is that more integrated communities would be more efficient, rather than the atomised society we live in. This way communities could share and exchange labour talent and make things more efficient. So for example a neighbourhood police service or more free exchange of abilities. My gas boiler is playing up a little, I will get Dave from number 27 to fix it, as I helped him with some pension advice the other week.

I think a more communal society would be more efficient than one that is atomised.

[Dave Timoney]

The South Yorks police focus on car crime was not just the result of political priorities, it also reflected the ease by which it could be measured (due to auto insurance) and an ideology that valued property more than people. We measure what we value, so all measurement is ideological, but we also measure things that are convenient to measure, and our assumptions about convenience are also ideological.

As Luke notes, David Boyle's criticism of "gaming" only makes sense if you believe that there is a greater social benefit in raising an A student to A* than in raising a D to a C. This is ideological both in what it values ("bright kids poorly-served by the state sector") and in its assumption that the metric is insufficiently discriminatory ("grade inflation").

Another example: "happiness economics" is clearly a case of changing the subject at a time when developed nations are facing stagnation (GDP growth is passé), but it also reflects an expectation, driven by preference marketing and big data (i.e. surveillance), that we can truly assess people's state of mind, rather than having to rely on a vox-pop.

On Ozil: passing up a shooting chance when clean through is not a stat that Opta or anyone else seem to measure, but it's something that the Arsenal crowd are acutely sensitive to (and not just in the German's case).

[gastro george]

It's not just what is measured but how you measure it. Aren't police statistics notoriously subject to reporting bias - which doesn't prevent the media nor the Home Office from making meaningless pronouncements. The National Crime Survey (or whatever the correct name is) was always a more reliable indication of changes in crime.

[FDUK]

Just a couple of things.

Managers commonly set targets based on what they can measure, but targets always sub optimise performance. The grade boundary between D and C is just a simple example. Schools should be getting the best out of all pupils, but that is not what the target drives. targets drive gaming not performance. Managers and politicians like targets because it give the illusion of control.

Also the Police didn't target car crime for political reasons as much as they went and asked the public. The public don't care about child abuse or murder. When you ask them the public care about parking and car crime, because that affects more individuals. I know I've asked. But its mad to let that run your policy.

[Icarus Green]

The fundamental problem with data sometimes is that it reveals the truth. And some people dont like the truth. And the truth is awkward and tawdry and dirty. And it compels us to act. And thats why the BBC has to write an article about the monster child sex rings in a passive tone and not mention the perpetrators and why the media doesnt want to talk about it. And why the police knew about it but didnt want to investigate.

[An Alien Visitor]

"The fundamental problem with data sometimes is that it reveals the truth. And some people dont like the truth."

While I like the idea of the establishment put on trial for child sex crimes I would argue that until the evidence is subjected to a court of law the truth has not been established in these cases. The problem is that people are jumping to conclusions and establishing the truth before it can really be established.

I notice how everything is given as a fact when the facts have not been tested or scrutinized to an appropriate level.

The truth that really would hurt is that people were making up the abuse claims. And nobody wants to raise that possibility.

So I think truth is already a casualty here.

[Graeme]

measurement is hard...and making real measurements and knowing what to measure is even harder. Consider driving to hit mpg targets versus speed limits, or soft measures such as driver satisfaction levels...