Measurements can mislead us, especially when they are used as management targets. That’s the thesis of Jerry Muller’s The Tyranny of Metrics. He writes:
What can be measured is not always what is worth measuring; what gets measured may have no relationship to what we really want to know. The costs of measuring may be greater than the benefits. The things that get measured may draw effort away from the things we really care about. And measurement may provide us with distorted knowledge – knowledge that seems solid but is actually deceptive.
Muller provides lots of examples of this, mostly from the US. But you’ll all have examples of your own. In universities the Research Assessment Exercise (now the REF) contributed to increased administration costs and perhaps to the replicability crisis by incentivizing the publication of mediocre research. In schools, targets can encourage teaching to the test, endless revision and a focus upon the marginal student to the neglect of both the strongest and weakest. Waiting-time targets might distort clinical priorities. Immigration targets deter foreign students and lead to the harassment of people who have lived here for decades. Sales targets encourage workers to mis-sell financial products, cook the books, or increase risk by encouraging (pdf) “liars’ loans. And so on.
All this will be familiar, especially to those of you acquainted with the work of John Seddon – although Muller puts it all concisely and well: Campbell’s law, which says that measures will be misused, was proposed way back in 1979. And it’s going to remain a big issue thanks to the rise of big data and machine learning – which, I suspect, will be the over-hyped discovery of weak correlations that don’t hold out-of-sample.
The Tyranny of Metrics is not, however, a diatribe against targets. Muller points to the experience of some US hospitals to show that metrics can work. They do so, he says, when they are “based on collaboration and peer review”:
Measurements are more likely to be meaningful when they are developed from the bottom up, with input from teachers, nurses and the cop on the beat.
In other words, metrics can succeed when they are complements to knowledge: when they organize the tacit and dispersed professional judgements of people who know ground truth. Tim Harford makes a good point when he criticizes Muller for overlooking Paul Meehl’s finding (pdf) that simple statistics often out-perform professional judgment, but I don’t think Meehl was arguing that statistical models should be constructed and used in complete ignorance of professionals’ know-how. The latter informs the former, whilst the former in turn checks the latter.
Indeed, Muller’s evidence shows that metrics can fail when they are substitutes for knowledge – when top-down bosses try to supplant professional know-how.
This raises an under-appreciated point - that questions of what statistics tell us and whether target work are not merely technocratic ones. They are deeply political, because claims to power and wealth are founded upon claims of expertise. As Alasdair MacIntyre wrote:
Civil servants and managers alike justify themselves and their claims to authority, power and money by invoking their own competence as scientific managers…But is this true? Do we now possess that set of lawlike generalizations governing social behaviour of the possession of which Diderot and Condorcet dreamed? Are our bureaucratic rulers thereby justified or not? (After Virtue, p86-7).
The belief that people and organizations can be managed and controlled by simple measures imposed from above is one of the main foundations of bosses’ claim to power. In showing us that this belief is at least sometimes wrong, Muller has written a much more political book than many readers might think. And in doing so, he has also helped to advance the case for greater worker democracy.
«the over-hyped discovery of weak correlations that don’t hold out-of-sample.»
But that's a description of much empirical research in Economics!
Isn't that anti-science? :-)
Posted by: Blissex | April 11, 2018 at 04:45 PM
The phrase missing from this review is "juking the stats" (see The Wire). Most stats can be juked. I don't say all stats are useless, but you need some healthy skepticism.
Tomorrow thousands of people will go to work and spend a large part of the day juking some stats. Mostly, if they didn't do it we would all be worse off.
Posted by: Bill Posters | April 11, 2018 at 06:52 PM
'The belief that people and organizations can be managed and controlled by simple measures imposed from above is one of the main foundations of bosses claim to power. In showing us that this belief is at least sometimes wrong, Muller has written a much more political book than many readers might think.'
I should read the book. Your idea is very good. It did puzzle me as a youngster, that this aspect of After Virtue: its engagement with philosophy of science (social science) was of great importance, but was only being read in the history department, as far as I could see (The Enlightenment Project and all that).
I wondered how you you could develop this at the time. What research programme could follow from his work? Your idea from Muller seems one good answer in this regard. Back then one sociologist that listened to my 'summary' of this element of After Virtue said, 'surely Max Weber had already said all this'.
Mmmmm
A bit like certain well known economists today saying that MMT is 'just' dressed up Keynesian 'fiscalism', which we have been discussing for years old chap.
Posted by: Mike W | April 12, 2018 at 07:45 AM
this is the Lies of Accountancy. You can't measure the things that are important, so you measure the things you can measure and pretend they are important.
Posted by: Dipper | April 12, 2018 at 03:04 PM
"juking the stats" Yes, but what to make of an organisation that is so rubbish they can't even fiddle the numbers?
Bower birds. they collect all sorts of shiny things like bottle tops into piles. The female mate with the birds with the best piles. The actual collection of these piles has no biological purpose, but there needs to be a competition, so in the absence of anything more biologically meaningful collecting bottle tops it is.
http://www.viralforest.com/bower-bird/
Posted by: Dipper | April 12, 2018 at 03:08 PM
All targets sub optimise your system. Instead measure what matters to your customer.
Posted by: FDUK | April 12, 2018 at 05:52 PM
This is a wonderful post. My frustrations in dealing with the economics profession often dealt with their lack of interest in really understanding what the data meant, how it was defined and where it came from. There was a complete lack of interest in non-quantifiable information - including primary evidence from archival material or people working out on the field. They look for a Phillips curve and if they see it - they go back to their models.
NK.
Posted by: Nanikore | April 13, 2018 at 06:43 AM
There's an old adage, 'knows everything, understands nothing'. Metrics provide useful information to complement and supplement understanding, but they're not substitute for understanding. This is where they get misused in managerialism. I've worked in BI/MIS for a long time, I've seen how carefully chosen metrics meaningful to the user bring real benefit, I've also seen how they are a crutch for charlatans.
I used to work for a major corporate where the 'superstar' mediocrity in charge of unit was 'all about the numbers'. My team produced pages of KPIs for him and his arse lickers. But most had no tacit understanding of underlying processes and mechanisms, so everything had to have a target, otherwise it would be meaningless. And although targets were profiled for seasonality etc they couldn't reflect real world events, which meant constantly explaining divergence obvious to those who had basic undertanding of underlying mechanisms. But execs rotated regularly, and many didn't do 'details' even when 'attention to detail' was supposedly their schtick, it was all very surreal.
Posted by: MJW | April 13, 2018 at 09:30 AM