Sir Michael Scholar of the UK Statistics Authority accuses the Home Office of releasing “premature, irregular and selective” statistics on knife crime.
This actually understates the case. Even if the Home Office’s numbers were accurate, they would still verge on the meaningless. There are (at least) seven questions they raise - questions which should be asked about many statistical claims:
1. Is that a lot or a little? The Home Offices says that the 10 police areas participating in the Tackling Knives Action Programme (TKAP)saw a 27% fall in the number of teenagers admitted to hospital for assault with a sharp object in Q3 2008 compared to Q3 2007.
But if a number is small, small changes it will lead to big percentage changes - as when Robert Peston says HBOS‘s bad corporate debts have risen by 560%. The question is: are such changes really big or important?
2. What’s the counterfactual? What would have happened to knife crime without the TKAP? One obvious possible way of answering this is to look at what happened in the police areas not participating in TKAP.
3. Is there a selection bias here? The 10 police forces participating in TKAP were not chosen at random. Instead, they are those that had seen increases in knife crime. But what if there’s an element of mean reversion in the numbers of knife crimes? We’d then get this sequence: crime rises - policy begins - crime falls. But there needn’t be a causal link between the policy and the fall in crime. And if the policy only applies to areas that have seen the biggest mean-reverting rise, it’ll look as if the policy has caused bigger falls in crime than in areas without it. But this would be a mere illusion.
4. What’s the volatility of the figures? Any statistical series will contain some fluctuations that cannot be attributed to identifiable causes such as policy or material conditions. The question is: are the changes we’re seeing bigger than these random fluctuations? If not, they have little significance.
5. What’s the seasonality? The Home Office says there were “17% fewer serious knife crimes against young people…in October 2008 than in June 2008 in the ten TKAP areas.” But I’d expect this simply because the weather is worse and nights are darker in October than in June, so there are fewer people on the streets and therefore less chance of stabbings. Is the fall more or less than you’d expect from this?
6. What exactly do the figures measure? The figures refer to teenage victims. But you can have fewer teenage victims of crime simply if criminals decide to attack 20 year olds rather than 19 year olds.
And this could have happened. The scare about teenage victims of knife crime might have led to fewer teenagers going out. A criminal looking for someone to stab would therefore have been more likely to attack an older person simply because there were relative more on the streets.
7. What’s the cost of this policy? The Home Office says there were 10,000 additional stop and searches per month. If we assume these take 30 man-minutes (two officers for 15 minutes), this represents 5000 man-hours diverted from other uses. What were these uses? What crimes have risen as a result of police time being diverted away from their prevention and detection? And if that time wasn’t being used productively, why have police resources been so badly mismanaged?
The Home Office statement makes no effort to answer any of these questions. This suggests either that whoever wrote it has no interest in statistical analysis, or that they thought they could get away with such sloppy twaddle because journalists wouldn’t ask these questions.
Whichever it is, it suggests a problem deeper than using merely premature numbers. It suggests that our political culture - politicians and journalists - has no interest in proper statistical analysis at all.
Which is odd, given that we are supposed to be living in a technocratic era without political ideology, where evidence-based policy-making rules and “what matters is what works.”
This actually understates the case. Even if the Home Office’s numbers were accurate, they would still verge on the meaningless. There are (at least) seven questions they raise - questions which should be asked about many statistical claims:
1. Is that a lot or a little? The Home Offices says that the 10 police areas participating in the Tackling Knives Action Programme (TKAP)saw a 27% fall in the number of teenagers admitted to hospital for assault with a sharp object in Q3 2008 compared to Q3 2007.
But if a number is small, small changes it will lead to big percentage changes - as when Robert Peston says HBOS‘s bad corporate debts have risen by 560%. The question is: are such changes really big or important?
2. What’s the counterfactual? What would have happened to knife crime without the TKAP? One obvious possible way of answering this is to look at what happened in the police areas not participating in TKAP.
3. Is there a selection bias here? The 10 police forces participating in TKAP were not chosen at random. Instead, they are those that had seen increases in knife crime. But what if there’s an element of mean reversion in the numbers of knife crimes? We’d then get this sequence: crime rises - policy begins - crime falls. But there needn’t be a causal link between the policy and the fall in crime. And if the policy only applies to areas that have seen the biggest mean-reverting rise, it’ll look as if the policy has caused bigger falls in crime than in areas without it. But this would be a mere illusion.
4. What’s the volatility of the figures? Any statistical series will contain some fluctuations that cannot be attributed to identifiable causes such as policy or material conditions. The question is: are the changes we’re seeing bigger than these random fluctuations? If not, they have little significance.
5. What’s the seasonality? The Home Office says there were “17% fewer serious knife crimes against young people…in October 2008 than in June 2008 in the ten TKAP areas.” But I’d expect this simply because the weather is worse and nights are darker in October than in June, so there are fewer people on the streets and therefore less chance of stabbings. Is the fall more or less than you’d expect from this?
6. What exactly do the figures measure? The figures refer to teenage victims. But you can have fewer teenage victims of crime simply if criminals decide to attack 20 year olds rather than 19 year olds.
And this could have happened. The scare about teenage victims of knife crime might have led to fewer teenagers going out. A criminal looking for someone to stab would therefore have been more likely to attack an older person simply because there were relative more on the streets.
7. What’s the cost of this policy? The Home Office says there were 10,000 additional stop and searches per month. If we assume these take 30 man-minutes (two officers for 15 minutes), this represents 5000 man-hours diverted from other uses. What were these uses? What crimes have risen as a result of police time being diverted away from their prevention and detection? And if that time wasn’t being used productively, why have police resources been so badly mismanaged?
The Home Office statement makes no effort to answer any of these questions. This suggests either that whoever wrote it has no interest in statistical analysis, or that they thought they could get away with such sloppy twaddle because journalists wouldn’t ask these questions.
Whichever it is, it suggests a problem deeper than using merely premature numbers. It suggests that our political culture - politicians and journalists - has no interest in proper statistical analysis at all.
Which is odd, given that we are supposed to be living in a technocratic era without political ideology, where evidence-based policy-making rules and “what matters is what works.”
Issue Kalashnikovs and we solve the problem.
Posted by: jameshigham | December 14, 2008 at 02:06 PM
You misunderstand 'what matters is what works' in the distorted realities of Superman and the Dishonorable M'Lord Mandelson. If they can get away with lies, 'it works' and that's what matters....
Posted by: Rob Atkins | December 14, 2008 at 02:47 PM
"Issue Kalashnikovs and we solve the problem"
Psst. Are the shares of many funeral directors and body armour suppliers listed?
Please let us all know before the Kalashnikovs get distributed around.
Posted by: Bob B | December 14, 2008 at 02:57 PM
"You misunderstand 'what matters is what works' in the distorted realities of Superman and the Dishonorable M'Lord Mandelson. If they can get away with lies, 'it works' and that's what matters.... "
Well, if only what works counts, that certainly rubbishes John Major, all right . .
Posted by: Bob B | December 14, 2008 at 03:25 PM
Bob P - incisive contribution (yawn). Unfortunately, (courtesy of Iain Dale) Superman himself recognises his own incompetence :
"A weak currency is the sign of a weak economy, which is the sign of a weak government" Mr G Brown (1995) - the last honest thing he ever said !
Posted by: Rob Atkins | December 14, 2008 at 03:32 PM
Has Lord Mortgagefraud changed his name to Mandelson? How odd.
Posted by: dearieme | December 14, 2008 at 04:04 PM
I am quite enjoying watching Niall Ferguson's documentary on money on C4, but it left a bit of a sour taste when he said something to the effect of 'Argentina's GDP for x years before the monetarists gave their advice was a measly 0.1%, and afterwards it was TWENTY TIMES THAT AMOUNT.' So, that would be 2% growth, then.
Posted by: Riz Din | December 14, 2008 at 05:07 PM
"'A weak currency is the sign of a weak economy, which is the sign of a weak government' Mr G Brown (1995) - the last honest thing he ever said !"
Ahem: I don't necessarily take GB seriously since I come from the "speak truth unto power" tradition and I'm long on criticisms here of New Labour governance for many reasons.
Besides, I've clear recollections of how Nigel Lawson, as Chancellor, screwed up monetary policy during the 1980s. And that's partly because I listened with rapt attention to a wonderfully lucid diagnosis of his muddle and confusion presented in an interview that Alan Walters, Mrs T's personal economic adviser, gave in an interview for the BBCR4 Today prog c. 1989. And I was among those economists who cheered when Ted Heath's Conservative government floated Sterling in June 1972.
OTOH as far as politicians are concerned, I've come to respect the diagnostics of our current economic predicament I hear in media interviews with Vince Cable but then he was previously Shell's chief economist and is intelligible as he isn't talking from sublime ignorance. Otherwise, if I want sound commentary on economic affairs, for the most part I follow Martin Wolf, Sam Brittan and other FT writers in the daily press, or what comes out of the Institute of Fiscal Studies and the NIESR.
The size of Britain's current account deficit on international payments suggests Sterling was significantly over-valued and the exchange rate was propped up by the inflow of funds attracted by the relatively high UK short-term interest rates and an economy that was fairly buoyant up to the 2nd quarter 2008. Once base interest rates were cut by the BoE - which the Conservatives have been pressing for - and the economy tanked in the 3rd quarter, holding Sterling lost many of its attractions so mobile funds switched to euro markets with higher interest rates and prospects of exchange rate appreciation - no surprise there.
http://www.economist.com/markets/indicators/displaystory.cfm?story_id=12777751
Posted by: Bob B | December 14, 2008 at 05:08 PM
"It suggests that our political culture - politicians and journalists - has no interest in proper statistical analysis at all."
Nail on head. It still amuses me to recall a conversation I had a few years ago with a senior policy officer in local government (probably 35-40k salary), where I had to repeatedly explain the concept of statistical significance. Indeed several research agencies no longer bother including details like confidence intervals in reports, because the clients don't understand what they mean.
Posted by: Planeshift | December 14, 2008 at 05:47 PM
This suggests either that whoever wrote it has no interest in statistical analysis, or that they thought they could get away with such sloppy twaddle because journalists wouldn’t ask these questions.
Welcome to the world of the Home Office.
Both are probably true. It frequently gets away with presenting "figures" and "studies" so transparently worthless that no-one would even try were they not both utterly innumerate and incurious as to whether evidence does in fact support the policy. The dozens of Home Office press officers are used to journalists printing whatever bald assertion is in the first paragraph, provided it is dramatic enough.
Posted by: Guy Herbert | December 14, 2008 at 06:33 PM
I've a sneaky impression this latest stats hash up to demonstrate recent anti-knife crime initiatives by the government had been a huge "success" was due to an OTT response by the Downing St rebuttal unit worried by the mounting series of press reports which inadvertently didn't accord with the official message that crime is under control and falling - for example:
"Violent crime by young people has risen by nearly 40% in the past three years. . . "
http://www.guardian.co.uk/uk/2008/may/16/ukcrime.gender
"The Home Office today provoked a major row when it blamed inaccurate police record-keeping for an apparent 22% rise in the number of the most serious violent crimes in England and Wales." [Guardian, 23 October 2008]
By those standards, issues of statistical significance are verging on . . err . . incidental . .
Posted by: Bob B | December 14, 2008 at 06:38 PM
With regard to seasonality, I know very little about the last hundred and fifty years, but I can assure you that in Sheffield in the 1840s, the average annual number of arrests for assault peaked at 415 in August, and sank to 240 in January. On the other hand, the variation between June (343) and October (357) was not so great - so this particular factor might not be relvevant, if our patterns continue. But patio heaters and taxis have probably had an effect on this century's figures.
If the Home Office wants to offer hospital admission data as some kind of gold standard for measuring violent crime (not a bad idea), then they are going to have to take on board some existing research which shows that CCTV has sod all effect on violent crime.
Posted by: Chris Williams | December 14, 2008 at 07:30 PM
Was there any noticeable seasonal variation in the Sheffield Outrages of the 1860s or did those occur as and when the inclination arose?
http://en.wikipedia.org/wiki/Sheffield_Outrages
Posted by: Bob B | December 14, 2008 at 08:14 PM
There was probably a correlation with the business cycle: depressions put the unions under pressure, and they needed to turn to intimidation to keep the trade 'on the box'. I think that Sidney Pollard nailed it more than 50 years ago in his article _The Moral Economy of the Sheffield Outrages_ .
Pollard nearly ran his car into me once. I was actually slightly disapointed that he'd missed: I was looking forward to being able to say to people 'See this limp? Pollard did that.'
Posted by: Chris Williams | December 14, 2008 at 10:32 PM
I've never read the late Sydney Pollard on the Sheffield Outrages - on which he was an acknowledged authority: perhaps I should to see how far historic traits might have influenced the recent history of industrial relations in South Yorkshire.
However, I'm aware that he regarded HM Treasury as the basic cause of almost all of what went wrong with the British economy since WW2, which seems to leave successive governments with little, if any, responsibility for policy and policy changes, such as the Medium Term Financial Strategy or the confusion and muddle of what followed it after autumn 1985 during Nigel Lawson's long stint as Chancellor.
Quite why anyone nowadays would want to uphold government economic policy of those times as something to emulate is a total mystery to me - and obviously overlooks that Britain was then a net exporter of oil and the government was raking in the proceeds of successive privatizations of the nationalised industries, from BT in 1984 onwards. I don't subscribe to Macmillan's regret that those privatizations amounted to selling the family silver but the present government no longer has similar options for raising capital.
I'm not sure there's much more to be said on the clear outrage by this government of issuing dodgy stats to prove it is successfully curbing knife-crime after the series of news reports about escalating violent yoof crime and the Police muddle with their stats which resulted in a huge extent of previously undisclosed crime.
Evidently, the rebuttal unit got over-excited.
Posted by: Bob B | December 15, 2008 at 12:02 AM
Oops - for 'Moral Economy' above, read 'Ethics'.
As it happens, I think that increases in the rate of reported crime since about 1960 are largely attributable to police reporting a larger (yet still, in the main, tiny) percentage of crime.
The 'police muddle with their stats' is the inevitable and unavoidable alternative to equally shrill accusations of 'Whitehall diktat and micromanagement'. You could have a nationally-uniform definition of which action counts as which crime, and rigorously enforce it, but boy, your cops are going to be filling in lots more forms, and your army of form-checkers is going to piss a lot of people off. Do we want that? Me, I don't want to finance it.
Police stats for reported crime must always be entirely discounted: they are highly vulnerable to changing definitions and political priorities. The Home Office know this (or ought to) but they are chasing brainstem headlines, as per.
The BCS _is_ a reliable way of measuring trends in victimisation, among the people that it surveys (private individuals aged over 16). A BCS for businesses and one for under 16s would be nice, but aside from that, it's just the same old same old.
Wake me up when the other lot get in and nothing changes, save only that the same lines are spoken by different people.
Posted by: Chris Williams | December 15, 2008 at 10:35 AM
"The BCS _is_ a reliable way of measuring trends in victimisation, among the people that it surveys (private individuals aged over 16)."
But my understanding is that the BCS doesn't include certain sorts of crime - such as rape:
http://www.homeoffice.gov.uk/rds/pdfs08/hosb0708.pdf
Besides all that, we need to refocus attention on conviction rates - which have been miserably low. This is the latest news report on conviction rates I could find by googling:
"The true picture of rising levels of violent crime in England and Wales and historically low conviction rates can be revealed today by The Observer.
"An investigation shows that conviction rates for many of the most violent crimes have been in freefall since Labour came to power in 1997 and are now well below 10 per cent. The chronically low figures for convictions come at the same time as reports that violent crime is increasing."
http://www.guardian.co.uk/uk/2006/may/28/ukcrime.immigrationpolicy
The bad news ahead is that robbery and other property related crime tend to rise when the economy is performing badly.
Posted by: Bob B | December 15, 2008 at 11:18 AM
The BCS is not a magic bullet, but it is the best we've got. Some crimes it's good at tracking, others it's less good. But it's better than plucking numbers or trends out of one's arse according to one's prejudices, which is the alternative adopted by many commentators in default of the correct one (which is to say "We don't know")
When police covered up most crimes, conviction rates were high. Now that they report more, conviction _rates_ have fallen. It's not rocket science. But has the number of crimes solved as a proportion of the number of crimes committed (the actual conviction rate, rather than the police-reported one) fallen? I doubt it.
Posted by: Chris Williams | December 15, 2008 at 11:30 AM
"But has the number of crimes solved as a proportion of the number of crimes committed (the actual conviction rate, rather than the police-reported one) fallen? I doubt it."
Why - in the light of this recent news report?
"Data supplied by the Home Secretary, Jacqui Smith, to her Opposition shadow, Dominic Grieve, indicated the detection rate for offences of violence against the person fell to 49 per cent in 2007/08. The Home Office said the figure was being pushed down because forces were no longer able to count as many cases where no further action was taken as 'solved' crimes."[18 November 2008]
http://www.independent.co.uk/news/uk/crime/under-half-of-violence-crimes-are-solved-figures-show-1023251.html
Posted by: Bob B | December 15, 2008 at 11:43 AM
The BCS is not a magic bullet, but it is the best we've got. Some crimes it's good at tracking, others it's less good. But it's better than plucking numbers or trends out of one's arse according to one's prejudices, which is the alternative adopted by many commentators in default of the correct one (which is to say "We don't know").
When police covered up most crimes, conviction rates were high. Now that they report much more, conviction _rates_ have fallen, whatever the moves in numbers of convictions[*]. It's not rocket science. But has the number of crimes solved as a proportion of the number of crimes committed (the actual conviction rate, rather than the police-reported one) fallen? I doubt it.
*Just looking these up now.
Posted by: Chris Williams | December 15, 2008 at 12:04 PM
SOrry about that double posting. In answer to the Grieve quote, what's the problem? What's going on there is that in the past, the cops used to add the 'no further action' column and the 'got the bastard' column together to make more than 50%. Now the HO won't let them do that, so the 'official rate' has come down - but the proportion of bad people being punished and those getting away with it might have stayed exactly the same.
Posted by: Chris Williams | December 15, 2008 at 12:16 PM
Hey, this is interesting. There was me, thinking that the numbers proceeded against for volume crime would have remained static. But they've gone down. Look:
burg's robbers theives fraudsters
2003 31300 7700 173600 23500
1993 53100 5800 238800 25600
1983 85000 4200 312000 27200
A graph from the 2003 return suggests that the absolute number proceeded against rose steadily from 1950, peaked in the early 1980s and has fallen ever so slightly since. Usual caveats apply about changing definitions of 'proceded against'.
For a longer view, note:
Burg+rob. Theft+forg
1953 5575 13432
1939 3473 12186
1914 1957 13669
1914 and 1939 figues are 5-year running averages, as reproduced in the 1953 return.
The Home Office makes it really easy to count reported crime, with their lovely C20th spreadsheet. But they make it very hard to count what's actually been done about it. Me and some associates were trying to sort out a project to solve that problem, but we didn't get the funding. Ah well.
Posted by: Chris Williams | December 15, 2008 at 12:39 PM
The scale of most crime in Britain pales into verging on trival stuff compared with the recent amounts lost through fraud and systemic failures in the financial system:
http://www.reuters.com/article/governmentFilingsNews/idUSLF6597520081215
But this shouldn't have come as a surprise. Warren Buffett was warning about the hazards of derivatives back in 2003:
http://news.bbc.co.uk/1/hi/business/2817995.stm
For years - years - I've been posting that Free Market Capitalism is a nonsense because what matters to enable the market system to function are protection of property rights by regulations and law enforcement agencies and that we really need to focus on the regulatory reforms needed to enable markets to function better. Naturally, no one took any notice and the near brain-dead went on chanting their usual silly mantra.
Few seem to have the gained essential insight that what can have untoward consequences if a few do it, can have catastrophic systemic consequences if many do the same at about the same time.
The glaring UK example of systemic failure was Northern Rock: I can draw down my account with a bank and there's no problem but if many depositers with that bank try to do the same thing at the same time then the bank collapses.
Besides that, there's nothing actually illegal about banks offering 125% mortgages to folks who can't afford to repay the mortgages. Sensible banks intent on protecting their shareholders' interests don't do that but - as we've seen - not all banks have sensible managers and directors and that can have dire systemic consequences.
Judging by his reported comments in the news today, these basic insights seem to have eluded David Cameron.
Posted by: Bob B | December 15, 2008 at 03:34 PM
By chance, on the BBC website I've just come across this mapping of where over 70 teenagers in Britain have been violently killed during this past year:
http://news.bbc.co.uk/1/hi/uk/7777963.stm
Posted by: Bob B | December 15, 2008 at 09:57 PM
"What matters is what works" is still true, but you must define what you want to work. For this ultra-cynical government, "what works" is what they think helps to get them re-elected. Period.
Posted by: Alfred T Mahan | December 16, 2008 at 10:41 AM