Stumbling and Mumbling

How bad are average incomes?

The Centre for Policy Studies claims (pdf) - to the glee of the Daily Heil - that disposable incomes for the average household have fallen since 2002. I've got three gripes with this.
1. The average and the median are very different. The CPS claims that the average household pays £7800 a year more in taxes than in 1997. But this is largely because a few super-rich have seen their incomes - and therefore their taxes - soar. For the more typical household - those in the fifth decile - direct taxes (which include council tax rose) by an average of £1441 between 1997-98 and 2005-06. Indirect taxes rose a further £753. And these increases are largely due to a 40% rise in pre-tax income. (table 6 of this pdf and table 21 of this pdf.)
2. There's a curious treatment of housing costs. The CPS's claim that the average household has got poorer  since 2002 rests upon the belief that mortgage costs have more than doubled. Now, ordinary mortgage rates have certainly not doubled since then. So how have payments done so? It's because house prices have soared, raising the average mortgage debt. But this is only an outgoing for that small fraction of people who bought recently. Someone who bought before 2002 has seen a much smaller rise in mortgage costs.
Regarding rises in house prices as a pure cost - as the CPS does - is about as daft as regarding them as a pure benefit.
3. It treats the public like fools. The CPS fails to ask the obvious question: why, if average people have gotten poorer since 2002, have retail sales boomed since then? Instead, it claims that "excessive debt" and lower disposable incomes mean "British households are more vulnerable to, and less prepared for, any downturn."
No doubt, this is true of some households - but anything is true of some people. What it fails to consider  is the possibility that households took on extra debt for a good reason - that they reasonably anticipate rising incomes, just as Milton Friedman's permanent income hypothesis says.
And herein lies a puzzle. The CPS describes itself thus:

The CPS was founded...to champion economic liberalism in Britain and has since played a global role in the dissemination of free market economics. Its policy proposals are based on a set of core principles, including individual choice and responsibility.

In presuming that people were wrong to spend and borrow, how are economic liberalism and individual choice being championed?

The bunching of freak events

Yesterday, I said that the trouble in the banking system was due in part to the inability to price credit derivatives accurately. This shows what I mean:

German police were investigating two cases of suspected infanticide yesterday that had resulted in the deaths of eight young children.
In the village of Darry, in the north of the country, five boys aged three to nine were found dead after their mother confessed to killing them earlier this week. Meanwhile, in the east of the country, the corpses of three other infants were found wrapped in plastic.

This shows that rare events - infanticide - can bunch together.
Similarly, defaults on debt are rare but also bunch together. The key question in pricing bundles of such debts, and their derivatives, is: to what extent do these events bunch?  How great is default correlation? Worse still, to what extent does this correlation change over time (in good times, it's low, in bad times high)? It's this question that underpins the fancy maths about pricing copulas. And the problem is that because rare events are, by definition, unusual, there's not much hard data to help. Hence the impossibility of accurate pricing.

Fat lesbians and Bayes theorem

One of the most moronic cliches of management speak - and there's fierce competition - is that we should think outside of the box.
Nonsense. We must always think inside the box. I was reminded of this by this story from Tim, that lesbians are more likely to be obese than straight women. A commenter added that  "fat women are more likely to be lesbians."
This is very wrong. And our Bayesian boxes show us why.
Sadly, the paper Tim cites isn't webbed, but this paper (pdf) by Christopher Carpenter is. It estimates that lesbians are half  as likely again to be obese as straight women.  We can use these data to fill in our boxes, thusly.
It's clear from this that although lesbians are more likely to be obese than straight women, the vast majority of obese women - 98% - are straight.
This is simply because the proportion of lesbians in the population is small.
Indeed, even if every single lesbian were obese, it'd still be the case that if you were to see an obese woman, there'd be a greater than 90% chance that she was straight.
Not that this fact would be any use to you.

Seeing Bayes' Theorem

I'm a journalist, so I'm contractually obliged to be innumerate. I can't therefore understand Bayes' theorem - I get me Pr(A¦B) and Pr (B¦A) mixed up. Sums is hard. So I use pictures instead.
To see how, take this from Glen Whitman.

Suppose that stupid people constitute 60% of the population (a generously low number). Suppose that smart people split 50-50 between conservatism and liberalism. And suppose that stupid people split 80-20 between conservatism and liberalism, consistent with Mill’s claim. Now, say you meet a random person who turns out to be conservative, and you know nothing else about him. How likely is he to be stupid?

Being a smart fella, he uses Bayes' theorem. Being stupid, I use boxes, like below. First, I fill in the bottom row;  out of 100 folk, 60 are stupid and 40 smart. As the 40 smarties split equally between Conservatives and Liberals, I can put 20 into each box in the smart column. This leaves only the "stupid" column to fill in. As the 60 stupids split 80-20 for the Conservatives, I figure this means they split 48-12 - hey, I can do percentages. 
Having filled in all the boxes, I can answer Glen's question. There are 68 Conservatives, 48 of whom are stupid. So the chances of a random Conservative being stupid are 48 out of 68. Which is just under 71%.
Exactly what Glen says. Without any sums harder than simple percentages.
This Bayes malarkey ain't so hard, is it?

Stupid Conservatives and Mill

Joe quotes John Stuart Mill's letter to Sir John Pakington:

I never meant to say that the Conservatives are generally stupid. I meant to say that stupid people are generally Conservative.

Joe says this is not a claim that most Conservatives are in fact stupid.
In logic, Joe's right. But in practice, he's not.
It all depends upon the proportions of Conservatives and stupid people in the population, and what Mill meant by "generally".
In the general election the year before Mill wrote those words, voters split 40-60 Conservative-Whig. Let's assume this represents the proportion of Conservatives in the population.
So, what did Mill think was the proportion of stupid people? Here's a clue from On Liberty:

On any matter not self-evident, there are ninety-nine persons totally incapable of judging of it, for one who is capable; and the capacity of the hundredth person is only comparative.

Let's put the weakest possible construal upon "generally" - just over 50%. It then follows that  over 50% per cent of the population are stupid Conservatives. That's more than all the Conservatives in the 1865 election result; maybe some were too stupid to vote.
On this weak reading of "generally",  the majority of Conservative voters in the 1865 election are stupid as long as more than 40% of all people are stupid.
On a a stronger reading of "generally" - three-quarters - then the proportion of stupids in the population has to be below 27% for the majority of Conservative voters not to be stupid. Given Mill's view of human intelligence, he probably didn't believe this.
One wrinkle here is that women and the working class could not vote in 1865. However,  large numbers of these would have to be Conservatives to overturn my maths.
So, I reckon Mill did mean to say that Conservatives were generally stupid, and he was fobbing Sir John off, in the belief that, as a Conservative, he'd be too stupid to do the maths.

News vs truth

The massive coverage of the Cumbrian rail crash shows that news and the truth are two different things.
Rail crashes make the news because they are rare. Most car crashes don't make the news because they are common.
But the truth is that car transport is far more dangerous. In Q3, 840 people died on our roads and 7630 were seriously injured (pdf). This means Cumbria's casualties  - one dead, eight seriously hurt - are equivalent to around three hours of road accidents.
Put this another way. In 2005 there were 6.35 deaths and 57.5 serious injuries per billion vehicle kilometres on the roads (pdf). On the railways, there were 0.1 deaths and 11.6 injuries per billion passenger kilometres. Even in the worst recent year on the railways (1999-00) there were just 0.9 deaths and 18.6 injuries per billion miles.
So, however you cut it, road transport is much more dangerous.
The media's disproportionate focus upon newsworthy railway accidents rather than road accidents invites the public to commit the salience heuristic - of judging the probability of train crashes to be more likely than they are, simply because they get so much attention.
This error might have dangerous consequences. If someone chooses to travel by car rather than rail as a result, they expose themselves to greater risk.
And if policy-makers insist upon tougher safety standards on the railway - which cost money and so drive up fares - they divert people onto roads, which are more dangerous.

Lone parents & gun crime

Ross Clark says something that is easy to misinterpret. It's:

Those involved with gun crime tend to have grown up fatherless and in the absence of good male role models have gravitated towards gangs.

It would, however, be wrong to infer from this that growing up fatherless is a cause of gun crime.
Let's put some numbers on this. The Home Office reckon (ch 2 of this pdf) that firearms were used in 11084 crimes in 2005-06. Let's assume this means there are 11084 gun criminals, and let's assume all these were 16-24 year-old men.
Now, in 2005 there were roughly 3.5 million such men. How many come from single-mother households? According to table 2.6 of this big pdf, the proportion of children in single-mother households has risen from 6% in 1972 to 16% in 1992 to 21% in 2005. Taking the 1992 figure, this means there were roughly 560,000 16-24 year-old men who grew up in lone-mother households.
Even if we assume that all gun criminals come from such households, it's still the case that only 2% - 11084 out of 565,000 - of men from single-mother families become gun criminals. To get the figure this high, I've had to make some heroic assumptions.
So, at least 98% of men from single-mother households don't become gun criminals. It would be odd to say that x causes y when over 98% of xs don't cause y.
Indeed, there's a good theoretical reason why single mother households, on their own, shouldn't produce criminals. It's that the alternative to a single-mother family is not, in many cases, a family where the father is a good role model. It's one where the father is a bad role model. Removing a bad role model might be good for a child's development. Had I grown up in a two-parent household and used my dad as a role model, I'd have become a criminal.

Fact!

Things I learned today:
Saggitarians are more tolerant of homosexuals than other star signs.
Motor sport fans are more likely to get into fights.
People who love maths at school really are less likely to have sex. People who like social sciences are more likely to have lots of partners.
Opera fans are less likely to get shot.
People who play musical instruments are more likely to go hunting and fishing.
All from the 1993 US General Social Survey, via Marginal Revolution. Go and waste an afternoon.

Terrorism and contagion

Daniel Finkelstein seems to think Islamic terrorism is non-ergodic. He reckons John Mueller (old pdf) understates the probability of an attack:

His argument, the “bloody nuisance” argument, depends on the idea that, without additional measures domestically and internationally, the number of terrorist incidents is unlikely to rise greatly.
There is, however, lots of evidence that crime doesn’t work like that. Instead of falling gently or rising gently in response to policy measures, crime behaves like a contagious disease. Potential offenders catch the idea of offending from each other. And just like a disease that starts with only a few people and becomes an epidemic, once it reaches a tipping point the amount of criminal behaviour explodes. 
If successful suicide bombings became even slightly more common, can we really be confident that other fundamentalists would not copy that behaviour?

I'm not sure this analogy works. As my old economics tutor used to ask: what's the mechanism? There are three mechanisms which cause ordinary crime to be contagious which don't apply to suicide bombing:
1. Informational cascades. People copy others if they believe others know something they don't - it's a process of social proof. In the case of ordinary crime, I can see how this works. People see muggers making money without getting caught  and infer that mugging is worth doing. And the more such muggers they see, the stronger they'll infer this, and so the more likely it is that they'll become muggers themselves. Being a successful mugger, then, sends a signal to others that mugging is worth doing. Do suicide bombers really send comparable signals?
2.  Social norms. If enough people commit crime, their friends and neighbours might do so because the social norm against crime - the censure we'd get from others - breaks down. But although there are many communities of petty criminals, I'm not sure there are communities of suicide bombers, who weaken  the social norm against blowing people up.
3. Power laws. Crime soars not merely because more people become criminals but because a few people commit lots of crime. Criminal activity is distributed as a power law (pdf). Here. the analogy with suicide bombing breaks down completely. The good thing about suicide bombers is that none of them is a recidivist.

Sexist iPods and social sciences

My iPod is sexist. I put it onto shuffle the other day, and of the first 10 songs, three were by Johnny Cash, but not one by one of the many women singer-songwriters I like. The chances of this happening by accident are amazingly remote.

Of the 4513 songs on my iPod, 133 are by the Man in Black.  So the chances of three being in the first 10 are just 1.8 per cent.

What’s more, 361 songs are by five women singers: Gillian, Emmylou, Dar and the Kates Campbell and Rusby. Not one was in the first 10. There’s only a one-in-five chance of this happening.

The chance of the conjunction of these two events is 0.2 x 0.018 = 0.36%, or 1 in 278.

Such a chance is way beyond any standard of statistical significance.

So, my iPod is sexist.

What’s wrong with this reasoning? Three things.

1. I’ve mis-specified the data. Had I put the question: what are the chances of three songs in the first 10 by a male country singer, the odds would be much higher than they are for Johnny Cash alone. – sufficiently high to be quite normal.

2. I’ve omitted any mechanism whereby the iPod might be sexist.

3. I’ve ignored a basic fact – that random events produce odd patterns. Other shuffles would produce other strange patterns.

And here’s the point. This is not a post about my iPod. It’s about the social sciences.

To fix ideas, take this great paper (pdf) on sexism in the labour market. It claims that women earn 87p per hour less than men simply by being women.

Does this prove that women are discriminated against by 87p per hour? No. All three of my iPod errors come into play.

1. Mis-specification. Feminists can argue that 87p per hour under-states discrimination, because some of the other explanations for wage differences are in fact indirect forms of discrimination; for example, women are less likely to work n large firms where pay is higher. Anti-feminists can argue that it over-states discrimination, because it ignores women’s greater preference for fulfilling rather than remunerative work.

2. The paper tells us nothing about mechanisms whereby women are discriminated against. There’s no evidence that an employer, faced with two otherwise equal workers, pays the man more. And no evidence that he either can do so or wants to. But outcomes alone are often uninformative. It’s processes and mechanisms that matter.

3. There is a teeny chance that even such a huge gap could have arisen through chance. In this case, it’s only around 0.01%. But as we’ve seen, rare chances can crop up in single samples of data. What is really powerful evidence for sexism is that women are paid less than men in different societies; Germany, Italy, the US , and so on. The chances of such a conjuncture of gender differences happening by chance are truly negligible.

My message here is simply that purely statistical evidence in the social sciences is rarely clear-cut.

Put it this way. Has there been a single controversial issue in any of the social sciences which has been settled by the statistical evidence alone? I’m not sure there has.