Why do ugly men chat up good-looking women when better looking ones don't? Why do idiots sometimes get good jobs whilst smarter people don't? Why don't the long-term unemployed look harder for jobs? Why are women under-represented in "top jobs"?
These questions might have a common solution. It's all to do with self-confidence in the search process, as a clever experiment described in this paper shows.
The authors set subjects a maths test and told the subjects the result of their own test, but not others, so subjects were uncertain of their relative ability. The subjects were then given the option of playing (or not) a sequence of lotteries where the probabilities of winning were high for people of above-average ability and low for those of below-average ability. The pay-offs were such that risk-neutral people with above-average ability would always play the lottery, whilst below average ones wouldn't.
This experiment revealed several curious things:
1. On average, people's self-confidence is biased. High ability people had less confidence than they should, whilst low ability ones had more.
2. People update their self-confidence too slowly. In theory, someone who lost lotteries should have revised down their estimation of their ability, and those who won should have revised up their estimates. They did so, but much more slowly than Bayes' rule warranted. So high-ability people searched less, and low ability ones more, than they should.
3. Women were less self-confident than men.
4. People who believe they have low ability don't want to find out the truth about their ability.
These findings have obvious real world explanations. They explain why ugly men chat up (search for) beautiful women - they're over-confident - and continue to do so even after persistent knock-backs; they don't update beliefs properly. They explain why high-ability people - especially women - sometimes don't get the jobs they deserve; they just don't have the confidence to search. And they explain why the unemployed stop looking for work - they don't want to be knocked back.
I suspect we've known all this all along. But we now have objective empirical evidence to corroborate our view that search markets - the labour market or the dating market (analytically these are very similar) - can be systematically inefficient, in that bad matches can occur and good matches don't.
The question is: how great are the costs of this?