Too often the speculative motive has tended to overwhelm the risk motive, and active markets have been a source of instability rather than security. As they were in the last financial crisis.
I'm not sure about this.
Let's imagine there had been a big, active, liquid market in house price futures and options in the 00s, as Shiller would like. What would have happened?
We know that, in a market of noise traders and rational investors, prices can overshoot because the smart money doesn't always trade against the stupid money, and might even trade with it (pdf). To this extent, house price derivatives would have been bubble-prone.
But there's another type of trader to consider - those with hedging demand.
Some of these would have gone long of the derivatives. Young people wanting to buy a house in future, or home-owners wanting to trade up, would/should have bought derivatives as a hedge against prices rising out of their reach. But other hedgers would have gone short. These would not only be home-owners planning on trading down but also mortgage lenders seeking a hedge against the falls in house prices that would increase mortgage defaults.
So, how would our market have looked in the mid-00s?
One possibility is that the weight of shorting from hedgers would have been great, and this would have held derivatives prices down. Arbitrage would then have caused physical house prices to be lower. We'd have had less of a housing bubble and hence less of a burst.
Another possibility is that the weight of noise traders - Kay's speculative motive - would have been so great that there would still have been a bubble in both physical houses and derivatives. But so what? If the derivatives had been over-priced in 2006, those mortgage lenders who did short them would have had a huge payout. The financial crisis would thus have been alleviated, as this payout offset losses on mortgages.
These two possibilities suggest that Kay is wrong and Shiller right.
However, there are two other possibilities.
One is that the bubble in derviatives might have been accompanied by counterparty risk. Those who wrongly went long might not have been able to pay the shorters.
The other is that mortgage lenders might not have used derivatives to hedge their exposure - either because they feared counterparty risk or were just over-optimistic.
In both ways, we might still have had a crisis.
I suspect there's no way of telling what would really have happened in this parallel universe. This, though, doesn't mean this thought experiment is useless.Had we had a bubble and crisis with such derivatives, it would not have been because of a market failure (a bubble) but rather because of an organizational failure - the failure of lenders to use the market to hedge or the failure of the long side of the market to manage risk.
In this sense, Shiller and Kay are both half-right. Shiller's right that more markets can mitigate market failure, if not organizational failure. But Kay's right to say that markets can't always protect us from human stupidity.