The illusion of skill - III

Pundits on TV are another who confidently predict future events that are essentially unpredictable. TV experts make it sound as if predicting the future is only slightly more messy than solving mathematical equations. And they rarely mention the word 'luck'.  As Kahneman says in his book, Thinking, Fast and Slow 'The idea that large historical events are determined by luck is profoundly shocking, although it is demonstrably true.' History is full of low predictability and large impact Black Swan events that are predictable only in retrospect. History looks more easy to explain when viewed backwards than it does when the events are actually happening, what Nassim Nicholas Taleb calls 'the narrative fallacy'. As Taleb says in The Black Swan:

Our inability to predict in environments subjected to the Black Swan, coupled with a general lack of the awareness of this state of affairs, means that certain professionals, while believing they are experts, are in fact not.  Based on their empirical record, they do not know more about their subject matter than the general population, but they are much better at narrating -  or, worse, at smoking you with complicated mathematical models. They are also more likely to wear a tie.

 Philip Tetlock, a psychologist at the University of California, Berkeley selected 284 people who made their living "commenting or offering advice on political and economic trends," including journalists, foreign policy specialists, economists and intelligence analysts, and began asking them to make predictions. Over a couple of decades, he asked them to rate the probability of outcomes of several questions: Would George Bush be re-elected? Would apartheid in South Africa end peacefully? Would Quebec secede from Canada? Would the dot-com bubble burst? Overall he had over 80,000 predictions.

How did the experts do? When it came to predicting the likelihood of an outcome, the vast majority performed worse than random chance i.e, they would have done better picking their answers blindly out of a hat. Liberals, moderates and conservatives were all equally ineffective. Most of the subjects had post-graduate degrees but they were mostly useless when it came to forecasting. Even in the region they had most knowledge of, the experts were not much better than non-specialists.

The main reason for the inaccuracy has to do with overconfidence. Because the experts were convinced that they were right, they tended to ignore all the evidence suggesting that they were wrong - they had an enhanced illusion of their skill. Those with the most knowledge were less reliable. This is because these experts were cocooned in their area of specialisation and tended to view the world through a narrow lens, what Nassim Nicholas Taleb calls 'the philistinism of the over-specialised scholar'. It is like the blind who touch different parts of an elephant and conclude that it is like a rope, a pillar, etc.

Tetlock also found that the experts were resistant to admitting error when it was pointed out to them, offering a number of excuses for their mistakes. The problem is that the over-specialised expert who can come up with the catchy one-liner is more likely to be invited to TV studios since  he is more interesting to listen to than the expert who uses a lot of 'ifs' and 'buts' even though the latter may be closer to the truth. The preferred expert will be the one who gives short, snappy answers to a screaming host who demands, 'India wants to know.' A safe rule of thumb to follow is to ignore the views of the experts who sound very confident about their forecasts. As Kahneman says in his book:

The first lesson is that errors of prediction are inevitable because the world is unpredictable.  The second is that high subjective confidence is not to be trusted as an indicator of accuracy (low confidence could be more informative).

P.S :  The Trouble With Experts

The illusion of skill - II

Although professional investors are able to beat amateurs, Kahneman says that few, if any, have the skill to beat the market consistently. The logic for testing this is simple: if the rankings of the funds in any one year is entirely due to luck, then the year-to year correlation of their rankings should be zero. But if skill is involved, then the rankings will be more stable and therefore there will be year-to-year correlation, showing persistent achievement. This is similar to the rankings in any game, say tennis. If the rankings are random, then there will be no year-to-year correlation. But the top players are consistently ranked at the top so there will be year-to-year correlation of their rankings.

More than 50 years of research has shown that professional investors are playing a game of dice, not of skill. The year-to-year correlations are barely higher than zero. The traders think that they are making sensible, educated guesses but Kahneman says, 'In highly efficient markets, however, educated guesses are no more accurate than blind guesses.'

In Thinking, Fast and Slow, Kahneman relates a story of a speaking assignment that he had with a firm of investment advisers. He was given a spreadsheet having the investment data of 25 wealth advisers for 8 consecutive years which determined their year-end bonuses. He proceeded to calculate the correlation coefficients between the rankings in each pair of years giving a total of 28 correlation coefficients.If there was skill involved then there should be significant correlation between the year-to-year rankings of the advisers.

What Kahneman found was that the average of the 28 correlations was .01 which was statistically equivalent to zero. It meant that the wealth advisers were playing a game of chance, not of skill. No one in the firm including the advisers were aware of this reality. They all thought that they were doing a professional job. But even after being shown that the firm was rewarding luck not skill (a result that they couldn't deny because they had provided the data and could check the results) Kahneman was sure that it would be forgotten immediately.

He thinks that the key reason for the persistence of this illusion is because the traders are indeed using a type of skill. It requires a lot of hard work and training to be able to check economic forecasts , read P&L statements and balance sheets, study competition, etc. But is this skill enough to answer the key question: is all this information already incorporated into the stock price? Here traders switch to feelings and the data shows that it becomes a guessing game.It does not give one correct answer - except in retrospect, which is not very useful. But after all that hard work, traders become resistant to admitting this.

It is like what happens when I type. Sometimes, after typing a few sentences, I will feel that they are not quite fitting into the rest of the post. The logical thing to do would be to delete them. But after having spent all that time and effort in typing those sentences, I don't feel like seeing them disappear into the ether in less than a second. So I think of some excuse to retain them in the post and I generally succeed.

Kahneman adds:

...the illusions of validity and skill are supported by a powerful professional culture. We know that people can maintain an unshakable faith in any proposition, however absurd, when they are sustained by a community of like-minded believers. Given the professional culture of the financial community, it is not surprising that large numbers of individuals in that world believe themselves to be among the chosen few who can do what they believe others cannot.

If there is a discussion with Kahneman on CNBC, I would be a very interested listener,  although the chances of that happening are quite remote. You will not be happy if someone walks into the studio (an economics Nobel Prize winner at that) and says that the self-styled 'Masters of the Universe' are being paid because of their luck and not because of their skill. (But Kahneman seems a polite person so he may have some nice way of saying that they are fooling themselves.)

PS: Intuition - Marvels and Flaws. Daniel Kahneman, Nassim Taleb, Gillian Tett.