How investors can beat the machine

Since common sense beats expert intuition, more investors should perhaps use it.

Psychologist Daniel Kahneman's bestselling and highly regarded book Thinking Fast and Slow is having an increasing impact on the way I analyse shares.

Kahneman and Amon Tversky are famous for their work on the biases that cause us to make poor decisions. These biases are systematic errors, instinctive judgements that we would not make if we thought deliberately - slowly instead of fast. The trouble is: thinking slowly is exhausting. Often we think we're being cold and rational but we simply don't have the resources to be that way, for example we rush to judge statistics, often using insufficient information.

The exposure of these biases leads commentators to claim investors are doomed to make bad decisions, "hard-wired" in the stone-age to take flight at the first sign of trouble and find safety in numbers, a.k.a. following the herd. We are told that when investing, we are our own worst enemies.

Kahneman's book, though, goes well beyond cataloguing biases. His book celebrates "the marvels as well as the flaws of intuitive thought."

Expert intuition

He repeats a story originally told by psychologist Gary Klein: The commander of a team of firefighters pulls his men out of a house they have just entered because of a feeling - an intuition - before he's had time to think about it. Later, he remembers, the fire had been unusually quiet, and he'd sensed danger. He was right, because the heart of the fire was in the basement beneath, and soon after his men had retreated, the house collapsed.

In 'noisy' environments like stockmarkets, algorithms tend to beat human intuitionExpert intuition, Kahneman says, can improve decision making when the environment is sufficiently regular to make accurate predictions and there is the opportunity to learn those regularities through prolonged practice.

In an irregular environment, like the stockmarket where there are countless potential cues for decisions and it takes many years for investors to learn whether they have followed the right ones, he prefers algorithms. In these "noisy" environments, cues - the quietness of the fire - are only weakly predictive and algorithms spot them more readily and deal with them more consistently.

Rules to solve a problem

People associate algorithms with computers, but they have a much longer history. An algorithm is a collection of rules that can be applied consistently to solve a problem. Evidence quoted by Kahneman shows simple algorithms often work best, indeed complicated mathematics (multiple regression) used to weight the components of algorithms can actually make them less predictive.

He says: "The surprising success of equal-weighting schemes has an important practical implication: it is possible to develop useful algorithms without any prior statistical research. Simple equally weighted formulas based on existing statistics or on common sense are often very good predictors of significant outcomes."

I have asked a friendly econometrician what he thinks of this conclusion.

Inspired by the work of pioneering psychoanalyst Paul Meehl and charged with devising a better system for recruiting soldiers to the Israeli army in the 1950's, Kahneman used an algorithm to identify the best recruits.

Where statistical rules tie with expert judgment, it's really a win - as they are much cheaperDispensing with the expert judgement of interviewers, he instructed recruiters to collect information that would allow them to rate each candidate on a scale of one to five on the basis of seven criteria including "responsibility", "sociability" and "masculine pride". Those with the highest ratings would join the Army.

The recruiters were so upset he'd by-passed their expert judgement, Kahneman added a step to the process. After they'd ranked the candidates on each criterion individually in an attempt to prevent favourable impressions from one judgement contaminating another, he said they could "close their eyes" and give each candidate an intuitive score. He then blended the intuitive score with the algorithm's to derive an overall score.

To his relief, subsequent testing confirmed the algorithm had improved the quality of recruits, and to his surprise, informed by their individual assessments of each criteria, the recruiters' intuition was just as effective.

Of nearly 200 subsequent studies reporting comparisons of clinical and statistical predictions, 60% report better accuracy for algorithms. The other 40% score draws, but, Kahneman says: "A tie is tantamount to a win for the statistical rules, which are normally much less expensive to use than expert judgment."

Quite an edge

That's quite an edge, and it's inspired me to make the Decision Engine, my process for selecting shares, more algorithmic over time. The latest version of the algorithm, which I will document next week, bears some resemblance to Kahneman's system for identifying good recruits.

Meehl's book, the one that inspired Kahneman, who has, in-turn, inspired me, was first published in 1954. You might be wondering how an edge identified more than sixty years ago could still exist today. Surely everybody would be using algorithms.

That wasn't Meehl's experience. His book was republished in 1996 with a new preface. It's fascinating reading.

He wrote: "This little book made me famous - in some quarters, infamous - overnight; but while almost all of the numerous prizes and awards that my profession has seen fit to bestow upon me mention this among my contributions, the practicing profession and a large segment - perhaps the majority - of academic clinicians either ignore it entirely or attempt to ward off its arguments, analyses, or empirical facts.

"Thus I am in the unusual position of being socially reinforced for writing something that hardly anybody believes!"

Kahneman says the statistical evidence of expert inferiority contradicts experts' everyday experience - that we know stuff. While that is undoubtedly true, it isn't necessarily the most important stuff. Then there is ego. None of us want to admit that a formula could be better than us (least of all me).

It would seem, though, that in my quest to beat the machine, I'm becoming more machine-like.

Contact Richard Beddard by email: richard@beddard.net or on Twitter: @RichardBeddard

This article is for information and discussion purposes only and does not form a recommendation to invest or otherwise. The value of an investment may fall. The investments referred to in this article may not be suitable for all investors, and if in doubt, an investor should seek advice from a qualified investment adviser.