Daniel Kahneman on Intuition and the Outside View

I had the privilege of attending another Santa Fe Institute “Risk Conference” at Morgan Stanley. There was a stellar lineup of accomplished speakers focusing on Old Wine in New Bottles: Big Data in Markets and Finance. The grand finale was “A Conversation with Daniel Kahneman” led by Michael Mauboussin. These two gentlemen are amongst the finest thinkers in finance and two of the most important influences in my effort to compound knowledge while remaining cognizant of my limitations. As Mauboussin is intimately familiar with the subject matter, he was the perfect person to elicit the deepest insights from Kahneman on the most important topics. Below are my notes, which are reproduced here in the form of a dialogue. When I started jotting these down in real-time, I had no visions of writing the conversation up in this form; however, I found myself writing an awful lot with the output resembling an actual transcript. I attempted to be as thorough as possible in keeping the language as consistent with the spirit of the spoken dialogue as possible, though this is hardly perfect. I apologize in advance for the lack of completeness and the tense shifts, but nonetheless I am delighted to share the following in hope that others will be able to learn as much from this conversation as I did.

Michael Mauboussin: When does intuition work or fail?

Daniel Kahneman: Intuition works less often than we think. There is no such thing as professional “expertise.” The Intuitions in chess masters develop with “big data” comes from experience. For people, the immediacy of feedback is especially important to learn the basis of expertise. When feedback comes closer in time to the decision, intuition tends to be a lot stronger. Gary Klein, author of The Sources of Power is hostile to Kahneman’s view. Together they studied the boundary between trustworthy and untrustworthy sources of intuition. Confidence of intuition is NOT a good guide of intuition. If you want to explore intuition, you have to ask “not how happy the individual is” but what domain they are working in. There are some domains where intuition works, and some domains where it does not.  You need to ask “did the individual have an opportunity to learn irregularities on the way to building intuition? In domains where a lot of people have equal degrees of high confidence, they often do not know the limits of their expertise. 

Mauboussin: People blend quantitative and qualitative intuition, but what about disciplined intuition? Is there a better structure to decision-making?

Kahneman: When you put human judgment against simple models, after reading Paul Meehl’s book which showed where the human has access to all of the data behind the model, the model still wins in making decisions. There are no confirmed counter-examples. Studied an interviewing system for combat units. Asked multiple interviewers to speak with each candidate with a focus on one topic only per subject. Previously the interviewers had experienced a looser system without restriction—one interviewer per subject, with a broad focus. Unfortunately the previous system had zero predictive value on subsequent performance. At first, when the interviewers were instructed on a “disciplined” focus/topical breakdown, they were furious. People like using their broad intuitions. The interviewers were given a rating scale of 1 to 5 in each area they were assigned to cover. Eventually we got the data on how performance turned out based on the revised interview process. It turned out that interviews done in this way had much better predictive value for subsequent performance.

The problem with intuitions is how they come too fast. They are subject to confirmation biases. If you look at just one thing independent of all else and reserve judgment until the very end, what ultimately comes to mind will be more valid than if you don’t have discipline. It’s important to stress the independence (focus on 1 topic) to resist and overcome associative coherence—aka the halo effect.

Mauboussin: Define regression to the mean and the problems with it (causality, feedback)? 

Kahneman: Regression is a familiar concept, but not well understood. We see articles like “Why do smart women marry men less smart than they are?” That is an effect without a cause. We can reformulate that question to say that “the distribution of intelligence in men and women is the same” but the sound/implication of the two statements is not equivalent. You have to rid yourself of causation in making such statements. There was a study of the incidence of kidney cancer which described it as mostly rural, Republican districts in the center and south of the USA. Why? Everyone has a theory. But, if you look at the areas where incidence is small, it’s the same answer—mostly rural, Republican districts in the center and south of the USA.  This is so because the rural counties have smaller samples (a lower “n”) so incidences of high and low are more pronounced.

Mauboussin: Talk about the inside vs outside view, and base rates…

Kahneman: Was involved in writing a textbook on decision-making without math for a high school curriculum. Asked the team: “when will we finish the book?” Everyone answered somewhere between 18 and 30 months. Asked another colleague how long it took to write other textbooks in similar situations. This colleague’s answer had been somewhere in the 18 to 30 month range. The answer: 1) not all textbooks ever finished, with somewhere around 40% of them having given up; and, 2) those that were completed all took more than 7 years.

There are two different ways to look at a problem: 1) make an estimate based on a plan and reasonable extrapolation of progress—the inside view. 2) Abstract to the category of the case and ask “what are its characteristics”—the outside view. Intuition prefers the inside view, while the outside view is non-causal and statistical. If you start your analysis from the outside view, with a known base rate, it gives you a fair anchor and  ballpark from which to work.

Mauboussin: People are optimistic. There was a story you told of a few product launch at a company. At what point do you balance optimism vs just giving up? Society wants risks and all the good things that come with them.

Kahneman: Entrepreneurs don’t take risks because they love risk. They do it because they don’t know the odds. They don’t fully appreciate the risks they are taking. Optimism is the engine of capitalism. When you look at big successes, it’s because someone tried something they shouldn’t have.

Everyone should wish their children be optimists. They are happier, persevere more. Though, I don’t want a financial advisor who is an optimist. 

Mauboussin: As we embrace big data, it suggests change. When baseball learned about Moneyball, scouts resisted. With loss aversion, how do you relate this with the degree to which people are willing to embrace big data?

Kahneman: Losses loom larger than gains. Disadvantages are more salient and heavily weighted. In the context of change, one thing is guaranteed: there will be losers and winners. We can know ahead of time that the losers will fight harder than the winners. Losers know what they will lose, winners are never sure exactly what they will gain. People who initiate change don’t appreciate the resistance they will encounter. When reform is done in the regulatory arena, the reforms often compensate the losers making change very expensive. The prescription is to take the outside view.

The endowment effect is strong. The selling price someone sets on a sandwich they already owns and possesses is higher than that same person would price one they do not own. Giving up is more painful than selling something. This is evident in the financial arena. Advisors are helpful, because when they do the selling on someone’s behalf they do not have the same possessive connection and there is no endowment effect. Loss aversion is emotional, so if you make a decision in an advisor role, you can do so without emotion.

Mauboussin: When we look at decision making in an organization, there is noise. What does “noise” mean and why does it matter?

Kahneman: We know why Meehl was right on formulas being better than judges. For example, there was a situation that for each judge, there was a model built to predict what the judge will rule based on their past decisions. You can then compare the judge’s actual decisions with the model. The model is better than the judge. This tells you why people are inferior to formulas. A formula always has the same output. People vary and vary over time. When x-ray readers are asked to view the same image two separate times, 20% of the time they conclude differently. That’s what noise is.

Many organizations have functionaries who decide, but in principle they are interchangeable (credit-rating agencies, etc.) We would want all people to be interchangeable. How many individuals would be random in their actions? 45-50% tend to be variable. That variability is costly. Noise is costly. Most organizations think their employees agree with each other, but they don’t. Experience doesn’t bring convergence, it brings increased confidence. Convergence and confidence are not the same. If a financial advisory asked their advisors to prioritize a list of clients, does each advisor list the same clients in order? Probably not. When there is no selection, noise is costly.

Mauboussin: Give us a synopsis of Philip Tetlock's Superforecasting.

Kahneman: His book Expert Political Judgment was very important. It looked at predictions 10 years after experts made them and concluded forecasters can’t do it. And, the more a forecaster thinks they can do it, they less they actually did. With that knowledge, Tetlock built an IARPA tournament with predictions that covered timespans 6 weeks to a few months out (see my notes from Tetlock’s talks at two past SFI conferences here). He ID’d the superforecasters (the top 2%), which included a wide range of experts and ability. Short-term prediction being possible isn’t revolutionary. What makes superforecasters? A mixture of the inside and outside view. Disciplined intuition. Independent judgment, collated. 

I am skeptical of applying these findings in the political area where political figures themselves take actions that can be deterministic and statements have to be crafted to multiple constituencies, but in the financial arena these findings are very interesting.

The Psychology of Markets at All-Time Highs

Here is our latest market commentary from RGA Investment Advisors, taking a look at the psychology of markets trading at all-time highs.  This is an interesting moment, for it is the first time in my professional investment career that markets are in fact at highs.  This is our attempt to put things into perspective:

The Market at an All-Time High 

Last month, we pointed out the significance of all the major market indices (sans the NASDAQ) surging to record highs.  April was an interesting month in a very different way.  While the major indices digested their gains, there was absolute carnage in the commodity space.  Most notably, gold, the safe-haven of choice for investors over these last few tumultuous years, shed 7.57% on the month.  In one day alone, gold lost 9.6% of its value.  Many continue to blame the decline in gold on some sinister plot or dismiss it as a warning sign for the broader economy.  We think these explanations are far more indicative of the “religion” around gold as an asset, than it is of something meaningful for the economy; we discussed this in our February 2013 Investment Commentary.  To that end, we attribute the decline in gold to two important forces: 1) gold’s failure as a safe-haven during the worst of the Euro crisis, during which the price actually declined; and, 2) the market’s continued resilience at all-time high levels.  Today, we would like to focus on this second point and what it means for the broader investment environment.

First, a necessary digression: we are pleased to say this is the first time in the history of RGA Investment Advisors with the U.S. stock market in milestone, record high territory.  This company was founded amidst the biggest financial crisis since the Great Depression and we take pride in how we have navigated through what even the most experienced sages of market wisdom declare as one of the least forgiving, most challenging investment environments ever.  We promise both to you and ourselves that these years will serve as an important lesson in patience, strategy and humility, for we all know that while today the market giveth, it can just as easily taketh away.

We are self-reflexive at this moment because we think it’s important to be cognizant of the many emotions induced by the market over time.  Further, we constantly want to learn more about how and why the market does what it does from a behavioral perspective, and to that end, have studied the great thinkers in that arena.  One particularly intriguing school of thought is Prospect Theory, which we have introduced in commentaries past.  Investopedia defines prospect theory as “A theory that people value gains and losses differently and, as such, will base decisions on perceived gains rather than perceived losses. Thus, if a person were given two equal choices, one expressed in terms of possible gains and the other in possible losses, people would choose the former.”[1]  To that end, it is the study of how people make probabilistic decisions when there is a degree of both risk and reward, and it holds that people are more sensitive to losses than they are to gains.  In other words, the pain from losses leaves a more profound impact on the human psyche than the pleasure derived from gains.  This idea was introduced by Daniel Kahneman and Amos Tversky in 1979 in a paper entitled Prospect Theory: An Analysis of Decision Under Risk[2] and has been expanded upon ever since.

A corollary of Prospect Theory is an idea known as “the disposition effect.”  This idea holds that people sell stocks that have gone up far quicker than stocks that have gone down.  The disposition effect is closely tied to the concept of “myopic loss aversion” covered in our January 2013 commentary.  Why are these ideas relevant today?  Interestingly, the disposition effect has particularly strong consequences with markets in all-time high territory, and we feel it is important to review them in light of today’s investment environment.  Since people sell gains quicker than they do losses, there is a greater propensity for selling above all-time highs than below.  Therefore, when markets are at all-time highs, selling pressure tends to increase, leading to greater portfolio turnover in Bull than Bear markets.[3] 

Drawing this out further is the important idea that loss aversion does not exist in a vacuum, and as such, there is a reflexive relationship between the success (or lack thereof) of prior decisions and the nature of present and future decisions to be made.  For the typical investor, “after prior gains, he becomes less loss averse.”[4] In other words, people are most risk-seeking after periods of success and most risk-averse after periods of failure.  Meanwhile, prudence would dictate practicing the converse.  This pendulum of risk tolerance fluctuates back and forth dependent on the most recent market action.  Collectively, these ideas relate to another concept we have long discussed—the notion that people fear a crisis most in its aftermath, rather than its inception.  It is thus no surprise that some of the biggest doomsayers in today’s press are those who were caught most off-guard by the crisis in 2007-2009. 

So where do we stand today on this sliding scale of risk aversion?  We think it remains clear that investors have been severely impacted by the recent crisis.  Investors are so risk averse in today’s environment, that in aggregate, they would rather flee into the perceived safety and certainty of returns in bond markets, while simultaneously ignoring longer-term bond market risks and foregoing a more reasonable tradeoff between risk and reward in equity markets.  Despite markets making all-time highs, there remains a healthy skepticism, and even anger, at the very fact this is happening amidst what remains an economy performing closer to trough than peak levels.

As always, we continue to make our decisions based on our sensitivity to price in each and every individual investment that we make, though we are equally cognizant of the sentiment towards risk and reward around us.  When markets are making highs, so many want to be “the one who called the market top” yet we can assure you of only this—on the way up there will be many tops before there is THE top, and it is our belief that through prudent bottoms-up fundamental analysis and disciplined asset allocation we will best insulate ourselves from the type of risks that were borne out in 2007-09.


Past performance is not necessarily indicative of future results.  The views expressed above are those of RGA Investment Advisors LLC (RGA).  These views are subject to change at any time based on market and other conditions, and RGA disclaims any responsibility to update such views.  Past performance is no guarantee of future results. No forecasts can be guaranteed. These views may not be relied upon as investment advice.   The investment process may change over time. The characteristics set forth above are intended as a general illustration of some of the criteria the team considers in selecting securities for the portfolio. Not all investments meet such criteria.  In the event that a recommendation for the purchase or sale of any security is presented herein, RGA shall furnish to any person upon request a tabular presentation of:
(i) The total number of shares or other units of the security held by RGA or its investment adviser representatives for its own account or for the account of officers, directors, trustees, partners or affiliates of RGA or for discretionary accounts of RGA or its investment adviser representatives, as maintained for clients. (ii) The price or price range at which the securities listed in item (i) were purchased. (iii) The date or range of dates during which the securities listed in response to item (i) were purchased.


[1] http://www.investopedia.com/terms/p/prospecttheory.asp

[2] http://www.jstor.org/discover/10.2307/1914185?uid=3739808&uid=2&uid=4&uid=3739256&sid=21102235131477

[3] Notes from Nicholas Barberis at Sante Fe Institute’s Risk: The Human Factor conference. /compounding-the-blog/?currentPage=2

[4] http://forum.johnson.cornell.edu/faculty/huang/prospect.pdf