Read on Twitter
Humans are astonishingly bad at setting goals.
We consistently establish targets that invite manipulation.
THREAD: A mental model on goal setting and unintended consequences:
We consistently establish targets that invite manipulation.
THREAD: A mental model on goal setting and unintended consequences:
Goodhart’s Law is a simple mental model:
When a measure becomes a target, it ceases to be a good measure.
If a measure of performance becomes a stated goal, humans tend to optimize for it, regardless of any associated consequences.
The measure loses its value as a measure!
When a measure becomes a target, it ceases to be a good measure.
If a measure of performance becomes a stated goal, humans tend to optimize for it, regardless of any associated consequences.
The measure loses its value as a measure!
Goodhart’s Law is named after British economist Charles Goodhart, who referenced the concept in a 1975 article on British monetary policy.
Goodhart wrote that, “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.”
Goodhart wrote that, “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.”
But the concept was popularized by anthropologist Marilyn Strathern.
In a 1997 paper, she generalized the thinking and called it Goodhart’s Law.
“When a measure becomes a target, it ceases to be a good measure.”
It became a mental model with considerable practical relevance...
In a 1997 paper, she generalized the thinking and called it Goodhart’s Law.
“When a measure becomes a target, it ceases to be a good measure.”
It became a mental model with considerable practical relevance...
The phenomenon has been observed throughout history.
Humans consistently establish targets that invite manipulation.
Indian Cobras & Soviet Nails.
Let’s quickly cover each situation:
Humans consistently establish targets that invite manipulation.
Indian Cobras & Soviet Nails.
Let’s quickly cover each situation:
The Cobra Effect
There were too many cobras in India, so the British colonists started offering bounties for cobra heads.
Locals began breeding cobras, killing them, and turning in the heads to earn bounties.
Many cobras were released, increasing the population of cobras.
There were too many cobras in India, so the British colonists started offering bounties for cobra heads.
Locals began breeding cobras, killing them, and turning in the heads to earn bounties.
Many cobras were released, increasing the population of cobras.
The British measure of cobra elimination (cobra heads) became an explicit target, and thus ceased to be a good measure.
Humans optimized for a goal, in spite of the clear negative consequences.
(I wrote about The Cobra Effect at length in the thread below!) https://twitter.com/sahilbloom/status/1318926201772101632
Humans optimized for a goal, in spite of the clear negative consequences.
(I wrote about The Cobra Effect at length in the thread below!) https://twitter.com/sahilbloom/status/1318926201772101632
Soviet Nails
Soviet factories set goals based on the number of nails produced.
The workers produced thousands of tiny nails.
The factories changed to goals based on the weight of nails produced.
The workers produced a few massive nails.
In both cases, the nails were useless.
Soviet factories set goals based on the number of nails produced.
The workers produced thousands of tiny nails.
The factories changed to goals based on the weight of nails produced.
The workers produced a few massive nails.
In both cases, the nails were useless.
The Soviet measure of nail production (quantity or weight) became an explicit target, and thus ceased to be a good measure.
Humans optimized for it, in spite of the clear negative consequences.
So those are from history, but where do we see Goodhart’s Law in action today?
Humans optimized for it, in spite of the clear negative consequences.
So those are from history, but where do we see Goodhart’s Law in action today?
In Education?
Traditional education has lost its way.
Measures such as standardized test scores and graduation rates became targets, so humans optimized around them.
The result?
An assembly line system that fails to promote creativity and critical thinking in our children.
Traditional education has lost its way.
Measures such as standardized test scores and graduation rates became targets, so humans optimized around them.
The result?
An assembly line system that fails to promote creativity and critical thinking in our children.
We can do better, but it will require rethinking our measures of success and leveraging technology to scale new models such that no child is left behind.
Innovation is coming.
@SynthesisSchool is building something special in early child education. https://www.synthesis.is/
Innovation is coming.
@SynthesisSchool is building something special in early child education. https://www.synthesis.is/
In Business?
You’ve seen this movie before...
Wells Fargo opening fake accounts to hit new account quotas.
Amazon managers hiring to fire to hit internal turnover goals.
CEOs managing to short-term stock goals.
Measures become targets and humans begin manipulating them.
You’ve seen this movie before...
Wells Fargo opening fake accounts to hit new account quotas.
Amazon managers hiring to fire to hit internal turnover goals.
CEOs managing to short-term stock goals.
Measures become targets and humans begin manipulating them.
In Politics?
Approval ratings are a classic measure of the performance of politicians.
But when approval ratings become the goal, they cease to be a good measure.
Bad, short-term decision making to manipulate approval ratings becomes the norm.
Long-term progress stalls.
Approval ratings are a classic measure of the performance of politicians.
But when approval ratings become the goal, they cease to be a good measure.
Bad, short-term decision making to manipulate approval ratings becomes the norm.
Long-term progress stalls.
Goodhart’s Law becomes a mental model that you can’t unsee.
Once you internalize it, you observe it all around you.
It will force you to think carefully about measures, goal setting, and the potential for unintended consequences from the human desire to game the system.
Once you internalize it, you observe it all around you.
It will force you to think carefully about measures, goal setting, and the potential for unintended consequences from the human desire to game the system.
For more, check out these additional resources below:
Article by @petercoy: https://www.bloomberg.com/news/articles/2021-03-26/goodhart-s-law-rules-the-modern-world-here-are-nine-examples?sref=UGGkDWbK
Blog by @koehrsen_will: https://towardsdatascience.com/unintended-consequences-and-goodharts-law-68d60a94705c
Article by @petercoy: https://www.bloomberg.com/news/articles/2021-03-26/goodhart-s-law-rules-the-modern-world-here-are-nine-examples?sref=UGGkDWbK
Blog by @koehrsen_will: https://towardsdatascience.com/unintended-consequences-and-goodharts-law-68d60a94705c
If you enjoyed this, follow me for more threads on business, finance, and mental models.
You can find all of my threads in the meta thread below. https://twitter.com/sahilbloom/status/1284583099775324161
You can find all of my threads in the meta thread below. https://twitter.com/sahilbloom/status/1284583099775324161
If you want to receive more content like this directly to your inbox, sign up for my newsletter.
I’m going to be sharing more exclusive, high-signal content there in the weeks and months to come. Subscribe so you don’t miss anything! https://sahilbloom.substack.com
I’m going to be sharing more exclusive, high-signal content there in the weeks and months to come. Subscribe so you don’t miss anything! https://sahilbloom.substack.com
