Read on Twitter
I recently listened to @katy_milkman and @angeladuckw share an update on the brilliant and bold work they are doing with the @ChangeBcfg initiative!
The talk was largely around the results from their first BCFG mega-study with @24hourfitness
Here is what I learnt
The talk was largely around the results from their first BCFG mega-study with @24hourfitness
Here is what I learnt
Okay, firstly what is a mega-study?
"A very large field experiment in which small, sub-experiments are run synchronously with the same dependent variable."
"A very large field experiment in which small, sub-experiments are run synchronously with the same dependent variable."
What are the benefits of mega-studies?
1. Better study comparability
2. Lower marginal costs for researchers
3. Reduced risk of learning nothing
4. Tournament-like setup means diverse hypotheses
5. Behavioural phenotyping is possible
6. Accelerates pace of scientific discovery
1. Better study comparability
2. Lower marginal costs for researchers
3. Reduced risk of learning nothing
4. Tournament-like setup means diverse hypotheses
5. Behavioural phenotyping is possible
6. Accelerates pace of scientific discovery
The mega-study structure took inspiration from the common task framework which revolutionised AI.
What is a common task framework?
Researchers from different institutions all compete to solve the same problem, under the same constraints, using the same dataset.
What is a common task framework?
Researchers from different institutions all compete to solve the same problem, under the same constraints, using the same dataset.
Okay, so that's what a mega-study is 
What does it look like in practice?
@ChangeBcfg completed their first (model) mega-study with @24hourfitness, a gym chain that has around 4 million members from 430 U.S locations.
What does it look like in practice?
@ChangeBcfg completed their first (model) mega-study with @24hourfitness, a gym chain that has around 4 million members from 430 U.S locations.
The focus of their mega-study was on physical activity.
Why physical activity?
Exercise is proven to lower the risk of premature death, and all the nasty things that start emerging on the other side of your healthspan.
Majority of US adults don't exercise sufficiently.
Why physical activity?
Exercise is proven to lower the risk of premature death, and all the nasty things that start emerging on the other side of your healthspan.
Majority of US adults don't exercise sufficiently.
@24hourfitness mega-study design:
"30 scientists designed 20 different pre-registered research studies with 53 experimental treatments."
Intervention opportunity spaces:
- Sign up survey
- Incentives
- Reminders
- Interactive texts
- Weekly emails
Study registration flow:
"30 scientists designed 20 different pre-registered research studies with 53 experimental treatments."
Intervention opportunity spaces:
- Sign up survey
- Incentives
- Reminders
- Interactive texts
- Weekly emails
Study registration flow:
Key results of the mega-study:
Dependent variable: the increase in average weekly gym visits (regression estimated)
45% of treatments outperform the placebo control and 9% of treatments outperform the best practice approach of using planning, reminders and micro-incentives
Dependent variable: the increase in average weekly gym visits (regression estimated)
45% of treatments outperform the placebo control and 9% of treatments outperform the best practice approach of using planning, reminders and micro-incentives
The top-performing treatments include:
1. Offering a bonus on return
2. Paying more for scheduled workouts
3. Reinforcing a rising social norm
4. Choice of gain or loss framed pay
1. Offering a bonus on return
2. Paying more for scheduled workouts
3. Reinforcing a rising social norm
4. Choice of gain or loss framed pay
1. Offering a Bonus on Return:
Participants earned an extra $0.09 if they came back to the gym the day after missing a workout. (increased gym visits by 27%)
Participants earned an extra $0.09 if they came back to the gym the day after missing a workout. (increased gym visits by 27%)
2. Paying More for Scheduled Workouts:
Participants earned $1.75 - $1.79 for scheduled workouts. (increased gym visits by 22%)
Participants earned $1.75 - $1.79 for scheduled workouts. (increased gym visits by 22%)
3. Reinforcing a Rising Social Norm:
Participants learned Americans are exercising more than before. (increased gym visits by 24%)
Participants learned Americans are exercising more than before. (increased gym visits by 24%)
4. Choice of Gain or Loss Framed Pay:
Participants chose between earning points each day they visited the gym or losing points each day they failed to visit the gym. (increased gym visits by 19%)
Participants chose between earning points each day they visited the gym or losing points each day they failed to visit the gym. (increased gym visits by 19%)
Could people predict the results beforehand?
Interestingly, the researchers asked citizens, health workers and scientists to predict what would work.
They found no correlation between the predictions and the actual results! BIG lesson here.
Interestingly, the researchers asked citizens, health workers and scientists to predict what would work.
They found no correlation between the predictions and the actual results! BIG lesson here.
