Read on Twitter
New working paper alert! "Inferno: A guide to field experiments in online display advertising"
http://ssrn.com/abstract=3581396
THREAD: This guide reviews challenges & solutions from a decade of research.
#marketingacad #econtwitter #fieldexperiments
“Abandon all hope, ye who enter here” - Dante’s Inferno
Online display ad experiments are hell. They are also a proving ground for field experimenters, & have much to teach us. The guide is organized into the nine 9 circles ofhell as applied to #displayad #fieldexperiments
Online display ad experiments are hell. They are also a proving ground for field experimenters, & have much to teach us. The guide is organized into the nine 9 circles of
hell as applied to #displayad #fieldexperiments
Circle 1 Display ad effects are so small
that observational methods fail
. Ad effects explain so little variation in ad outcomes, that they get swamped
by unobserved confounds. Like Dante entering the inferno, we resign ourselves to the necessity of experiments.
Circle 2 Non-compliance is a problem.ITT estimates are imprecise. Placebo
campaigns can recover ATET. BUT, placebos are often invalided because ad delivery algorithms match different types of users to the focal ad vs. placebo ad. New solutions: ghost ads
& tag campaigns.
Circle 3 Statistical power
is pure hell. Median sample sizes (Lewis & Rao '15) required to reliably reject differences of:
100% ROI 
3.3M exposed users 
5% ROI 1.3B exposed users 
Circle 4 Marginal effect of an ad is complex:Users choose browsing
intensity which limits ad frequencyAdvertiser only control
spend
& frequency caps
Ad frequency affects ad outcomes affects ad frequencySolution: Impression-level randomization
Circle 5 Identity fragmentation: People are broken into fragmented 
by devices 
& cookies
We only observe fragments
of person’s total ad exposure & outcomes ad effect estimates are biased (
OR 
) 
Solutions: Person-level
, stratified aggregation
Circle 6 Market spillovers: Ad effect estimates also reflect ads shown to control group.Solutions:
-When competing campaigns overlap, consider ad effects with & without competitor
-ALWAYS Deactivate own
non-experimental campaigns in the same marketplace
Circle 7 Frontier econometrics can help optimize ad incrementality: causal ML
, continuous time
modelling, contextual bandits
/Thompson sampling
.Destination
: Estimate the expected incremental effect of each ad ( #attribution), to bid in real time
Circle 8 Even more problems
: influential outliers
, (incomplete) evolution
of ad metrics
, long run
effects of ads have worse power, activity
bias, spillovers to household members
, lack of offline
ad & sales data, & untraceability of cash
purchases.
Circle 9 Conclusion: Biologists learn a lot from thermophilic microbes
living in temperatures up to 122°C
. In the same way, online display ads provide many lessons for field experimenters
. Research here is rapid, relevant, & exciting!
