New working paper alert!
"Inferno: A guide to field experiments in online display advertising"
http://ssrn.com/abstract=3581396

THREAD: This guide reviews challenges & solutions from a decade of research.
#marketingacad #econtwitter #fieldexperiments

“Abandon all hope, ye who enter here” - Dante’s Inferno
Online display ad experiments are hell. They are also a proving ground for field experimenters, & have much to teach us. The guide is organized into the nine 9 circles of hell as applied to #displayad #fieldexperiments

Circle 1 Display ad effects are so small that observational methods fail. Ad effects explain so little variation in ad outcomes, that they get swamped by unobserved confounds. Like Dante entering the inferno, we resign ourselves to the necessity of experiments.

Circle 2 Non-compliance is a problem.
ITT estimates are imprecise. Placebo campaigns can recover ATET. BUT, placebos are often invalided because ad delivery algorithms match different types of users to the focal ad vs. placebo ad. New solutions: ghost ads & tag campaigns.

Circle 3 Statistical power is pure hell.
Median sample sizes (Lewis & Rao '15) required to reliably reject differences of:
100% ROI 3.3M exposed users
5% ROI 1.3B exposed users

Circle 4 Marginal effect of an ad is complex:
Users choose browsing intensity which limits ad frequency
Advertiser only control spend & frequency caps
Ad frequency affects ad outcomes affects ad frequency
Solution: Impression-level randomization

Circle 5 Identity fragmentation: People are broken into fragmented by devices & cookies
We only observe fragments of person’s total ad exposure & outcomes ad effect estimates are biased (OR )
Solutions: Person-level , stratified aggregation

Circle 6 Market spillovers: Ad effect estimates also reflect ads shown to control group.
Solutions:
-When competing campaigns overlap, consider ad effects with & without competitor
-ALWAYS Deactivate own non-experimental campaigns in the same marketplace

Circle 7 Frontier econometrics can help optimize ad incrementality: causal ML, continuous time modelling, contextual bandits/Thompson sampling.
Destination: Estimate the expected incremental effect of each ad ( #attribution), to bid in real time

Circle 8 Even more problems: influential outliers, (incomplete) evolution of ad metrics, long run effects of ads have worse power, activity bias, spillovers to household members, lack of offline ad & sales data, & untraceability of cash purchases.

Circle 9 Conclusion: Biologists learn a lot from thermophilic microbes living in temperatures up to 122°C. In the same way, online display ads provide many lessons for field experimenters. Research here is rapid, relevant, & exciting!