Quick thread on Mr P. What is it?

Often we want to generalize some statistic from a potentially biased sample to a population. It might be a measurement (say, partisan support) or some model parameter (maybe an elasticity or treatment effect). How can we do this? 1/

One way is to be more deliberate in constructing the sample to look like the target population (see http://thegeneralizer.org ). Yet often we don’t get to construct our own data. The classical approach here is to use survey weights, to learn more from the relatively rare obs 2/

Sometimes this is fine. Other times you end up giving several thousand times’ as much importance to one observation as another, which can make models hard to fit (to say nothing of whether the overweighted obs’ response is representative of its cell). 3/

Weighting also strays from the generative ideal. It’s only a measurement strategy, not a model.

Mr P instead starts by breaking the sample down into cells (typically demographics), obtaining an estimate for each of those cells, and then using the cell weights /4

of the target population to reweight. Easy, right? If you’re Facebook, sure. Split by demographic cell, run your experiment in each, and reweight by population (that is, slightly)

For the rest of us, the hard bit here is that some cells will be very rare in our sample. /5

For example, we want to generalize to all 24 year old Asian men in Idaho, but have few (or none) in our sample. So how do we get that estimate? That’s where the multi-level regression comes in. Sure, we can’t estimate the effect for 24yo Asian men in Idaho. /6

But can we get a good prior for what that estimate should be? Well, yes. Hierarchical methods _shrink_ the estimate towards higher-up-cells’ estimates (men, Asian men, 24yo men, men in Idaho) when there are few/no observations, and allow close-to-raw estimates for big-N cells. /7

What does this get us? To my mind, two benefits.

1) change your target population? No need to refit. Just take a different weighted average of your cell estimates.
2) easier to validate using standard CV methods.
Give it a try! /end