New paper with @danjholman and Kelvyn Jones out now in Methodology journal – and #openaccess as well! The paper focuses on #intersectionality, but the message of it actually applies to all multilevel models… so here’s a nerdy methods thread https://econtent.hogrefe.com/doi/abs/10.1027/1614-2241/a000167 (1/16)

In a MLM, we are often interest in what higher-level units are doing. Eg are schools different from each other? Or neighbourhoods? Or countries? Or different intersections of society? If so, which schools? Which nhoods? (2/16)

To do this, we might look at a model’s random effects – but these might be unreliable, eg if there aren’t many observations in a given unit. So a neighbourhood might appear high performing in some way, but that’s just because there aren’t many measurements of people in it. (3/16)

Fortunately, multilevel models / random effects use Bayesian shrinkage – if higher level units are small, these will be shrunk closer to the mean – ie the random effect will be closer to zero than its raw effect. (4/16)

This shrinkage is determined by two other things too: (1) how much variance there is between higher level units, and (2) how much variance there is within those units. (5/16)

If there’s lots of variance *between* higher level units, that implies there are generally real differences between places. So we don’t want as much shrinkage. (6/16)

Conversely, if there’s lots of variance *within* higher level units, that suggests actually differences between units might be more to do with chance differences of composition. So we want more shrinkage. (7/16)

The problem arises if there is some omitted variable not considered. So let’s say our higher level units are schools, and we have state-funded and private-funded schools in our sample. (8/16)

There will be lots of between-school variance because of the difference between private and state schools, meaning less shrinkage. But if we were to control for school type, the variance of those random effects would reduce, and so the shrinkage would increase. (9/16)

Which is correct? Well, probably the second one controlling for schools – because there are real differences between the two groups. The sample of schools isn’t random – it’s split in two. (10/16)

If we didn’t control for that variable, we are saying we are actually more certain about the differences between the schools than we actually are. So we might end up saying schools are different from each other when actually they aren’t. (11/16)

Even 2 state schools might appear different in the model without controlling for school type, when that difference is actually down to chance, because not enough shrinkage will happen because of the state school/private school differences. (12/16)

This applies to intersectionality studies (eg https://www.sciencedirect.com/science/article/pii/S0277953617306664) as shown in the paper, where the higher level is social intersection. These intersections aren’t a random sample – they are divided by the vars making up the intersections (gender, age, ethnicity etc) (13/16)

A fixed effects approach (dummy variables for each intersection) is even worse as it has no shrinkage at all. So what's the solution?, think about what might be causing differences between your higher-level units, and control for them. (14/16)

So w/intersectionality, control for the main effect of gender etc, and (if there are still diffs between intersections) 1st order interactions, etc Or in education, control for the variables making the big differences between schools – state/private, or whatever else. (15/16)

Without doing so, the residuals might show bigger differences than there really are. Shrinkage is useful, and some is better than none (as you’d get if using dummy variables / fixed effects in a regression). But it relies on assumptions that we know are often imperfect (16/16)

Ps, if you want to know more about how shrinkage works, this is a good explainer http://www.bristol.ac.uk/cmm/learning/videos/residuals.html