Read on Twitter
Today I'll be live tweeting
"THE STATE OF THE NATION: A 50-STATE COVID-19 SURVEY REPORT #10: THE PANDEMIC AND THE PROTESTS"
(Lazer et al. August 2020)
The 5th paper I've reviewed purporting to be able to prove large gatherings do/don't spread COVID.
https://kateto.net/covid19/COVID19%20CONSORTIUM%20REPORT%2010%20PROTEST%20AUGUST%202020.pdf
"THE STATE OF THE NATION: A 50-STATE COVID-19 SURVEY REPORT #10: THE PANDEMIC AND THE PROTESTS"
(Lazer et al. August 2020)
The 5th paper I've reviewed purporting to be able to prove large gatherings do/don't spread COVID.
https://kateto.net/covid19/COVID19%20CONSORTIUM%20REPORT%2010%20PROTEST%20AUGUST%202020.pdf
I picked this for a couple of reasons. First, it appears to be another example of a harmful covid research trend that we pointed out in our recent review- applying coarse observational data to erroneously make causal claims about micro level events https://twitter.com/RexDouglass/status/1295391498477826053
Second, the question of whether we can safely engage in mass gatherings isn't getting less important. Protests continue to escalate. Large entertainment events are resuming. The national election requires knowing whether traditional campaign events and in person voting are safe.
Third, whether experts are actually reflecting the scientific state of the art has massive implications for legitimacy and public trust. Lamenting the loss of trust in expertise while behaving in ways that forfeit it, costs lives now and will cost many lives come the vaccine.
The research question is: what is the transmission rate at mass-events? Both on average and in the quantiles from mass-spreader events. If that transmission rate is comparable to or greater than other forms of mobility and commerce then it has public health risks like any other.
It might come as a shock to learn we still do not know the answer to this research question for at least 3 reasons.
1) Public messaging campaigns have reduced scientific uncertainties to moral certainties to encourage good behavior and avoid confusion.
1) Public messaging campaigns have reduced scientific uncertainties to moral certainties to encourage good behavior and avoid confusion.
2) And endless parade of popular media has encouraged and cherry picked scientific findings in an attempt to bolster their preferred political and social messages.
3) The underlying problems of statistical inference aren't simple or obvious. Statistical rigor isn't even universal accepted across fields, departments, or researchers. Mathematical crimes are often ignored if not encouraged in different corners.
To illustrate, the one study I know that did it right-ish is Streeck et al. who compared SARS-CoV-2 antibody rates in a small German town that held a carnival just before a quarantine. It was nearly a natural experiment, and hasn't been replicated since. https://www.medrxiv.org/content/10.1101/2020.05.04.20090076v2
It was done right-ish because:1) It is at the correct unit of analysis - individual people, 2) samples were taken from both the treatment and control group, 3) both groups were plausibly comparable being from the same town and 4) immediate quarantine avoided additional treatments
They found people who went to the carnival had rates of infection 5 times higher than those that didn't, 15.5% vs 3.1%. That's a reasonable treatment effect, we see comparable effects on planes, buses, universities, other spreading events. Masks probably reduce that effect by 1/4
This research design is the floor for what we would accept as medium strength evidence on this question. We should have weak but reasonable priors based on it that crowds spread covid. Any better evidence is going to have to be at least this good of a design.
What we've gotten instead is a mountain of even worse research designs: the wrong unit of analysis (states not people); no direct measure of those who actually went to the event next to those who didn't; no good measurement even of infection rates at all, moving goal posts, etc.
I often joke that these research designs boil down to just taking a cross-tab and declaring protests safe and republicans stupid whenever that date slice points the right way. Lazer et al. 2020 is special, in that it literally just does that. It's not hyperbole. Here's the plot:
Their left hand side is the rate of growth in confirmed covid cases over 2 months of the summer June 1st to August 1st as measured by the NYT.
Their right hand side is the percent of the state's population that attended a protest - as extrapolated from self reporting in an online nonprobability sample survey.
They report the negative bivariate correlation as evidence that protests don't spread covid. You might want to cut them slack that this is really just a policy brief and they admit they shouldn't be saying this since there's a million confounders... but...
Like clockwork this is how they publicized the result
https://news.northeastern.edu/2020/08/11/racial-justice-protests-were-not-a-major-cause-of-covid-19-infection-surges-new-national-study-finds/
https://news.northeastern.edu/2020/08/11/racial-justice-protests-were-not-a-major-cause-of-covid-19-infection-surges-new-national-study-finds/
The write up in that note illustrates how this argument is framed. It starts with a false null hypothesis
"In other words, the demonstrations “were very unlikely to be” the big driver behind the surge in infections in June and July, says David Lazer"
That's not the null.
"In other words, the demonstrations “were very unlikely to be” the big driver behind the surge in infections in June and July, says David Lazer"
That's not the null.
Our prior was that transmission rates at mass events are probably 5-ish times higher than regular mobility, 3 times with masks. Setting the bar at whether an entire state, post-covid-peak has a resurgence larger than states experiencing their first upswing, is NOT THE NULL.
The next move is assert that false sense of certainty in the state of the art:
"One possible explanation may lie in the increasing scientific consensus that indoor proximity may be the dominant path in the transmission of SARS-CoV-2, the coronavirus that causes COVID-19."
"One possible explanation may lie in the increasing scientific consensus that indoor proximity may be the dominant path in the transmission of SARS-CoV-2, the coronavirus that causes COVID-19."
If we knew ^, we wouldn't be desperately trying to find evidence for or against spread in mass outdoor events.
The third move is make the headline "protests don't cause covid" then sprinkle the article with reasons you should never ever say that, and then conclude with the need for more research that can't happen:"Still, researchers acknowledged that a fuller analysis will be required"
For bonus points, if you're the Washington Post you run the same article "scientists find protests don't cause covid" every single time a bad pre-print hits while simultaneously running several dozen articles chasing specific infections from a Trump rally, the RNC, or Sturgis.
If you're not convinced, here's a direct experiment: Dave et al. released two back to back NBER papers using the same research design. One says the protests don't cause covid. The other says Trump's Tulsa Rally didn't cause covid. Neither paper shows that, my reviews below.
For the first paper, WaPo ran a full article and has cited it several times since in multiple articles making this same argument over and over again. https://www.washingtonpost.com/health/protests-probably-didnt-lead-to-coronavirus-spikes-but-its-hard-to-know-for-sure/2020/06/30/d8179678-baf5-11ea-8cf5-9c1b8d7f84c6_story.html
For some reason, WaPo never wrote up the same result, for another event, from the same researchers, in the same NBER preprint, on the same topic. Instead they run several articles like this one https://www.washingtonpost.com/nation/2020/07/09/trump-tulsa-rally-coronavirus/
The main point, is these results aren't good to begin with. I go in depth on both papers and how neither of them establish that COVID-19 doesn't have a higher or at least similar transmission rate to other kinds of mobility. Observational, location level, confirmed counts CAN'T.
A 2nd point is that it's wildly irresponsible to suggest it can, when mathematically it can't. Real people are listening. Real news orgs are picking and choosing these results to feed to their readers which will lead them to make cost-benefit calculations with real consequences.
Third,we still don't have a damn answer to this important question. There needs to be a serious commitment in funding and research to answering this question now, if for no other reason as part of election safety and making sure voting is safe and available to as many as possible
Finally, I hope, and we should all hope, that there is some combination of density, masks, reductions in other kinds of mobility, that we can use to make mass-events safe. I hope that exists. Asserting one way or the other isn't getting us closer to answer, it's setting us back.
Reviews of other works below:
First Dave et al. paper which tries to use synthetical controls to show states didn't get large outbreaks after the first round of protests. No out of sample evaluation, no reasonable null. https://twitter.com/RexDouglass/status/1275186049455583232
First Dave et al. paper which tries to use synthetical controls to show states didn't get large outbreaks after the first round of protests. No out of sample evaluation, no reasonable null. https://twitter.com/RexDouglass/status/1275186049455583232
The second Dave et al. paper which tries to show that the Trump Tulsa rally didn't spread COVID-19. I calculated that it would have at most produced 2 extra cases per county - ridiculously below any even optimistic measurement ability of these models https://twitter.com/RexDouglass/status/1282708491656814594
This paper tried to show that sporting events spread COVID-19 but has a completely broken matching system pairing cities with sports teams to their rural neighboring counties that don't. https://twitter.com/RexDouglass/status/1295009952008830976
This one claimed to find no large uptick after protests. I love the paper for lots of reasons, the front disagregates cases and testing in a way I love, but the back end is a complete mess and they don't setup a reasonable null either. https://twitter.com/RexDouglass/status/1301208895961149442
For the above papers and others I've discussed you can find the full threads here https://twitter.com/RexDouglass/status/1278115752747253760
