Read on Twitter
The authors said they used “ci means” in Stata on the 0/1 variables (test - / test +) for the 3330 participants, and they don’t know how the weights were used. I think I know, thanks to help from @achouldechova, @DonskerClass, @j_kalla, @StatStas, and Jas Sekhon.
“ci means” is designed for a different purpose: a confidence interval for the population mean from a normal sample. In Stata lingo, it accepts only “aweight” or “fweight,” per the manual page here: https://www.stata.com/manuals/rci.pdf ">https://www.stata.com/manuals/r...
Using either of these is telling Stata that a data point with a large weight is summarizing a large number of independent observations: #u11.1.6weight">https://www.stata.com/manuals/u11.pdf #u11.1.6weight">https://www.stata.com/manuals/u... But that isn& #39;t what the authors’ weights represent.
From those inputs, Stata would think there were roughly 94 participants in the data set who independently tested positive — and badly underestimate the variance, since there were really just 50. Not Stata’s fault: it was misinformed by what is effectively a data entry error.
For stats nerds: I think this was a roundabout calculation of a Wald interval for a binomial proportion estimate of 2.81% with n = 3330, matching the [2.24%, 3.37%] interval of Table 2. Thanks @goodmanmetrics for pointing out this suspicious coincidence.
Stata has commands like svy that can help with poststratification (stats term for what the authors were trying to do), but ci means isn’t meant for that purpose.
I asked the authors to confirm if they used aweight or fweight, or if I’ve misunderstood anything about what command they used. They declined to say but offered kind words of appreciation for my critical feedback.
They also haven& #39;t answered how they did the “cluster adjustment” or whether their LA County press release was based on the same calculation. They say they don’t have bandwidth to answer questions like these now. Fair enough, I do understand they& #39;re fielding hundreds of emails.
In sum, the preprint& #39;s error bars are a meaningless product of flubbed calculations on top of misused software. They haven& #39;t been corrected and can& #39;t be without the raw data. No one knows where the error bars in the LAC press release came from, but I& #39;m not inclined to trust them.
Again, to my knowledge, the only mathematically defensible analyses to date — offered by external statisticians — find that the data in the preprint are too noisy to conclude anything interesting about prevalence in Santa Clara County.
I& #39;m still academically curious to see how a good analysis of the full data will turn out, if one ever materializes, but any data analysis may be a pointless exercise from the start, given other issues that have come to light about the data. A couple new-ish ones below:
@jjcherian points out that the authors seem to have used the wrong numbers for the test& #39;s false positive rate, ignoring the false positive rate for IgM https://twitter.com/jjcherian/status/1252716924812828676">https://twitter.com/jjcherian...
The preprint only mentions the lower IgG false positive rate, even though they call a participant seropositive if the test kit is positive for *either* IgG or IgM. Total false positives in the specs could be as high as 5/371 (confidence interval [0.4%, 3.1%])
![The preprint only mentions the lower IgG false positive rate, even though they call a participant seropositive if the test kit is positive for *either* IgG or IgM. Total false positives in the specs could be as high as 5/371 (confidence interval [0.4%, 3.1%])](https://pbs.twimg.com/media/EWZX1UVUcAAXjzW.png "The preprint only mentions the lower IgG false positive rate, even though they call a participant seropositive if the test kit is positive for *either* IgG or IgM. Total false positives in the specs could be as high as 5/371 (confidence interval [0.4%, 3.1%])")
![The preprint only mentions the lower IgG false positive rate, even though they call a participant seropositive if the test kit is positive for *either* IgG or IgM. Total false positives in the specs could be as high as 5/371 (confidence interval [0.4%, 3.1%])](https://pbs.twimg.com/media/EWZYSI3U0AYCw-C.png "The preprint only mentions the lower IgG false positive rate, even though they call a participant seropositive if the test kit is positive for *either* IgG or IgM. Total false positives in the specs could be as high as 5/371 (confidence interval [0.4%, 3.1%])")
![The preprint only mentions the lower IgG false positive rate, even though they call a participant seropositive if the test kit is positive for *either* IgG or IgM. Total false positives in the specs could be as high as 5/371 (confidence interval [0.4%, 3.1%])](https://pbs.twimg.com/media/EWZYYjMU8AAH0Bi.png "The preprint only mentions the lower IgG false positive rate, even though they call a participant seropositive if the test kit is positive for *either* IgG or IgM. Total false positives in the specs could be as high as 5/371 (confidence interval [0.4%, 3.1%])")
Link to package insert: https://imgcdn.mckesson.com/CumulusWeb/Click_and_learn/COVID19_Package_Insert_Rapid.pdf">https://imgcdn.mckesson.com/CumulusWe...
If we change 369/371 to 366/371, then by my calculations the author& #39;s delta method confidence interval for prevalence (based off their "cluster-adjusted interval" for q) becomes [-0.4%, 3.6%].
![If we change 369/371 to 366/371, then by my calculations the author& #39;s delta method confidence interval for prevalence (based off their "cluster-adjusted interval" for q) becomes [-0.4%, 3.6%].](https://pbs.twimg.com/media/EWZfqucUcAA-SSh.png "If we change 369/371 to 366/371, then by my calculations the author& #39;s delta method confidence interval for prevalence (based off their \"cluster-adjusted interval\" for q) becomes [-0.4%, 3.6%].")
![If we change 369/371 to 366/371, then by my calculations the author& #39;s delta method confidence interval for prevalence (based off their "cluster-adjusted interval" for q) becomes [-0.4%, 3.6%].](https://pbs.twimg.com/media/EWZf0vaU8AAam6y.png "If we change 369/371 to 366/371, then by my calculations the author& #39;s delta method confidence interval for prevalence (based off their \"cluster-adjusted interval\" for q) becomes [-0.4%, 3.6%].")
@DanielJHemel gives further information about the false positive rate: https://twitter.com/DanielJHemel/status/1252598622455119873">https://twitter.com/DanielJHe...
Also a new story by @stephaniemlee raises very serious concerns about how participants were recruited: https://www.buzzfeednews.com/article/stephaniemlee/stanford-coronavirus-study-bhattacharya-email">https://www.buzzfeednews.com/article/s...
At this point there seem to be converging strains of evidence that the study can& #39;t tell us much about COVID-19 prevalence. Fortunately there are other seroprevalence studies coming out soon which I hope will shed better light.
I sincerely appreciate the heroic efforts of so many dedicated scientists who are working very fast to answer critical questions. It& #39;s much harder to design a flawless study and analysis than to criticize it as I& #39;m doing here.
Having done some applied work myself I know how hard it is. Some mistakes are inevitable even given best efforts to be careful. That makes transparency of methods, code, and (where possible) data all the more important.
