1. A paper came out on medRxiv a few days ago, reporting that almost half of a Brooklyn population were serotype positive for COVID, and suggesting an association between disease severity and strength of the IgG response. What should we make of it?

https://www.medrxiv.org/content/10.1101/2020.05.23.20111427v1.full.pdf

2. A few observations. One big issue, of course, is whether the seroprevalence estimate accurately reflects the situation in Brooklyn.

My feeling is that there are a few possible sources of selection bias in the sample.

3. One is that we are not told anything about the sample population. It would be useful to know more about the demographics of those included in the study.

4. Another is that the participants all presented at an urgent care clinic. With an IgG test, we don’t expect positive results until several days after onset of symptoms. But during a pandemic, many visits may be associated with proximal or downstream disease consequences.

5. Finding that 47% of those at a Brooklyn urgent care clinic have had COVID does not imply that 47% of the Brooklyn population have had the disease. It would be interesting to know what fraction of those with no history of ILI were IgG positive, but we are not told.

6. We are not told anything about the selectivity of the test used, though the manufacturer suggests it should be in the 98% range. If so, false positives should have minimal effect here. https://www.diasorin.com/sites/default/files/allegati/liaisonr_sars-cov-2_s1s2_igg_brochure.pdf.pdf

7. Overall, the study is consistent with other results suggesting high incidence in the Brooklyn area, but without addressing the sampling bias issues I do not think this study allows provides strong evidence against incidence in the 20-30% range.

8 . The other major claim involves a positive relationship between symptom severity and IgG response.

9. I’m not sure quite what to make of this. First of all, the number of patients scored in this was was small: 240. Of those, almost all fell on the low-end of the severity spectrum, leaving very small sample sizes across 40% of the severity score range.

10. Moreover it is not obvious to me how these 240 patients were selected for inclusion. If randomly, little surprise that showing symptoms correlates w/ seropositivity, and this tells us next to nothing about the role of disease severity in generating a powerful immune response.

11. Finally, a caution about the graphs used to illustrate this relationship. These purport to show a tight relationship between symptom severity index (SSI) and immune response. At a glance, they look pretty impressive.

12. But these have to be confidence intervals around the mean, rather than instead of uncertainty ranges for an individual observation. Why does this matter? The former provide a misleading sense of the spread of data.

13. In our forthcoming book (link in bio), we illustrate with the following example. The graphs below show the same data in two different ways.

At left they are binned, and we plot confidence intervals around the means for each bin. At right, we show the raw data.

14. In the Brooklyn COVID paper, showing the confidence intervals around the mean obscures something critical to interpreting the data: the size of the IgG=0 class and its role in determining the trend observed. Without knowing this, we can’t distinguish two hypotheses.

15. Does the pattern arise because people with higher SSI scores have larger immune responses?

Or does everyone who gets sick has the same immune response, but people with higher scores are more likely to have had the disease in the first place?

16. In conclusion, I don’t see that this paper provides compelling evidence that immune response increases with symptom severity. It may well be true, and perhaps the raw data even support this strongly. But as presented, we simply cannot tell.