Observational studies looking at drug effects in COVID are rolling in at mind blowing speed

Outlining common issues perhaps not well appreciated outside the pharmacoepi community with hopes that this may be useful to folks as they consume unvetted stories based on such studies

An observational/non-randomized study comparing patients treated with drug A vs not treated w A, the most common comparison I've seen for COVID preprints, is vulnerable to 2 main biases..

Bias 1- confounding: some receive treatment and some dont because of a reason e.g symptom severity, this difference manifests into differences in outcomes even if the drug did nothing to cause the outcome

But regression/propensity score adjustment is supposed to fix this, right?

Not fully, 2 points- 1) statistical adjustment only accounts for measured differences, and 2) even an efficient procedure like propensity scores need a large-ish sample size to appropriately account for differences in many measured factors, not what we've seen in COVID preprints

Bias 2- 'immortal time bias'- patients receiving treatment are able to do so because they survive long enough- eg- people dying 1 day after hospitalization are less likely to end up in treated group compared to those surviving longer. Not fixable w stats, only fixed by design

Obs studies comparing alternative treatments are generally slightly less prone to these 2 biases, but they answer a different question- does treatment A work better/worse/similar compared to B? Obs studies are not well suited to answer the question of whether treatment A works

Hopefully this helps unpack some key issues in conduct and interpretation of observational studies of drug treatments- thanks for reading..