Read on Twitter
New paper w/ Leila Agha ( @LeilaEcon), David Chan, Daniel Singer, and Diana Zhu: doctors who depart from guideline recommendations cause many preventable strokes. …https://spinup-000d1a-wp-offload-media.s3.amazonaws.com/faculty/wp-content/uploads/sites/29/2020/06/Mastering_the_Art_of_Cookbook_Medicine_final.pdf
Summary:
1) Guidelines change behavior, but docs often don't adhere
2) We run causal forests on RCT microdata, validate external validity
3) Docs don't respond to treatment effects not in guidelines
4) More adherence can prevent 93 strokes/10,000 AFib patients -- a lot!
1) Guidelines change behavior, but docs often don't adhere
2) We run causal forests on RCT microdata, validate external validity
3) Docs don't respond to treatment effects not in guidelines
4) More adherence can prevent 93 strokes/10,000 AFib patients -- a lot!
We know that docs often depart from guidelines -- do they do this because they have info not in guidelines? Would it help patients if we inform docs about guidelines (promote adoption)? Would it help patients to make it harder to depart from guidelines (promote adherence)?
To test, we combine chart data from the Veteran's Health Administration (VHA) where we can see if docs use guidelines (and all patient characteristics) with microdata from 8 RCTs where we flexibly estimate heterogeneous treatment effects using causal forest models.
The clinical setting is the use of warfarin (and other anticoagulants) to reduce the risk of stroke in atrial fibrillation patients (Warfarin also increases bleed risk). About 5 million patients have AFib in the US.
One of our co-authors, Danny Singer, was the original author of a guideline for treating AFib patients based on the CHADS2 risk score and a leading expert on the treatment of AFib. https://pubmed.ncbi.nlm.nih.gov/18574273/
The CHADS2 score, shown below, is a simple risk score based on demographics and comorbidities designed to quantify untreated stroke risk.
When physicians adopt the CHADS2 score, they become immediately less likely to prescribe Warfarin to low CHADS2 patients. However, they still depart from the guideline for 40% of patients.
To evaluate these departures, we estimate causal forest models on RCT microdata using techniques developed by @Susan_Athey and Stefan Wager. We estimate these models to predict treatment effects "out of bag" across trials.
We validate that models estimated on all but one trial successfully predict heterogeneity in treatment effects in left out trials. In other words, these models are externally valid across our trial database. We show that the VHA data is similar on observables to included trials.
The causal forest model shows the CHADS2 score highly correlated with stroke treatment effects, but still only accounts for about 25% of the observed variation. This is both due to misweighting included variables and leaving out important variables.
Next, we write down a structural model where physician treatment decisions depend on beliefs about treatment effects, which in turn depend on guideline use.
After adopting the CHADS2 score, physicians become more sensitive to the component of treatment effects predicted by the CHADS2 score, but they are never sensitive to the residual component of stroke treatment effects.
Are docs selecting on other variables that predict treatment effects not in our causal forest model? We don't find much evidence of this. Variables not in the causal forest model explain a tiny fraction of variation in treatment probabilities.
We use the model and estimated treatment effects to evaluate a number of counterfactuals, including broader CHADS2 adoption, or perfect adherence to the CHADS2 or newer risk scores.
To evaluate these rules, we construct the "efficient frontier" of stroke/bleed outcomes possible as docs go from treating 0% to 100% of patients (marginal cost of pills is low so the real trade-off is stroke vs. bleeds).
In this sense, the status quo is only slightly better than random! Docs are sensitive to stroke treatment effects, but they are also reluctant to treat older patients who tend to have favorable ratios of strokes to bleeds (in the paper, we do a within age robustness check).
Perfect adherence to the CHADS2 score would be better than the status quo. Adherence to a CHADS2 score with better weights is better still. Using all variables that predict stroke treatment effects even better. Optimal is predicting ratio of stroke to bleed treatment effects.
Interestingly, the CHADS2-vasc score, a more modern version of the CHADS2 score is *worse* than the CHADS2 score. Vascular disease does predict stroke treatment effects, but CHADS2-vasc gives it too much weight.
Upshots:
1) Policies which promote adherence can lead to huge patient benefits. This could be order sets, electronic reminders, direct incentives (pay for performance) or hassle costs like prior authorization.
1) Policies which promote adherence can lead to huge patient benefits. This could be order sets, electronic reminders, direct incentives (pay for performance) or hassle costs like prior authorization.
2) If properly weighted, more variables can be better, but guidelines with more variables are not always better! (caution: docs who estimate risk in untreated patients with many variables rather than treatment effects).
3) Guidelines that target the social objective function--in this case, strokes per bleed--do better than guidelines which target one part of the objective function and rely on doc discretion or separate guidelines to handle the other part.
Finally, the future in which robots take over for doctors will be much better and fundamentally change the role of doctors. It is coming soon, but there are some logistical sort out (e.g. https://www.researchgate.net/publication/341830741_Designing_Conducting_and_Reporting_Clinical_Decision_Support_Studies_Recommendations_and_Call_to_Action)
And lastly, this tweet is filler because I deleted one earlier tweet when I was making this thread and Twitter won't let me post it with an empty tweet.
