1) New publication by David Tuller ( @davidtuller1), Caroline Struthers ( @Healthy_Control) and me:

Bias caused by reliance on patient-reported outcome measures in non-blinded randomized trials: an in-depth look at exercise therapy for CFS https://www.tandfonline.com/eprint/KZQD7CZMIXPTJKY9HPWH/full?target=10.1080/21641846.2020.1848262

2) Our analysis focuses on a discrepancy between several randomized trials that report graded exercise therapy (GET) for chronic fatigue syndrome (CFS) to be effective and multiple surveys by patient organizations that indicate just the opposite.

3) All trials on GET for CFS were not blinded and relied on patient-reported outcome measures. Some (for example Edwards 2017 and Wilshire et al. 2018) have suggested that this trial introduces bias that could distort the main outcomes.

4) Patients who know they are receiving an active intervention rather than a passive control might be more optimistic about its effect on their health or report symptoms according to what they think will please the investigators etc.

5) We’ve tried to investigate this further.

We investigated the size of patient-reported improvements, their consistency over time and congruence with objective measurements. We reviewed mediation analyses, plausible mechanisms of improvement and the risk of expectancy effects.

6) First, we found that the reported improvements were really small, comparable to the minimally clinically important differences for each of the questionnaires, and similar to bias that has been attributed to blinding in other studies.

7) Second, we note that the differences between the GET and control group are often no longer statistically significant if assessments are made, not directly after treatment ends, but a couple of months later.

(caveat: this can also be due to decreased statistical power).

8) Third, Vink & Vink-Niese showed in their analysis of the Cochrane review that objective measures indicate little improvements, in contrast to subjective outcomes.

This is important because objective outcomes are believed to be more robust to bias due to lack of blinding.

9) Fourth, mediation analyses suggest that the reported improvements in GET-trials are not the result of an increase in fitness as the reconditioning theory behind this treatment assumed.

Exercise therapy in CFS currently lacks a plausible explanation for improvements.

10) Lastly, expectancy effects were likely large as the treatment manuals of GET show how therapists were instructed to encourage optimism.

Patients were told to interpret their CFS symptoms as a consequence of deconditioning, stress or anxiety instead of an unknown disease.

11) Based on the arguments outlined above, we conclude in our paper, that treatment effects seen in exercise trials for CFS are likely the result of bias associated with a lack of blinding.

12) We acknowledge some uncertainty about our hypothesis but argue that the burden of proof lies, as for all interventions, on those claiming efficacy.

Those who claim that a treatment is effective should provide evidence that reported improvements are not the result of bias.

13) We hope this case study furthers critical assessment of patient-reported improvements in areas of medicine where blinding of therapists and trial participants faces practical limitations.