(1/n) The Pfizer SARS-Cov-2 vaccine trial has 4 interim analyses. Each interim has a set number of events and certain splits result in declaring efficacy or futility. If they were to announce “the trial is continuing after interim X”, what might we learn?
https://pfe-pfizercom-d8-prod.s3.amazonaws.com/2020-09/C4591001_Clinical_Protocol.pdf

(2/n) First off, I wouldn’t make these announcements. The trial is continuing per protocol. We need the trial to finish per protocol to conclude anything. I think people misinterpret these announcements all the time (hence this tweetorial….). Other opinions differ….

(3/n) The interims require specific splits. The first interim, for example, happens with 32 events (instances of COVID). If 6 or less of these are in the vaccine arm, the trial declares efficacy. If 15 or more are in the vaccine arm, the trial declares futility.

(4/n) Thus, knowing the required splits and knowing “the trial is continuing”, we can determine the data must lie in a range. If the announcement says “we are continuing after the first interim” we know those 32 events had between 7 and 14 events on the vaccine arm.

(5/n) If I had prior beliefs about the vaccine, I can update them when I hear "the trial continued after interim X. Suppose my original beliefs looks like the graph below (Beta(4,3) for stat folks). I’m generally optimistic, but have a lot of uncertainty. Red is 95% CI.

(6/n) My prior mean vaccine efficacy is 57%. My prior guess is that the trial has a 63% chance of declaring efficacy. This isn’t the power, which assumes the vaccine works. This 63% incorporates my uncertainty on whether the vaccine works or not.

(7/n) After a “continue” at interim 1, here are my revised belief. Importantly, they haven’t changed much. I’m eliminated some of the values near 100% VE (would have stopped for efficacy) but my posterior mean 0.536 and my probability of eventual success is 58%.

(8/n) While continuing interims do make me a little more pessimistic, even continuing after interim 4 my posterior mean is 0.493, probability of eventual success 40%, and CI roughly between 30-70%. I’m never “sure” of what will happen, nor can I accurately guess VE.

(9/n) If you walked into the trial incredibly optimistic or pessimistic (but with at least some uncertainty!), then your beliefs would change. The incredible optimist would be disheartened the trial didn’t stop early for efficacy, the pessimist would grow more hopeful.

(10/n) But if your prior was focused on the 30-70% range, most of that relative probability is unchanged. If the trial is continuing, you tend to stick with that range and have to just wait and see what happens (I’m happy to do these plots for anyone’s preferred prior).

(11/n) Quick note, people are WAY too quick to analyze timing of these announcements, etc. If you want to see overreactions to interims…google Oncothyreon, too optimistic early and then disappointment later. https://www.fiercebiotech.com/r-d/oncothyreon-shares-routed-as-stimuvax-phiii-trial-soldiers-on

(12/12) Again, I wouldn’t make these announcements on principle, but the information conveyed publicly is fairly small. If the trial is continuing, the vaccine likely is somewhere between a dud and good, and it might end up winning or losing the trial.