Read on Twitter
From a study design perspective, if you're trying to figure out "what works for whom" between two treatments—which patient characteristics, moderators predict success in one treatment but not the other—what should you keep in mind if you want reliable effects? Some thoughts...
To start, in a psychotherapy trial, moderation of outcome by treatment and a trait approximately implies at least one of the following is true: (1) treatment mechanism A that is activated by treatment X more than Y is more helpful when a patient has a given trait;
or (2) a trait predicts that patients will activate helpful treatment mechanism A more in treatment X as compared to Y. One example of this in the wild is insight in a trial of psychodynamic therapy with or without transference interpretations: https://psycnet.apa.org/fulltext/2010-10440-015.html
What can we do to maximize possible "real" effects? (a) Meaningfully different comparisons: Sometimes, you see attempts to find moderators between two treatments that look awfully similar to one another. Maybe they differ by one intervention, or the format of the same treatment.
This seems ill-advised; very similar treatments are likely going to share most mechanisms of treatment (case 1 above), and there is less room or difference between approaches for a patient to engage better, resonate more with one over the other to pursue a helpful goal (case 2).
If they differ by one intervention, it best be a huge, high-impact one (e.g., exposure-focus in PTSD treatment). Another way to know beforehand: are there studies on mechanisms/mediators of treatment effects, and do the treatments act on different mediators?
We know less than we'd like about differential effects on plausible mediators of treatment effects for even well-hewn comparisons, like cognitive therapy compared to interpersonal psychotherapy for depression. Maybe they have more similar mechanisms than we'd like to think!
This above all is a huge reason I think it's vital for treatment selection work for us to have multiple, effective treatments for the same problems that seem to work from meaningfully different theories of pathology and treatment. The bigger the separation, the better!
(b) Multiple sites: Even if a given moderator is “really” predictive within sample, other patient or setting factors may accentuate/attenuate the influence of a moderator. This is really hard to examine with typically available datasets, but having multiple sites helps.
Different treatment sites will draw from different patient populations/settings. Well-identified moderators that are predictive across multiple sites that may have different underlying distributions of these contextualizing characteristics seem more likely to further generalize.
(c) Many therapists, not all expert: An RCT of expert therapists very highly trained in a therapy is possibly not going to be representative of therapists in a healthcare system applying a treatment, and IMHO this has implications for moderator generalizability.
It could be that expert practitioners of the therapy are *really* good at doing the core work of the therapy, but that those specific treatment mechanisms will be less powerfully employed "in the wild." On the flip side, perhaps the differences between therapies end up...
... flattened at high expertise, and you will see more homogeneity of treatment mechanisms activated. This could lead to different problems with lack of identification of moderators or poor transfer into meaningful treatment differences that maybe are extant at the high echelons.
(d) Measure more than the basics: This is maybe a rarer problem, but I have seen attempts to do moderator research using nothing but basic demographics and something like the very highly correlated subscales of the SCL-90.
My take is that a moderator needs to index something about their psychopathology and how it is maintained (i.e., what treatment mechanism for whom) or differences that might affect a patient's personal engagement/fit with a given treatment (i.e., what frame for whom).
