Following the brouhaha with the meta-analysis with the curious effect size calculations, I wanted to share some of the effect calculation/analysis errors and mixups in meta-analysis I often see when reviewing or consulting. (Feel free to add your own below!)... https://twitter.com/jackferd/status/1282196815908134913

#1: Not smell checking individual effects. Sometimes, effects just look wrong—typically, implausibly large. Have a sense for how big effects tend to be in your field. In psychotherapy trials, anything above a d=1.0 is really . This may be due to a simple typo while coding.

You have a different problem if the effect size seem correct per what's reported in the paper—you can contact the authors, use (hopefully preregistered) procedures for dealing with study outliers like leave-one-out analyses for influence, the SAMD statistic, etc.

#2: Mixing up standard deviation and standard error. Sometimes, authors report in summary tables the mean (SE) versus mean (SD), and it's an easy thing to flub when coding many studies. I believe this mix-up is going to typically produce erroneously large effect sizes.

If the authors have not provided the SD but you have the sample size and SE for a mean, you can calculate the SD using this formula: SD= SE * √n ... Problem, in many cases, solved!

#3: Violating ES independence. In a given analysis, no two ESs are supposed to have an overlapping population. E.g., two active interventions are compared in the same study to the same control treatment. Including both of these overweights the study in your meta-analysis.

There are at least two ways of addressing this issue: (1) if defensible, you can combine the mean/SDs from the overlapping groups (e.g., combine the two active interventions) using standard formulas ( https://handbook-5-1.cochrane.org/chapter_7/table_7_7_a_formulae_for_combining_groups.htm) ...

(2) if you have moderator codes that hinge on the individual groups (e.g., Tx A vs. Tx B compared to controls), the R package "robumeta" ( https://cran.r-project.org/web/packages/robumeta/index.html) provides an (IMHO) easy to implement framework for analyzing effect sizes that are nested or correlated.

#4: Comparing frank "apples and oranges" without further analysis. This comes up in psychotherapy trials when authors explore effects of a treatment versus control groups. A wait-list control will typically produce a weaker effect than an active control. Including both? Well...

This will increase between-study heterogeneity and potentially make it difficult to interpret the resulting effects. A perhaps defensible way to address this would be to conduct subgroup analyses within control group classes, or moderator analyses examining how they differ.

#5: Ignoring between-study heterogeneity. With some
oft-lost nuance (including by yrs truly; cf. https://onlinelibrary.wiley.com/doi/full/10.1002/jrsm.1230?casa_token=yuhOduiS0joAAAAA%3AB7IWrtq64zGF_e2f-l7lvQ6GvzUdGPjZXyf03DLZR32p0snauqd4jxrg1aC_fl_9iP6KuU3z16YeLA), the I-square index measures the percentage of total variation across studies in a meta-analysis that is due to heterogeneity rather than chance.

These days, it tends to be reported ubiquitously as it appears by default in many meta-analysis packages now. But I've often see *quite* high heterogeneity (75%+) reported on without any attempts to examine it statistically or incorporate it meaningfully into interpretation.

Especially if one happens to still get a "significant" ES despite high heterogeneity that widens the CI bands of an effect estimate. High heterogeneity is a yellow flag: these studies really aren't all the same, so one must carefully think about what the results mean.