Read on Twitter
Had a chat with @CharlieEbersole about the mounting meta-sci evidence that bias protection shrinks effects.
It made me think: meta-psychologists should shift gears and get to the bottom of QRPs. 22 years post Kerr & 9 years post Simmons et al. we still don& #39;t have the smoking gun!
It made me think: meta-psychologists should shift gears and get to the bottom of QRPs. 22 years post Kerr & 9 years post Simmons et al. we still don& #39;t have the smoking gun!
What we have is a somewhat unspecific hypothesised mechanism with a decent amount of indirect & anecdotal evidence. One group of ppl finds it completely plausible and recognisable, another group swears by William James that they& #39;ve never p-hacked (and I buy that they think that).
Let me know if I& #39;ve missed sth, but AFAIK we still haven& #39;t observed QRPs *directly*, as they are happening (with the exception of author-driven pub bias, Franco et al).
I predict that some will tell me it doesn& #39;t matter - "we" don& #39;t need to win the hearts & minds of bias deniers.
I predict that some will tell me it doesn& #39;t matter - "we" don& #39;t need to win the hearts & minds of bias deniers.
I think it does matter though. Here are a few semi-random reasons my 11-pm brain came up with:
1) We currently have some predictive power, but the applied solutions based on it (e.g. prereg, RRs) rely on heuristics, and they& #39;re already breaking in the way coarse heuristics break.
1) We currently have some predictive power, but the applied solutions based on it (e.g. prereg, RRs) rely on heuristics, and they& #39;re already breaking in the way coarse heuristics break.
We& #39;ve thrown the term "prereg" at everything and everyone w/o specifying details, and we& #39;re predictably seeing poor preregs & obviously biased studies that carry the badge. We get reactance and I don& #39;t blame ppl. Being forced into what feels like a mindless ritual is infuriating.
Long story short: I want predictions and solutions to be more precise. That way reform can be way more efficient and effective: We could avoid wasteful & reactance-inducing "overprescription" AND we could work on flexible solutions for tricky cases. How cool would it be to know >
> exactly how a given source of bias plays out IRL and how to best neutralise it in different settings?
Ok, point 1 turned out to be two points (improved prediction and intervention). The remaining ones will be quick to compensate for that:
2) It would be sooooooo much fun!
Ok, point 1 turned out to be two points (improved prediction and intervention). The remaining ones will be quick to compensate for that:
2) It would be sooooooo much fun!
