Read on Twitter
Had a chat with @CharlieEbersole about the mounting meta-sci evidence that bias protection shrinks effects.
It made me think: meta-psychologists should shift gears and get to the bottom of QRPs. 22 years post Kerr & 9 years post Simmons et al. we still don't have the smoking gun!
It made me think: meta-psychologists should shift gears and get to the bottom of QRPs. 22 years post Kerr & 9 years post Simmons et al. we still don't have the smoking gun!
What we have is a somewhat unspecific hypothesised mechanism with a decent amount of indirect & anecdotal evidence. One group of ppl finds it completely plausible and recognisable, another group swears by William James that they've never p-hacked (and I buy that they think that).
Let me know if I've missed sth, but AFAIK we still haven't observed QRPs *directly*, as they are happening (with the exception of author-driven pub bias, Franco et al).
I predict that some will tell me it doesn't matter - "we" don't need to win the hearts & minds of bias deniers.
I predict that some will tell me it doesn't matter - "we" don't need to win the hearts & minds of bias deniers.
I think it does matter though. Here are a few semi-random reasons my 11-pm brain came up with:
1) We currently have some predictive power, but the applied solutions based on it (e.g. prereg, RRs) rely on heuristics, and they're already breaking in the way coarse heuristics break.
1) We currently have some predictive power, but the applied solutions based on it (e.g. prereg, RRs) rely on heuristics, and they're already breaking in the way coarse heuristics break.
We've thrown the term "prereg" at everything and everyone w/o specifying details, and we're predictably seeing poor preregs & obviously biased studies that carry the badge. We get reactance and I don't blame ppl. Being forced into what feels like a mindless ritual is infuriating.
Long story short: I want predictions and solutions to be more precise. That way reform can be way more efficient and effective: We could avoid wasteful & reactance-inducing "overprescription" AND we could work on flexible solutions for tricky cases. How cool would it be to know >
> exactly how a given source of bias plays out IRL and how to best neutralise it in different settings?
Ok, point 1 turned out to be two points (improved prediction and intervention). The remaining ones will be quick to compensate for that:
2) It would be sooooooo much fun!
Ok, point 1 turned out to be two points (improved prediction and intervention). The remaining ones will be quick to compensate for that:
2) It would be sooooooo much fun!
3) It's an area where we can play out all the strengths of our psych expertise. Rather than dressing up as amateur pol scientists, sociologists, philosophers of science, or statisticians, here we're actually talking about bread-and-butter cognition and behaviour!
It's bedtime so I have to stop here.
Of course catching researchers in the questionable act won't be easy at all (otherwise we would have done it already). But hey, don't we pride ourselves on our ingenious experimental designs? Lack of creativity shouldn't be the issue...
Of course catching researchers in the questionable act won't be easy at all (otherwise we would have done it already). But hey, don't we pride ourselves on our ingenious experimental designs? Lack of creativity shouldn't be the issue...
