Read on Twitter
=grant reviewers as gatekeepers hindering replication work=
Two years of Hong Kong's GRC grant requests for doing replications+extensions.
Last year: Score 3.5/5
Asked clarifications, I resubmitted with details.
This year: Score 2/5. Fascinating feedback.
Examples below
Two years of Hong Kong's GRC grant requests for doing replications+extensions.
Last year: Score 3.5/5
Asked clarifications, I resubmitted with details.
This year: Score 2/5. Fascinating feedback.
Examples below
Disconnected feedback:
"Mere replication studies are difficult to publish in top journals" (we published plenty this year, & there were extensions in each project)
AND
"especially concerned author wishes to embark on this as an assistant professor (assuming he is untenured)"
"Mere replication studies are difficult to publish in top journals" (we published plenty this year, & there were extensions in each project)
AND
"especially concerned author wishes to embark on this as an assistant professor (assuming he is untenured)"
Want more?
Here's another...
"contribution to theory, methods, or practice can be questioned"
and
"The latter [null replication effect] finding would be difficult to publish in high quality journals, in my view."
![Want more? Here's another..."contribution to theory, methods, or practice can be questioned"and "The latter [null replication effect] finding would be difficult to publish in high quality journals, in my view."](https://pbs.twimg.com/media/EbwDbInVAAAFOCi.jpg "Want more? Here's another...\"contribution to theory, methods, or practice can be questioned\"and \"The latter [null replication effect] finding would be difficult to publish in high quality journals, in my view.\"")
Here's another...
"contribution to theory, methods, or practice can be questioned"
and
"The latter [null replication effect] finding would be difficult to publish in high quality journals, in my view."
You gotta love this next one:
No need to do replications because @OSFramework (CC @BrianNosek) is already doing those.
&
"good science also requires extensions" (which we do & proposed)
&
I successfully replicated results so why "claimed large replication crisis" (I didn't)
No need to do replications because @OSFramework (CC @BrianNosek) is already doing those.
&
"good science also requires extensions" (which we do & proposed)
&
I successfully replicated results so why "claimed large replication crisis" (I didn't)
In conclusion - perhaps the best of all:
"Mass replications may not be a worthy research direction for an assistant professor, who is under time pressure for earning tenure and should focus on
building/testing theories and developing expertise in a subject area."
"Mass replications may not be a worthy research direction for an assistant professor, who is under time pressure for earning tenure and should focus on
building/testing theories and developing expertise in a subject area."
Panel conclusion:
"there is little attention to theory and possible theoretical processes"
"Any replication and failure to replicate
needs some reflection at a theoretical level" oh dear.
"focus on replication moved psychology even further from thoughtful reflection"
"there is little attention to theory and possible theoretical processes"
"Any replication and failure to replicate
needs some reflection at a theoretical level" oh dear.
"focus on replication moved psychology even further from thoughtful reflection"
Going beyond the rejected grant application second year straight - which I didn't really have expectations of getting, there's a clear message here:
Doing replications/open-science is harmful for careers of early career researchers.
Is this the message HK's RGC wants to send?
Doing replications/open-science is harmful for careers of early career researchers.
Is this the message HK's RGC wants to send?
To be fair, there was one grant reviewer this year who was absolutely delighted by this proposal, and last year I had 2 of these very positive reviewers (out of 5).
But in both cases, they got wiped out by extremely low scores by reviewers hostile to the general aim.
But in both cases, they got wiped out by extremely low scores by reviewers hostile to the general aim.
Other insights I got from these two failures (HK context):
Grant reviewers are gatekeepers partly responsible for lack of replications in our field.
Most grant reviewers do not take time to read grant proposals in detail.
Unfortunately, we need specialized replication grants.
Grant reviewers are gatekeepers partly responsible for lack of replications in our field.
Most grant reviewers do not take time to read grant proposals in detail.
Unfortunately, we need specialized replication grants.
Following another rejected Teaching Development Grant from last week (which sponsored us last semester) - I'm currently left with no budget for replications for upcoming academic year.
Used up what I could for this from my seed money.
Need to think creatively, or move on.
Used up what I could for this from my seed money.
Need to think creatively, or move on.
Final note:
We have amazing team & students, proven track record completing over 70 replications & extensions, already published few with many more to come.
All we needed was money to pay for online data collection, nothing more.
Know grants that can sponsor us? let me know.
We have amazing team & students, proven track record completing over 70 replications & extensions, already published few with many more to come.
All we needed was money to pay for online data collection, nothing more.
Know grants that can sponsor us? let me know.
