The #NARPS paper is now out @Nature: Variability in the analysis of a single neuroimaging dataset by many teams https://www.nature.com/articles/s41586-020-2314-9 thanks to @rotembot @ten_photos @tschonberg and almost 200 others! Commentary by @fMRIstats https://www.nature.com/articles/d41586-020-01282-z

Like many fields, analysis of fMRI data is highly complex with many choices at each stage. #NARPS assessed how differences in analysis workflows affect ultimate hypothesis tests. 70 teams analyzed an fMRI dataset collected by @tschonberg team @TelAvivUni & tested 9 hypotheses.

For 4 of the hypotheses there was good agreement b/w teams, but for the other 5 there was 20-40% endorsement, showing that the same data can lead to markedly different conclusions across teams.

Analysis of the underlying statistical maps showed that there was substantial similarity in the maps between many teams, but even in these cases they often reported divergent hypothesis test results.

But a meta-analysis across teams identified regions that were active across all teams, showing that there were consistent activations when combining across teams

A prediction market (run by @flxhlzmstr et al) assessed researchers' predictions of the results. Participants not on an analysis team greatly overestimated the likelihood of finding positive activation - but strikingly, so did participants who had actually analyzed the data!

What are the takeaways? 1) This is science at its best - testing ourselves severely, honestly admitting our shortcomings, and working to fix the problems for the future. Many thanks to all of the team members who spent tens or hundreds of hours each on this project!

2) It’s likely that any research domain with complex and flexible data analysis procedures will have similar irreproducibility. We need to see similar shootouts across many areas of science.

3) Any dataset should be analyzed using multiple workflows, by multiple independent analysts, in order to assess sensitivity to specific methods. #NARPS results suggest that it is the statistical model, rather than preprocessing, that makes the biggest difference for FMRI.

All data/analyses for the team results are entirely open and reproducible. https://github.com/poldrack/narps/tree/master/ImageAnalyses Raw fMRI data @OpenNeuroOrg https://doi.org/10.18112/openneuro.ds001734.v1.0.4

If you can't pierce the paywall, here is the preprint: https://www.biorxiv.org/content/10.1101/843193v1

read-only version available at https://rdcu.be/b4iHe