A huge thanks to @russpoldrack, @tschonberg, @rotembot, and everyone else who was involved in the design and logistics of this study! Compared to that, which must have been very difficult, analyzing the data was the fun part (at least for me). A couple of thoughts below 1/n https://twitter.com/russpoldrack/status/1263173122875777026

The study used a gambling task, and we all know that gambling activates regions like the ventral striatum - right? But when you and a team of people decide on how to define the Vstr beforehand and then compare it with the actual results, making a judgment becomes tricky. 2/n

E.g., let's say that you run a whole-brain analysis and the center of mass for a cluster is just outside of the Vstr (although some voxels fall within the region). If you knew nothing about the neuroscience literature on gambling and the Vstr, what would you conclude? 3/n

Before seeing the results, our criteria was that the region had to contain the cluster's center of mass. After seeing the result, we were conflicted; we had to say that the effect wasn't in that area, although this went against our gut response 4/n

(And in fact, running a follow-up ROI analysis defined by an independent atlas showed a significant p<0.001 effect in the Vstr - just what you would expect if you were doing a confirmatory analysis, although this wasn't our original objective.) 5/n

One way to address this is by doing just what we did as the analysis teams: Upload the data to a website like Neurovault, report what your criteria were for determining whether the effect existed, and then let the reader make up his own mind 6/n

Allowing the analysis teams to report their confidence in the result was also a brilliant move by the designers. These kinds of judgments made on a continuous scale, should they become more widely accepted, would go a long way towards fixing the file-drawer problem 7/n

(Then again, if wishes were horses, beggars would ride) 8/n

Lastly, I'll admit that I cheated and ran the analysis twice: Once with data preprocessed with fMRIPrep, and once with data preprocessed in AFNI. The results were virtually identical, although slightly stronger & cleaner with fMRIPrep 9/n

Which one I actually submitted to the paper, I guess we'll never know. 10/10