Reproducible reporting

NYT didn’t fully fact check their front page story.

They just aggregated content from local papers and classified a murder victim as dying of COVID.

Importantly, the public then attempted to reproduce the reporting. And found an error by the 6th name. https://twitter.com/timcrimmins/status/1264340227436556291

Now, this error doesn’t affect the substance of *this* story. Many have indeed died from COVID!

But it does point to an important difference between open source and closed source journalism.

It’s less trustworthy if the public can’t reproduce the reporting.

It’s analogous to the reproducibility crisis in academia.

Scientists at least publish their methodology & in theory encourage independent replication. Yet many papers don’t replicate.

How much worse is the replication crisis in closed source journalism?
https://en.wikipedia.org/wiki/Replication_crisis

Closed source journalism means anon quotes, no links to primary sources, a “trust us” atttiude based on publication’s brand.

Open source journalism means linking to public tweets as quotes, linking to primary sources on (eg) Document Cloud, and allowing public to confirm facts.

Reproducibility and open source would also mean a change in the culture.

Rather than treating corrections as humiliations and marketing NYT as the TRUTH, an alternative is to encourage corrections as pull requests and embrace a model of iterative truth. https://twitter.com/balajis/status/1252725165126873089