My take on the COVID-CODE review scandal (here https://lockdownsceptics.org/code-review-of-fergusons-model/ and reported https://www.telegraph.co.uk/news/2020/05/10/chilling-truth-decision-impose-lockdown-based-crude-mathematical/ amongst other places)

There is also a good thread on linkedin on this subject (not an oxymoron!) https://www.linkedin.com/feed/update/urn:li:activity:6665353875127578624/?commentUrn=urn%3Ali%3Acomment%3A(activity%3A6664414093383876608%2C6665353766952280064)

First some disclaimers: I haven't reviewed the code myself, and I am not a professional software developer. Nor am I an epidemiologist (it's an old joke, but I really can't even spell it): neithier is the original code review author.

What I have got is a fair amount of experience in writing research code in academic and industry environments, and in trying to predict when both the inputs into the model and the model itself are unknown, and the outputs are non-deterministic.

My comments are highly general then, and not specifically related to the COVID code.

Research and production code are not the same thing. Most research code is shockingly poor quality and shouldn't be exposed to the public gaze. However it should be, so that people can make sure you haven't screwed up. Reinhart and Rogoff will tell you all about that.

It's poor quality because it's written by people who aren't software developers (like me). The model in my industry is that poor quality research code is rewritten as production code by high quality developers eg @ManQuantTech. But academics don't have that kind of resource.

Clearly trading software running in real time needs to be production quality. But does this kind of code need to be? You could argue it should be, given the lives at stake. But this is unneccessary, IMHO.

At a minimum the code should be free of major flaws that would produce a wildly different policy response in the absence of those flaws. In Reinhart and Rogoff that was not the case. The error in the spreadsheet swung the paper from 'anti' to 'pro' austerity.

I am not convinced that this is the case here. In any modelling exercise the result will be influenced by risk from (i) the underlying model, (ii) the inputs into that model, (iii) the implementation of that model. Bad code (part iii) will only be problematic if i and ii are not.

Again I am not an epidi... epido... scientist. But in portfolio optimisation you could make a fairly bad job of implementing a classic CAPM optimiser and it would make no difference. The results would be unstable and not robust, but they would also be if the code was bug free.

The simplifying assumptions of the model, and the huge uncertainty about the inputs, are the major cause of error. A few bugs here and there are not.

I do not think that we are in a very different place here. The COVID model is definitely better than CAPM. Human interactions are easier to model when they are not trading with each other. But I doubt the inputs are known with much certainty.

And crucially, as with CAPM, *a small difference in inputs is likely to produce wildly different results* That's one reason why, sensibly, the model is run stochastically and the results were reported with confidence intervals.

(I should have added that I haven't seen @rjthillman take on this and he has a far better understanding of this sort of work than I do)

Experts have become briefly fashionable again, but we seem to be assuming that all scientists are infallible all seeing people that know everything with certainty. And to be fair, they are a lot better at forecasting than economists or the average HF PM.

There has also been a big call for openness and transparency. This is good! But it does make it easy for people to criticise. And in some cases I think this criticism is unfair.

In particular I personally doubt a production quality implementation would have resulted in a very different outcome. And that is the important test here.

PS From the code review "On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best done by the insurance sector."

I mean, no.