The biggest problem in science is the myth that every scientist can make true, novel discoveries, and worse--- that we incentivize smart people to claim discovery, while giving them massive analytic flexibility & the ability to selectively report their findings



[thread]

Half of original articles I participate in have been purely descriptive.

How often does something happen? What is the maximum number of people who might be eligible for a cancer drug?

Questions that seek merely to describe the world, or describe it under stated assumptions

But that is not the coin of the realm!
Claiming discovery is.
We found x causes y
We found drug x shrinks tumor z
Our analysis suggests that all patients should get treated with z
Q is very promising for Y

It is likely that each year ~ a million articles claim to find a relationship, a treatment, a discovery, a potential cure that no one ever knew about before.

Wow!

The vast majority of these claims of discovery are incorrect. They cannot survive replication & are not true.

Side note: Replication truly means replicating the inference of the study.

Not re-doing your exact same experiment-- that's trivial.

If you use your experiment to draw a certain conclusion about the world; the best replication efforts ask whether or not, under a variety of circumstances, your conclusion holds up

ok, side note over.

The majority of novel clams of discovery occur in a system where scientists are under massive professional pressure to be generating discoveries.

Just popping them out of the kitchen like means in a busy restaurant on a Friday night (pre-covid)

Moreover, we allow the scientists to keep trying to find discoveries, over and over again, pulling all the levers on the machine, until one it lines up just right.

Not only do we hand over total analytic flexibility

We let them decide when to report results!

Imagine if we administered a test this way!

You can take as many or as few questions as you want, and then score yourself.

Re-do as you wish.

When you are happy with your score, you can turn it in

Won't everyone get 100%?

If you stand back and think about it, it is truly ironic.

Here we are, a highly intelligent and dominant animal on a planet. As we read our history, we see a species constantly fooling itself by believing things that are not true.

How dumb, we laugh!

Of course, that just applies to our ancestors, we think, and yet we continue to just fool ourselves again and again

Just painfully unable to stop; to set standards; to raise the bar.

We just make different errors, more technical errors, but the same fundamental error

Obvious fixes would be:
pre-registering hypothesis/ analytic plans
really understanding your data inputs
using appropriate endpoints/ controls
abandoning some Qs for the time being that cannot be answered with 2020 tools
changing career incentives

I don't think I have made a single novel discovery in my entire career, and I don't expect I will in the near future.
That may sound like a disappointing self-assessment, but at least I am not fooling myself