1/ Brief EBM thread on cognitive bias related to interpreting early information, or "How to assess if an unconventional arguments is based on shared vs selective facts".

Inspired by posts with @drjohnm

2/ One difficulty with EBM is assessing novel arguments based on new information that go against the status quo. Especially if such views are unconventional!

3/ The biases for "early information" can go two ways.

Our preference for early information might weigh it more strongly.

Or our preference for known information may weigh strongly against "early information".

4/ Early information may offer insights that make existing evidence less valid.

Alternatively, early information may be more biased/erroneous, whereas known evidence is often more validated & robust.

5/ When assessing novel arguments, its important to consider:

1. Does it consider both biases of "early information"?
2. Does it evaluate all the evidence (shared facts), or is it selective?
3. Are limitations addressed sincerely, or merely to "say" it was done?

6/ Re #2: How do we know if evidence is selective?

This is hard. No easy answers. We have to do our homework on all existing data/research & all new information.

Non-experts are at higher risk for confirmation bias, as by definition their knowledge on the topic is selective.

7/ In summary, appraising unconventional arguments is hard.

Assess for "early information" bias. Look to see if presented evidence is selective or all-inclusive (shared). And partner with topic experts when possible!

/end