I spend much of my time navigating contrasting expert opinion and trying to adjust for biases, and to identify when expert consensus may be simply wrong. Here's the heuristics I use. What do you suggest for improvement? -

2/ A. identify experts' actual area of expertise. I.e. a doctor is not an expert on public health policy. A political science academic is not an expert on running a successful campaign.

3/ B. identify relevant metrics of "expertise" relevant to the question we're answering. Do we look for economists running a top college department or advising countries or advising banks? Identify key 'political' schools of thought among those experts.

4/ C. Identify incentives/biases. i.e. economists advising politicians are generally biased to tell politicians what they want to hear (print money). Doctors hired by Big Tobacco end up finding that smoking is harmless.

5/ D. Take a rough poll of both a. opinion and b. confidence, among relevant experts. Ask experts why other experts who disagree with them are wrong. This helps further expose biases/incentives.

6/ E. What we're left with is a kind of bias/incentive-adjusted weighted opinion from relevant experts. When this provides an overwhelming consensus, it's usually worth accepting. Last question is if there's something structural that may lead the entire field of experts astray

7/ When the bias/incentive-adjusted expert opinion fails to yield a consensus, I usually just think of the question as currently unanswered or unanswerable. If it's very high value, I may keep digging and try to become an expert myself, but usually accept that it's "unknown."

8/ One key premise here is my belief that it's not productive to try to become an expert in all things relevant to you. That's not possible, and often leads to comical false confidence. E.g. "do your own research" can quickly become, "I read WebMD and know more than doctors."

9/ One example of where this process originally failed (and I had to add the possibility that an entire field may be led astray) is in nutrition. 40 years ago, the vast majority of doctors and medical experts of all types recommended high carbohydrate, low saturated fat diets.

10/ Many doctors accepted on scant evidence that consuming lots of dietary cholesterol led to high blood serum cholesterol which is linked (although causality still not totally clear imo) to heart disease. Eating cholesterol -> high blood cholesterol. Intuitive but wrong.