Read on Twitter
In this paper in @NatureEcoEvo we outline 8 major problems that can occur with traditional ways of reviewing the literature, and provide concrete advice on how to avoid them: https://rdcu.be/b8pp0
This thread outlines our key points! (1/23)
#EvidenceSynthesis #SystematicReview
This thread outlines our key points! (1/23)
#EvidenceSynthesis #SystematicReview
Traditional ways of reviewing the literature may be susceptible to bias and end up giving us incorrect conclusions.
This is of particular concern when reviews address key policy- and practice- relevant questions. (2/23)
This is of particular concern when reviews address key policy- and practice- relevant questions. (2/23)
Systematic reviews aim to maximise rigour and minimise susceptibility to bias using rigorous methods.
But despite these methods being available, poor reviews are still published: these could be improved with a few key processes; some not prohibitively costly. (3/23)
But despite these methods being available, poor reviews are still published: these could be improved with a few key processes; some not prohibitively costly. (3/23)
Here follow 8 problems with traditional literature reviews, each with suggested solutions. (4/23)
Problem 1: Lack of relevance - limited stakeholder engagement can produce a review that is of limited practical use to decision-makers (5/23)
Solution 1: Stakeholders can be identified, mapped and contacted for feedback and inclusion without the need for extensive budgets - check out best-practice guidance: https://stakeholdersandsynthesis.github.io/ (6/23)
Problem 2: Mission creep - reviews that don't publish their methods in an a priori protocol can suffer from shifting goals and inclusion criteria. (7/23)
Solution 2: carefully design and publish an a priori protocol that outlines planned methods for searching, screening, data extraction, critical appraisal and synthesis in detail. Make use of existing organisations to support you (e.g. @EnvEvidence; http://www.environmentalevidence.org/guidelines/section-4) (8/23)
Problem 3: A lack of transparency/replicability in the review methods may mean that the review cannot be replicated (a central tenet of the scientific method!)
https://onlinelibrary.wiley.com/doi/10.1002/ece3.1722 (9/23)
https://onlinelibrary.wiley.com/doi/10.1002/ece3.1722 (9/23)
Solution 3: Be explicit, and make use of high-quality guidance and standards for review conduct (e.g. CEE Guidance, http://www.environmentalevidence.org/information-for-authors) and reporting (PRISMA, http://www.prisma-statement.org/ or ROSES, http://www.roses-reporting.com ). (10/23)
Problem 4: Selection bias (where included studies are not representative of the evidence base) and a lack of comprehensiveness (an inappropriate search method) can mean that reviews end up with the wrong evidence for the question at hand. https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-018-0599-2 (11/23)
Solution 4: Carefully design a search strategy with an info specialist; trial the search strategy (against a benchmark list); use multiple bibliographic databases/languages/sources of grey literature; publish search methods in an a priori protocol for peer-review. (12/23)
Problem 5: The exclusion of grey literature and failure to test for evidence of publication bias can result in incorrect or misleading conclusions. https://onlinelibrary.wiley.com/doi/abs/10.1002/jrsm.1433 (13/23)
Solution 5: Include attempts to find grey literature, including both 'file-drawer' (unpublished academic) research and organisational reports.
Test for possible evidence of publication bias.
https://www.sciencedirect.com/science/article/abs/pii/S0006320715300689 (14/23)
Test for possible evidence of publication bias.
https://www.sciencedirect.com/science/article/abs/pii/S0006320715300689 (14/23)
Problem 6: Traditional reviews often lack appropriate critical appraisal of included study validity, treating all evidence as equally valid - we know some research is more valid and we need to account for this in the synthesis. (15/23)
Solution 6: Carefully plan and trial a critical appraisal tool before starting the process in full, learning from existing robust critical appraisal tools.
https://methods.cochrane.org/bias/resources/rob-2-revised-cochrane-risk-bias-tool-randomized-trials (16/23)
https://methods.cochrane.org/bias/resources/rob-2-revised-cochrane-risk-bias-tool-randomized-trials (16/23)
Problem 7: Inappropriate synthesis (e.g. using vote-counting and inappropriate statistics) can negate all of the preceding systematic effort.
Vote-counting (tallying studies based on their statistical significance) ignores study validity and magnitude of effect sizes. (17/23)
Vote-counting (tallying studies based on their statistical significance) ignores study validity and magnitude of effect sizes. (17/23)
Solution 7: Select the synthesis method carefully based on the data analysed.
Vote-counting should never be used instead of meta-analysis.
Formal methods for narrative synthesis should be used to summarise and describe the evidence base.
https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.178.3100&rep=rep1&type=pdf (18/23)
Vote-counting should never be used instead of meta-analysis.
Formal methods for narrative synthesis should be used to summarise and describe the evidence base.
https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.178.3100&rep=rep1&type=pdf (18/23)
Problem 8: A lack of consistency and error checking (working individually) can introduce errors and biases if a single reviewer makes decisions without consensus.
https://onlinelibrary.wiley.com/doi/abs/10.1002/jrsm.1369 (19/23)
https://onlinelibrary.wiley.com/doi/abs/10.1002/jrsm.1369 (19/23)
Solution 8: Have two reviewers screen at least a subset of the evidence base to ensure consistency and shared understanding of the methods before proceeding. Ideally have two reviewers conduct all decisions separately and then consolidate. (20/23)
Summary: There is a lack of awareness and appreciation of the methods needed to ensure systematic reviews are as free from bias and as reliable as possible: demonstrated by recent, flawed, high-profile reviews. (21/23)
We call on review authors to conduct more rigorous reviews, on editors and peer-reviewers to gate-keep more strictly, and the community of methodologists to better support the broader research community. (22/23)
Only by working together can we build and maintain a strong system of rigorous, evidence-informed decision-making in conservation and environmental management. (23/23)
