A living systematic review published in the Journal of Clinical Epidemiology critically reviewed 485 articles published since 2000, and identified 67 problems that can jeopardize the reliability and validity of these evidence syntheses.
Lesley Uttley and her team categorized these 67 problems into four main areas:
Comprehensive: Is all relevant evidence included?
Rigorous: Were appropriate methods used
Transparent: Can it be reproduced?
Objective: Is the process fair and unbiased?
—
Lesley’s reviews should be looked at as a self-check to ensure the validity of your systematic review isn’t knowingly compromised. Before submission, ask:
Have we involved an information specialist?
Have we pre-specified and justified all analyses?
Have we explicitly managed and declared conflicts of interest?
Have we considered equity?
%20INFO.png)
For readers and users of systematic reviews, try to be more skeptical and critically appraise the work before you quote. For a more involved approach:
Ask Critical Questions Based on the Four Domains:
Comprehensiveness: Does the search strategy seem thorough? Did they look in the right places (databases, grey literature, overly stringent inclusion criteria leading to multiple updates later on) for all relevant evidence?
Rigor: Have they properly handled study synthesis, quality and heterogeneity? The Assessing the Methodological Quality of Systematic Reviews (AMSTAR) checklist is helpful here.
Transparency: Can you trace their steps? Is there a protocol you can consult? Is it unclear why certain studies were excluded?
Objectivity: Are there involved experts on the team that share knowledge throughout the process? Are declared conflicts of interest properly managed?
Change Your Appraisal Mindset:
The most reliable review isn't necessarily the one with the most exciting finding, it's the one whose process you can best understand and trust. Use the four domains as your critical appraisal lens to identify potential weaknesses before applying the results to practice.
References
Uttley, Lesley et al. “The Problems with Systematic Reviews: A Living Systematic Review.” Journal of Clinical Epidemiology 156 (2023): 30–41. Manuscript available at the publisher's website here.
https://www.jclinepi.com/article/S0895-4356(23)00011-2/fulltext
Uttley, L., Weng, Y., & Falzon, L. (2025). Yet another problem with systematic reviews: a living review update. Journal of Clinical Epidemiology 177: Article 111608. https://doi.org/10.1016/j.jclinepi.2024.111608
Manuscript available at the publisher's website here.
https://www.jclinepi.com/article/S0895-4356(24)00364-0/fulltext
