Is Your "Gold Standard" Truly Gold? A Living Review Exposes 67 Flaws in Systematic Reviews

 Systematic reviews and meta-analysis are often referred to as the ‘gold standard’ of evidence, often trusted to guide clinical guidelines and major healthcare decisions. With so much weighing on the accuracy of systemic review, what would happen if the review itself had cracks? 
A living systematic review published in the Journal of Clinical Epidemiology critically reviewed 485 articles published since 2000, and identified 67 problems that can jeopardize the reliability and validity of these evidence syntheses.

Lesley Uttley and her team categorized these 67 problems into four main areas:

1. Comprehensive: Is all relevant evidence included?

2. Rigorous: Were appropriate methods used

3. Transparent: Can it be reproduced?

4. Objective: Is the process fair and unbiased?

—

Lesley’s reviews should be looked at as a self-check to ensure the validity of your systematic review isn’t knowingly compromised. Before submission, ask: 

• Have we involved an information specialist? 

• Have we pre-specified and justified all analyses? 

• Have we explicitly managed and declared conflicts of interest? 

• Have we considered equity? 

For readers and users of systematic reviews, try to be more skeptical and critically appraise the work before you quote. For a more involved approach:

Ask Critical Questions Based on the Four Domains:

• Comprehensiveness: Does the search strategy seem thorough? Did they look in the right places (databases, grey literature, overly stringent inclusion criteria leading to multiple updates later on) for all relevant evidence?

• Rigor: Have they properly handled study synthesis, quality and heterogeneity? The Assessing the Methodological Quality of Systematic Reviews (AMSTAR) checklist is helpful here.

• Transparency: Can you trace their steps? Is there a protocol you can consult? Is it unclear why certain studies were excluded?

• Objectivity: Are there involved experts on the team that share knowledge throughout the process? Are declared conflicts of interest properly managed?

Change Your Appraisal Mindset:

The most reliable review isn't necessarily the one with the most exciting finding, it's the one whose process you can best understand and trust. Use the four domains as your critical appraisal lens to identify potential weaknesses before applying the results to practice.

References

Uttley, Lesley et al. “The Problems with Systematic Reviews: A Living Systematic Review.” Journal of Clinical Epidemiology 156 (2023): 30–41. Manuscript available at the publisher's website here.

https://www.jclinepi.com/article/S0895-4356(23)00011-2/fulltext 

Uttley, L., Weng, Y., & Falzon, L. (2025). Yet another problem with systematic reviews: a living review update. Journal of Clinical Epidemiology 177: Article 111608. https://doi.org/10.1016/j.jclinepi.2024.111608

Manuscript available at the publisher's website here.

https://www.jclinepi.com/article/S0895-4356(24)00364-0/fulltext

Fall 2025 Scholars Propose Innovative Projects in Cancer Care and Global GRADE Dissemination

Recently, the U.S. GRADE Network and Evidence Foundation had the privilege of welcoming three new scholars to the fall 2025 virtual GRADE Guideline Development Workshop. Scholars hailed from around the world, from the U.S. to Pakistan and Germany. 

Alexander Brooks, a Ph.D. candidate in exercise science at the University of South Carolina, presented a systematic review project in development to assess the long-term effectiveness of exercise interventions in cancer survivors. "Despite the growing number of clinical trials and guidelines in this area," says Brooks, "few systematic reviews have applied structured approaches to assess the certainty of the evidence underpinning these collective recommendations." Brooks' project directly addresses this gap in the literature while examining the trajectory of health and fitness markers in this population in the short- and long-term period after cancer treatment.

Attending the GRADE workshop "strengthened my ability to critically evaluate research and better understand how evidence is synthesized to inform recommendations," said Brooks. "The hands-on exercises were particularly valuable, and I found the workshop directly applicable to my own work."

Dr. Anna-Sophie Strauss, a resident physician at University Hospital Ulm in Germany and a Cochrane Urology review author, discussed her current review of the diagnostic test accuracy of prostate-specific membrane antigen (PSMA) imaging for the staging of primary prostate cancer. The aim of this review is to inform evidence-based guidelines for this testing strategy, ultimately "supporting more informed and unbiased decision-making in urological healthcare."

"Taking part in the GRADE Guideline Development Workshop improved my understanding of how structured evidence appraisal can be used to create transparent, patient-centered recommendations," said Strauss. "I also appreciated the opportunity to exchange global perspectives, which demonstrated how rigorous methods such as GRADE can be adapted to different clinical and health system contexts." 

Finally, Dr. Muhammad Tayyab Qureshi presented from Pakistan. A House Officer in the Pakistan Air Force Hospital Islamabad, Dr. Qureshi discussed the need for the dissemination of locally relevant guidelines in Lower- and Middle-Income Countries such as Pakistan. These efforts, he believes, will strengthen "health systems through practical, evidence-informed solutions that lead to measurable improvements in health outcomes and resource use."

The training provided in the three-day GRADE workshop will allow the scholars to apply the GRADE framework to the certainty and evidence assessment and formulation of recommendations within their specific projects. 

Interested in becoming a scholar? Applications to join us for the spring 2026 workshop in Kansas City, Missouri, May 18-20, 2026, close Saturday, January 31. See application details at evidencefoundation.org/scholarships.html.