The Time Cost of Rigorous Methods: One Team's Honest Accounting of RoB 2's Time Burden

Risk-of-bias assessment is one of those systematic review tasks everyone knows is time-consuming, but rarely gets measured. A new paper in the Journal of Clinical Epidemiology changes that. Crocker and colleagues tracked, in detail, how long their team spent applying the Cochrane risk-of-bias tool for randomized trials (RoB 2) while conducting a large systematic review and network meta-analysis of community-based interventions for older adults. The resulting numbers are worth a closer look, along with a few caveats about what they do and don't tell us.

The headline figure is striking: an average of 358 minutes, just shy of six hours, per study. But that average hides real variation. Studies in this review had anywhere from 1 to 33 "results of interest" (specific outcome-timepoint combinations), averaging 7.6 RoB2 assessments per study. When the authors modeled time as a function of complexity, they found a fixed cost of around 272 minutes per study regardless of outcomes, plus roughly 22 additional minutes for every additional result and 14 minutes for every additional linked report (a report being any separate publication tied to that trial: a results paper, a secondary analysis, a registry entry). Broken down per result rather than per study, the number comes to about 47 minutes, a more useful figure for reviewers trying to estimate their own timelines. Even reviewers who had already assessed 25 or more studies still averaged over five hours per study; experience helped, but it didn't come close to eliminating the burden.

It's worth being clear about what this study is and isn't. These numbers come from a single team, working on a single review of pragmatic, complex interventions. That underlying review is enormous by any standard: the final report runs 147 pages before appendices and 224 with the supplement, and it's exactly that scale (129 studies, 496 reports, 860 results of interest) that let the authors generate hard data on RoB 2's time cost in the first place. A smaller review might never produce enough data points to notice the pattern. But the authors aren't claiming this is a field-wide benchmark.

Some of their operational choices also show how much interpretation RoB 2 still requires, even with detailed guidance. For missing outcome data, the tool specifies that 95% availability is often sufficient for continuous outcomes, but for dichotomous outcomes it deliberately avoids a fixed number, tying the acceptable threshold to how common the event is instead. Crocker's team applied a flat 5%-missing rule to both types of outcomes, for consistency. That's a reasonable simplification, but it's also a clear example of reviewers translating general guidance into specific, study-level decision rules the tool doesn't spell out for every scenario. They describe similar judgment calls for deviations from intervention (domain 2), blinding of self-reported outcomes (domain 4), and selective reporting (domain 6).

There's a practical insight buried in the methods that's worth pulling out on its own. Not every RoB 2 domain has to be reassessed for every outcome. Domain 1 (the randomization process) is assessed once per study, since it concerns how participants were allocated, not what was measured. Domains 2 through 5 are assessed per result, because deviations from intervention, missing data, measurement bias, and selective reporting can all vary outcome by outcome. That split explains a lot of the paper's own findings: the time cost that scales with the number of outcomes is really being driven by four of the five domains, not all five equally. Reviewers planning their own timelines should budget accordingly, and pre-specifying decision rules, like Crocker's team did for missingness, before starting a review can meaningfully cut down on time spent re-litigating the same judgment call result after result.

For reviewers and guideline developers, this paper is less a warning about RoB 2 being impractical and more a case study in why transparent methods matter at every stage of evidence synthesis, not just in the final certainty rating. Knowing how an assessment was made is as important as knowing what it concluded.

References

Crocker TF, Lam N, Jordão M, Brundle C, Prescott M, Forster A, Ensor J, Gladman J, Clegg A. Risk-of-bias assessment using Cochrane's revised tool for randomized trials (RoB 2) was useful but challenging and resource-intensive: observations from a systematic review. Journal of Clinical Epidemiology. 2023 Sep;161:39-45. doi: 10.1016/j.jclinepi.2023.06.015. Epub 2023 Jun 24. PMID: 37364620.

https://www.jclinepi.com/article/S0895-4356(23)00163-4/fulltext 

Resulting paper from the team's RoB paper:

Crocker TF, Lam N, Ensor J, Jordão M, Bajpai R, Bond M, Forster A, Riley RD, Andre D, Brundle C, Ellwood A, Green J, Hale M, Morgan J, Patetsini E, Prescott M, Ramiz R, Todd O, Walford R, Gladman J, Clegg A. Community-based complex interventions to sustain independence in older people, stratified by frailty: a systematic review and network meta-analysis. Health Technol Assess. 2024 Aug;28(48):1-194. doi: 10.3310/HNRP2514. PMID: 39252602; PMCID: PMC11403382.

https://www.ncbi.nlm.nih.gov/books/NBK606947/

Risk of Bias Tools: 

https://www.riskofbias.info/

Do you Know Where Your Data are Coming From? AI-Assisted Paper Mills Are Flooding Health Research Databases

A new study in the Journal of Clinical Epidemiology puts numbers to what many researchers have suspected: the explosion of low-quality, formulaic manuscripts analyzing open-access health datasets may be systematic, AI-assisted, and accelerating.

Spick and colleagues analyzed publication trends to examine research derived from 36 major health and biomedical datasets indexed in the OpenAlex database between 2014 and 2025. They used a forecasting algorithm with three components -  past data, trend and seasonality consideration, and noise removal - to model expected publication rates based on pre-2023 trends. The authors then measured how far actual publication counts deviated from those projections after the widespread adoption of generative AI tools in late 2022.

Nine datasets stood out as likely targets of exploitation: NHANES, UK Biobank, the FDA Adverse Event Reporting System (FAERS), FinnGen, the Global Burden of Disease Study, MIMIC III/IV, the China Health and Retirement Longitudinal Study (CHARLS), CDC WONDER, and TriNetX. Together, these nine datasets had an estimated excess of roughly 11,577 publications above predicted trends in 2025 alone, a three-fold increase over their 2022 combined publication count.

Additionally, the authors found warning signs that point to AI use. Titles were becoming strikingly formulaic. For articles utilizing the CDC WONDER database, the word "trends" appeared in 69% of titles in 2025, up from 20% in 2022. For FAERS, the acronym itself appeared in the title of 55% of publications, up from 26%. 

Beyond quality concerns, papers following templates like "The association of [Predictor A] with [Outcome B] in [Population C] using [Open Access Dataset D]" introduce false discoveries into the literature (i.e., present statistically significant findings from large datasets without a corrected p-value), dilute genuine findings, and erode confidence in otherwise valuable data resources. For example, FAERS is a meaningful tool for drug safety monitoring, but only if the signal is not buried under thousands of (potentially) poorly conducted association analyses. The problem extends to AI language models themselves, which are trained on published literature: corrupted inputs will propagate corrupted outputs.

This matters enormously for systematic review teams. When a body of literature is flooded with papers sharing the same template, the same analytical shortcuts, and potentially the same selectively reported findings, the evidence base being synthesized becomes harder to trust. These papers are appearing in indexed journals and clearing peer review, meaning your literature search will likely pull these articles alongside other non-AI written articles. The authors estimate that excess publications represented roughly 50% of 2025 output for the nine most affected datasets.

The practical response is a renewed commitment to critical appraisal at every stage of the review process. 

Systematic reviewers should consider the plausibility of findings, the transparency of methods, and whether outcomes appear to have been pre-specified. Formulaic titles, an absence of pre-registration, implausible effect sizes, and a lack of false discovery correction are all worth treating as warning signs during screening and full-text review.

Searches that pull from datasets like NHANES, UK Biobank, or the Global Burden of Disease Study are now operating in a considerably noisier environment, and careful critical appraisal, with attention to formulaic methodology, implausible findings, and selective outcome reporting, is more important than ever.

References

Spick M, Onoja A, Harrison C, Stender S, Byrne J, Geifman N. Quantifying new threats to health and biomedical literature integrity from rapidly scaled publications and problematic research. J Clin Epidemiol. 2026 May;193:112203. doi: 10.1016/j.jclinepi.2026.112203. Epub 2026 Feb 23. PMID: 41740900; PMCID: PMC13178217.

https://www.jclinepi.com/article/S0895-4356(26)00078-8/fulltext

GRADE Takes on the Planet: Introducing GRADE Guidance 46, Integrating Planetary Health in Health Guidelines

For decades, health guideline developers using the GRADE framework have worked to ensure that recommendations reflect the best available evidence on patient outcomes, values, costs, and equity. A major new addition to that framework asks guideline panels to consider something broader: the health of the planet itself. In a paper recently published in Annals of Internal Medicine, Piggott, Saadat, Herrmann, and colleagues from the GRADE Planetary Health Project Group present the first official GRADE guidance for integrating planetary health into health guideline development.

Health systems are not passive bystanders in the planetary health crisis; they actively contribute to it. Health care accounts for an estimated 5.2% of global greenhouse gas (GHG) emissions, and its impacts on freshwater use may be even more significant relative to national benchmarks. Until now, however, health guidelines have rarely systematically considered these environmental consequences. A scoping review cited in the paper found just four health guidelines that had addressed planetary health dimensions at all, and even those did so with limited methodological rigor. This guidance is designed to close that gap.

The project group, working from January 2023 through May 2025, developed 7 domains of guidance through iterative case study analyses, expert workshops, and a two-round global Delphi consensus process involving 85 international experts. The result is a structured framework that maps onto the familiar steps of guideline development, from scoping and question formulation through to evidence synthesis and the evidence-to-decision (EtD) framework. 

The most concrete addition is a new criterion within the GRADE evidence-to-decision (EtD) framework, which is the table guideline panels use at the final stage of a review to move from evidence to recommendations. Alongside existing criteria like "balance of effects," "costs," and "equity," panels now have a dedicated row for planetary boundaries. For each boundary relevant to their guideline topic, panels work through a set of signaling questions drawn directly from environmental science, then record a summary judgment ranging from "large desirable" to "large undesirable." 

Four case studies bring the guidance to life, ranging from the choice between anesthetic gases (sevoflurane versus desflurane, with dramatically different GHG profiles) to dietary recommendations around red meat consumption. In the red meat case, applying the planetary health lens would have reversed the direction of a previously published conditional recommendation, shifting it from continuing current intake to reducing it, based on the intervention's large undesirable effects on climate, land use, freshwater, and nutrient cycles. This underscores that planetary health evidence can and should be considered when making recommendations.

This guidance is meant to fit into the existing GRADE framework, ensuring a seamless adoption. The framework provides signalling questions for each planetary boundary to help panels make informed judgments without requiring in-house environmental science experts. Including a planetary health expert on the panel is suggested but optional and the guidance is clear that not every guideline needs to address every planetary boundary. The goal is transparency and consideration.

As the authors note, the crises facing our planet warrant the best effort guideline panels can offer, and this framework gives them the tools to start.

References

Thomas Piggott, Pakeezah Saadat, Alina Herrmann, et al. Integrating Planetary Health in Health Guidelines (GRADE Guidance 46). Ann Intern Med 2026;179:874-884. [Epub 12 May 2026]. doi:10.7326/ANNALS-25-04761

https://www.acpjournals.org/doi/10.7326/ANNALS-25-04761 

Extra Resources

GRADE guidance articles in the Journal of Clinical Epidemiology (JCE) series: https://www.cochrane.org/learn/courses-and-resources/cochrane-methodology/grade/jce-series

GRADE Book is the most comprehensive and up-to-date resource on the GRADE approach. It will replace the previous GRADE Handbook by 2026, with updated content being progressively released. https://book.gradepro.org/

GRADE working group: https://www.gradeworkinggroup.org/

Is Your "Gold Standard" Truly Gold? A Living Review Exposes 67 Flaws in Systematic Reviews

 Systematic reviews and meta-analysis are often referred to as the ‘gold standard’ of evidence, often trusted to guide clinical guidelines and major healthcare decisions. With so much weighing on the accuracy of systemic review, what would happen if the review itself had cracks? 
A living systematic review published in the Journal of Clinical Epidemiology critically reviewed 485 articles published since 2000, and identified 67 problems that can jeopardize the reliability and validity of these evidence syntheses.

Lesley Uttley and her team categorized these 67 problems into four main areas:

1. Comprehensive: Is all relevant evidence included?

2. Rigorous: Were appropriate methods used

3. Transparent: Can it be reproduced?

4. Objective: Is the process fair and unbiased?

—

Lesley’s reviews should be looked at as a self-check to ensure the validity of your systematic review isn’t knowingly compromised. Before submission, ask: 

• Have we involved an information specialist? 

• Have we pre-specified and justified all analyses? 

• Have we explicitly managed and declared conflicts of interest? 

• Have we considered equity? 

For readers and users of systematic reviews, try to be more skeptical and critically appraise the work before you quote. For a more involved approach:

Ask Critical Questions Based on the Four Domains:

• Comprehensiveness: Does the search strategy seem thorough? Did they look in the right places (databases, grey literature, overly stringent inclusion criteria leading to multiple updates later on) for all relevant evidence?

• Rigor: Have they properly handled study synthesis, quality and heterogeneity? The Assessing the Methodological Quality of Systematic Reviews (AMSTAR) checklist is helpful here.

• Transparency: Can you trace their steps? Is there a protocol you can consult? Is it unclear why certain studies were excluded?

• Objectivity: Are there involved experts on the team that share knowledge throughout the process? Are declared conflicts of interest properly managed?

Change Your Appraisal Mindset:

The most reliable review isn't necessarily the one with the most exciting finding, it's the one whose process you can best understand and trust. Use the four domains as your critical appraisal lens to identify potential weaknesses before applying the results to practice.

References

Uttley, Lesley et al. “The Problems with Systematic Reviews: A Living Systematic Review.” Journal of Clinical Epidemiology 156 (2023): 30–41. Manuscript available at the publisher's website here.

https://www.jclinepi.com/article/S0895-4356(23)00011-2/fulltext 

Uttley, L., Weng, Y., & Falzon, L. (2025). Yet another problem with systematic reviews: a living review update. Journal of Clinical Epidemiology 177: Article 111608. https://doi.org/10.1016/j.jclinepi.2024.111608

Manuscript available at the publisher's website here.

https://www.jclinepi.com/article/S0895-4356(24)00364-0/fulltext

Fall 2025 Scholars Propose Innovative Projects in Cancer Care and Global GRADE Dissemination

Recently, the U.S. GRADE Network and Evidence Foundation had the privilege of welcoming three new scholars to the fall 2025 virtual GRADE Guideline Development Workshop. Scholars hailed from around the world, from the U.S. to Pakistan and Germany. 

Alexander Brooks, a Ph.D. candidate in exercise science at the University of South Carolina, presented a systematic review project in development to assess the long-term effectiveness of exercise interventions in cancer survivors. "Despite the growing number of clinical trials and guidelines in this area," says Brooks, "few systematic reviews have applied structured approaches to assess the certainty of the evidence underpinning these collective recommendations." Brooks' project directly addresses this gap in the literature while examining the trajectory of health and fitness markers in this population in the short- and long-term period after cancer treatment.

Attending the GRADE workshop "strengthened my ability to critically evaluate research and better understand how evidence is synthesized to inform recommendations," said Brooks. "The hands-on exercises were particularly valuable, and I found the workshop directly applicable to my own work."

Dr. Anna-Sophie Strauss, a resident physician at University Hospital Ulm in Germany and a Cochrane Urology review author, discussed her current review of the diagnostic test accuracy of prostate-specific membrane antigen (PSMA) imaging for the staging of primary prostate cancer. The aim of this review is to inform evidence-based guidelines for this testing strategy, ultimately "supporting more informed and unbiased decision-making in urological healthcare."

"Taking part in the GRADE Guideline Development Workshop improved my understanding of how structured evidence appraisal can be used to create transparent, patient-centered recommendations," said Strauss. "I also appreciated the opportunity to exchange global perspectives, which demonstrated how rigorous methods such as GRADE can be adapted to different clinical and health system contexts." 

Finally, Dr. Muhammad Tayyab Qureshi presented from Pakistan. A House Officer in the Pakistan Air Force Hospital Islamabad, Dr. Qureshi discussed the need for the dissemination of locally relevant guidelines in Lower- and Middle-Income Countries such as Pakistan. These efforts, he believes, will strengthen "health systems through practical, evidence-informed solutions that lead to measurable improvements in health outcomes and resource use."

The training provided in the three-day GRADE workshop will allow the scholars to apply the GRADE framework to the certainty and evidence assessment and formulation of recommendations within their specific projects. 

Interested in becoming a scholar? Applications to join us for the spring 2026 workshop in Kansas City, Missouri, May 18-20, 2026, close Saturday, January 31. See application details at evidencefoundation.org/scholarships.html.

2021 Evidence Foundation Scholar Receives Health Equity Travel Scholarship

The Evidence Foundation is proud to announce that spring 2021 GRADE Workshop Scholar Ifeoluwa Babatunde, Pharm.D, MS, received the 2025 Encoding Health Equity Summit Travel Scholarship for Patients and Emerging Leaders, an opportunity made possible by Doris Duke Foundation and the Council of Medical Specialty Societies (CMSS). 

About her experience at the summit, Babatunde shared,

"The Encoding Equity conference was an enriching experience for me. I found it quite inspiring to hear about the diverse initiatives and research projects geared towards achieving health equity. Hearing from biomedical researchers, clinicians, public health providers, guideline developers, and medical societies truly showcased the depth of work in this field.

"My personal takeaway from the conference was on the power of collaboration. Successfully achieving the central goal of health equity integration would require all stakeholders, from providers & researchers to members of the very communities we aim to impact, actively sharing their perspectives. By engaging in consistent mutual learning, we can inspire innovation and translate our findings to actionable change across the entire medical ecosystem.

"As someone in guideline development, I particularly resonated with the work being done on pioneering a 'Population Conscious Analysis' approach to handling race and ethnicity as a variable in biomedical research. This innovation has me very optimistic about the positive impact it will have on new and existing quality appraisal tools being used in evidence synthesis.

"Lastly, I enjoyed engaging in discussions with poster presenters working on projects closely aligned with that of my team. Specifically, those focused on creating a race-conscious framework for for guideline development. These discussions reinforced the significant impact of the work we are engaged in."

Open the doors to opportunity:

Interested in contributing to the future of evidence-based medicine? Become an Evidence Foundation Scholar and attend our upcoming virtual fall workshop (October 29-31, 2025) for free. Application close August 31, 2025. See evidencefoundation.org/scholarships.html 

for application details.

Latest Evidence Foundation Scholar Tackles Project to Create New Quality Assessment Tool for Prevalence Studies

Prevalence studies inform understanding of an individual's baseline risk for a medical or health condition as well as the burden of the disease and the characteristics of a given target population. Meanwhile, estimates of the the absolute risk, taken from prevalence data, informs the development of evidence-based guidelines. But how can one tell if the data on prevalence are trustworthy or seriously biased?

There currently exists no widely agreed-upon tool to assess the risk of bias within studies of prevalence analogous to the Cochrane RoB 2.0 tool for randomized controlled trials or QUADAS-2 for studies of diagnostic test accuracy. Additionally, the assessment of a prevalence study comes with its own challenges, as it requires assessment of population selection, condition treatment, and analysis.

Dr. Ali Choaib, a post-doctoral research fellow in the Evidence-based Practice and Impact Center at the University of Kansas Medical Center, discussed this gap and his proposed project to address it as part of his presentation as the spring 2025 Evidence Foundation scholarship recipient.

At the GRADE Guideline Development Workshop held in Arlington, Virginia, last month, Dr. Choaib presented his work in systematically reviewing and thematically analyzing all existing tools to assess risk of bias in prevalence studies. Choaib and his team identified four major conceptual domains shared across the existing tools: study population, sampling, and setting; case/condition definition and measurement; statistical analysis; and other (e.g., conflicts of interest and missing data).

Next, Choaib's team will work to develop and receive iterative feedback on a new tool that assesses all critical domains while taking care not to include items or questions that are not specifically related to bias, such as those assessing reporting quality.

About his experience of the workshop, Choaib said, 

"The workshop was a perfect opportunity to learn, network, and collaborate with experts. Topics were dynamic and sessions tailored to individuals at all stages of the guideline development process from researchers to admins. I’m honored to have received the scholarship and to have contributed to discussions that expanded my knowledge and understanding of the guideline development process overall and with regards to diagnostic test accuracy specifically."

Choaib has additionally received the Encoding Health Equity Summit 2025 Travel Scholarship Program for Patients and Emerging Leaders. This scholarship, which is made possible with support from the Doris Duke Foundation, the Council of Medical Specialty Societies (CMSS), allowed him to attend the 2025 Encoding Health Equity Summit. This year's theme was "From Evidence to Impact."

"Encoding Health Equity Summit was an eye opening  and career influencing experience. Being a scholar and having the opportunity to contribute to the discussion on a novel and noble approach to health equity at a time when it’s needed. The meeting introduced us to other other investigators, administrators, journal editors, and society leaders broadened my perspective and opening the way for new opportunities and collaboration. This meeting has also connected ideas and work from basic science to implementation. It was inspiring to see so many passionate professionals working to advance equity and improve the future of health."

Interested in contributing to the future of evidence-based medicine? Become an Evidence Foundation Scholar and attend our upcoming virtual fall workshop (October 29-31, 2025) for free. Application close August 31, 2025. See evidencefoundation.org/scholarships.html for application details.

Learn more about Ali's project and other ways to get involved with GRADE at our upcoming free webinar, Monday, November 17th. Register to attend at this link.