Is Your "Gold Standard" Truly Gold? A Living Review Exposes 67 Flaws in Systematic Reviews

 Systematic reviews and meta-analysis are often referred to as the ‘gold standard’ of evidence, often trusted to guide clinical guidelines and major healthcare decisions. With so much weighing on the accuracy of systemic review, what would happen if the review itself had cracks? 
A living systematic review published in the Journal of Clinical Epidemiology critically reviewed 485 articles published since 2000, and identified 67 problems that can jeopardize the reliability and validity of these evidence syntheses.

Lesley Uttley and her team categorized these 67 problems into four main areas:

1. Comprehensive: Is all relevant evidence included?

2. Rigorous: Were appropriate methods used

3. Transparent: Can it be reproduced?

4. Objective: Is the process fair and unbiased?

—

Lesley’s reviews should be looked at as a self-check to ensure the validity of your systematic review isn’t knowingly compromised. Before submission, ask: 

• Have we involved an information specialist? 

• Have we pre-specified and justified all analyses? 

• Have we explicitly managed and declared conflicts of interest? 

• Have we considered equity? 

For readers and users of systematic reviews, try to be more skeptical and critically appraise the work before you quote. For a more involved approach:

Ask Critical Questions Based on the Four Domains:

• Comprehensiveness: Does the search strategy seem thorough? Did they look in the right places (databases, grey literature, overly stringent inclusion criteria leading to multiple updates later on) for all relevant evidence?

• Rigor: Have they properly handled study synthesis, quality and heterogeneity? The Assessing the Methodological Quality of Systematic Reviews (AMSTAR) checklist is helpful here.

• Transparency: Can you trace their steps? Is there a protocol you can consult? Is it unclear why certain studies were excluded?

• Objectivity: Are there involved experts on the team that share knowledge throughout the process? Are declared conflicts of interest properly managed?

Change Your Appraisal Mindset:

The most reliable review isn't necessarily the one with the most exciting finding, it's the one whose process you can best understand and trust. Use the four domains as your critical appraisal lens to identify potential weaknesses before applying the results to practice.

References

Uttley, Lesley et al. “The Problems with Systematic Reviews: A Living Systematic Review.” Journal of Clinical Epidemiology 156 (2023): 30–41. Manuscript available at the publisher's website here.

https://www.jclinepi.com/article/S0895-4356(23)00011-2/fulltext 

Uttley, L., Weng, Y., & Falzon, L. (2025). Yet another problem with systematic reviews: a living review update. Journal of Clinical Epidemiology 177: Article 111608. https://doi.org/10.1016/j.jclinepi.2024.111608

Manuscript available at the publisher's website here.

https://www.jclinepi.com/article/S0895-4356(24)00364-0/fulltext

Fall 2025 Scholars Propose Innovative Projects in Cancer Care and Global GRADE Dissemination

Recently, the U.S. GRADE Network and Evidence Foundation had the privilege of welcoming three new scholars to the fall 2025 virtual GRADE Guideline Development Workshop. Scholars hailed from around the world, from the U.S. to Pakistan and Germany. 

Alexander Brooks, a Ph.D. candidate in exercise science at the University of South Carolina, presented a systematic review project in development to assess the long-term effectiveness of exercise interventions in cancer survivors. "Despite the growing number of clinical trials and guidelines in this area," says Brooks, "few systematic reviews have applied structured approaches to assess the certainty of the evidence underpinning these collective recommendations." Brooks' project directly addresses this gap in the literature while examining the trajectory of health and fitness markers in this population in the short- and long-term period after cancer treatment.

Attending the GRADE workshop "strengthened my ability to critically evaluate research and better understand how evidence is synthesized to inform recommendations," said Brooks. "The hands-on exercises were particularly valuable, and I found the workshop directly applicable to my own work."

Dr. Anna-Sophie Strauss, a resident physician at University Hospital Ulm in Germany and a Cochrane Urology review author, discussed her current review of the diagnostic test accuracy of prostate-specific membrane antigen (PSMA) imaging for the staging of primary prostate cancer. The aim of this review is to inform evidence-based guidelines for this testing strategy, ultimately "supporting more informed and unbiased decision-making in urological healthcare."

"Taking part in the GRADE Guideline Development Workshop improved my understanding of how structured evidence appraisal can be used to create transparent, patient-centered recommendations," said Strauss. "I also appreciated the opportunity to exchange global perspectives, which demonstrated how rigorous methods such as GRADE can be adapted to different clinical and health system contexts." 

Finally, Dr. Muhammad Tayyab Qureshi presented from Pakistan. A House Officer in the Pakistan Air Force Hospital Islamabad, Dr. Qureshi discussed the need for the dissemination of locally relevant guidelines in Lower- and Middle-Income Countries such as Pakistan. These efforts, he believes, will strengthen "health systems through practical, evidence-informed solutions that lead to measurable improvements in health outcomes and resource use."

The training provided in the three-day GRADE workshop will allow the scholars to apply the GRADE framework to the certainty and evidence assessment and formulation of recommendations within their specific projects. 

Interested in becoming a scholar? Applications to join us for the spring 2026 workshop in Kansas City, Missouri, May 18-20, 2026, close Saturday, January 31. See application details at evidencefoundation.org/scholarships.html.

2021 Evidence Foundation Scholar Receives Health Equity Travel Scholarship

The Evidence Foundation is proud to announce that spring 2021 GRADE Workshop Scholar Ifeoluwa Babatunde, Pharm.D, MS, received the 2025 Encoding Health Equity Summit Travel Scholarship for Patients and Emerging Leaders, an opportunity made possible by Doris Duke Foundation and the Council of Medical Specialty Societies (CMSS). 

About her experience at the summit, Babatunde shared,

"The Encoding Equity conference was an enriching experience for me. I found it quite inspiring to hear about the diverse initiatives and research projects geared towards achieving health equity. Hearing from biomedical researchers, clinicians, public health providers, guideline developers, and medical societies truly showcased the depth of work in this field.

"My personal takeaway from the conference was on the power of collaboration. Successfully achieving the central goal of health equity integration would require all stakeholders, from providers & researchers to members of the very communities we aim to impact, actively sharing their perspectives. By engaging in consistent mutual learning, we can inspire innovation and translate our findings to actionable change across the entire medical ecosystem.

"As someone in guideline development, I particularly resonated with the work being done on pioneering a 'Population Conscious Analysis' approach to handling race and ethnicity as a variable in biomedical research. This innovation has me very optimistic about the positive impact it will have on new and existing quality appraisal tools being used in evidence synthesis.

"Lastly, I enjoyed engaging in discussions with poster presenters working on projects closely aligned with that of my team. Specifically, those focused on creating a race-conscious framework for for guideline development. These discussions reinforced the significant impact of the work we are engaged in."

Open the doors to opportunity:

Interested in contributing to the future of evidence-based medicine? Become an Evidence Foundation Scholar and attend our upcoming virtual fall workshop (October 29-31, 2025) for free. Application close August 31, 2025. See evidencefoundation.org/scholarships.html 

for application details.

Latest Evidence Foundation Scholar Tackles Project to Create New Quality Assessment Tool for Prevalence Studies

Prevalence studies inform understanding of an individual's baseline risk for a medical or health condition as well as the burden of the disease and the characteristics of a given target population. Meanwhile, estimates of the the absolute risk, taken from prevalence data, informs the development of evidence-based guidelines. But how can one tell if the data on prevalence are trustworthy or seriously biased?

There currently exists no widely agreed-upon tool to assess the risk of bias within studies of prevalence analogous to the Cochrane RoB 2.0 tool for randomized controlled trials or QUADAS-2 for studies of diagnostic test accuracy. Additionally, the assessment of a prevalence study comes with its own challenges, as it requires assessment of population selection, condition treatment, and analysis.

Dr. Ali Choaib, a post-doctoral research fellow in the Evidence-based Practice and Impact Center at the University of Kansas Medical Center, discussed this gap and his proposed project to address it as part of his presentation as the spring 2025 Evidence Foundation scholarship recipient.

At the GRADE Guideline Development Workshop held in Arlington, Virginia, last month, Dr. Choaib presented his work in systematically reviewing and thematically analyzing all existing tools to assess risk of bias in prevalence studies. Choaib and his team identified four major conceptual domains shared across the existing tools: study population, sampling, and setting; case/condition definition and measurement; statistical analysis; and other (e.g., conflicts of interest and missing data).

Next, Choaib's team will work to develop and receive iterative feedback on a new tool that assesses all critical domains while taking care not to include items or questions that are not specifically related to bias, such as those assessing reporting quality.

About his experience of the workshop, Choaib said, 

"The workshop was a perfect opportunity to learn, network, and collaborate with experts. Topics were dynamic and sessions tailored to individuals at all stages of the guideline development process from researchers to admins. I’m honored to have received the scholarship and to have contributed to discussions that expanded my knowledge and understanding of the guideline development process overall and with regards to diagnostic test accuracy specifically."

Choaib has additionally received the Encoding Health Equity Summit 2025 Travel Scholarship Program for Patients and Emerging Leaders. This scholarship, which is made possible with support from the Doris Duke Foundation, the Council of Medical Specialty Societies (CMSS), allowed him to attend the 2025 Encoding Health Equity Summit. This year's theme was "From Evidence to Impact."

"Encoding Health Equity Summit was an eye opening  and career influencing experience. Being a scholar and having the opportunity to contribute to the discussion on a novel and noble approach to health equity at a time when it’s needed. The meeting introduced us to other other investigators, administrators, journal editors, and society leaders broadened my perspective and opening the way for new opportunities and collaboration. This meeting has also connected ideas and work from basic science to implementation. It was inspiring to see so many passionate professionals working to advance equity and improve the future of health."

Interested in contributing to the future of evidence-based medicine? Become an Evidence Foundation Scholar and attend our upcoming virtual fall workshop (October 29-31, 2025) for free. Application close August 31, 2025. See evidencefoundation.org/scholarships.html for application details.

Learn more about Ali's project and other ways to get involved with GRADE at our upcoming free webinar, Monday, November 17th. Register to attend at this link.

USGN Congratulates Fall 2024 Scholar Michelle Hannum

The U.S. GRADE Network, in partnership with the Evidence Foundation, is proud to be able to offer scholarships for deserving applicants to attend our semi-annual GRADE Guideline Development Workshop with a waived registration fee. 

The scholarship allows workshop participants of all disciplines, fields, career stages, and backgrounds to learn about the GRADE framework for assessing certainty of evidence and formulating guideline recommendations. 

The fall 2024 scholarship recipient, Michelle Hannum, was chosen due to her proposed project on "The Promising Future of Evidence-Based Clinical Pathways in Reducing Healthcare Bias," undertaken as part of her work as a Guideline Development Project Manager at the Value Institute at the Medical University of South Carolina (MUSC) Health. The role of clinical pathways is to provide the right information (guideline recommendations) to the right people (electronic health record end-users) at the right time (based on patient data) in the right formats and using the right channels. Hannum presented data on pathway utilization at MUSC Health, which demonstrated that Heart Failure pathway utilization was approximately 16-18% and that utilization was equal across patient race/ethnicity and sex among this population. Furthermore, pathway utilization coincided with lower mortality rates, fewer 30-day readmissions and ICU days, and overall lower cost. 

When asked about her experience as a scholar, Hannum stated,

"The workshop does an outstanding job meeting individuals with where they are at in their GRADE journey, whether it be for career roles or personal goals. The information was presented in an engaging manner with plenty of opportunities for discussion and question-answering. The workshop provides an excellent foundation for understanding GRADE and its application."

Interested in being a scholar at the Spring 2025 GRADE Workshop in Washington, DC, May 7-9, 2025? Applications are due January 31. See details at www.evidencefoundation.org/scholarships/html.

EF Scholar Success Story: Nirjhar Ruth Ghosh

 New publication alert!

Nirjhar Ruth Ghosh, who attended the winter 2022 Systematic Review workshop as an Evidence Foundation scholar, recently published the results of her hard work in The Journal of Nutrition! The project, "Evidence-Based Practice Competencies Among Nutrition Professionals and Students: A Systematic Review" was originally presented by Ghosh to her fellow attendees of the virtual workshop. The results of the systematic review are now able to be read in detail at this link. 

In an accompanying editorial lauding the publication, Francene M. Steinberg wrote that "Ghosh et al. have provided a foundation for further consideration of steps to advance interprofessional competencies in EBP to optimize clinical nutrition decision making and patient care outcomes. Improved clarity about core competencies, innovative EBP curriculum and pedagogic approaches, and more rigorous research evaluations of EBP application and outcomes are all necessary components of the path forward."

Congratulations, Ruth!

Interested in being our next scholar success story? Applications for scholarships to attend our fully virtual GRADE Guideline Development Workshop are now open (deadline: August 31). Learn more at https://evidencefoundation.org/scholarships. 

=

Three's a Crowd: How to Deal with More than Two Arms in a Meta-Analysis

It is not uncommon to come across the following scenario: when conducting a meta-analysis between two arms (e.g., an active therapy vs. a placebo), the meta-analyst includes a study that actually included two active arms (e.g., two different doses of the same experimental drug vs. placebo, two different routes of administration, etc.) Let's say that both of these arms were relevant to the clinical question. How should meta-analysis be undertaken in this case? A new tutorial article published in Cochrane Evidence Synthesis and Methods gives a primer on how to approach this common conundrum. 

Including the study twice in the forest plot – for instance, with one dose versus placebo and the other versus the same placebo group – is statistically problematic. It leads to a "unit of analysis" error by essentially "double-counting" the participants in the control group and violating the assumption that every individual participant is only counted twice. (Aside: this is also a common error in meta-analyses combining multiple similar outcomes – e.g., including the handgrip strength of both the dominant and non-dominant hand in the same forest plot – and risks committing the same violation unless advanced multi-level statistical techniques are used to account for this). 

This leaves two basic options for including the data from more than two study arms into the same forest plot: combining interventions that are similar, and splitting the control group in half. For instance, if the three groups in question are as such (assuming a dichotomous outcome):

• Experimental group A: 50 participants, 45 of whom had the event.
• Experimental group B: 50 participants, 41 of whom had the event.
• Control group: 50 participants, 22 of whom had the event.

These two experimental groups can either be combined (100 participants, 89 of whom had the event) and compared to the control group as-is, or they can be split out and compared (on separate lines of the forest plot) to half of the control group on each line:

• Experimental group A (45 out of 50) versus control (11 out of 25)
• Experimental group B (41 out of 50) versus control (11 out of 25).

Both approaches will yield very similar pooled results.

In the case of a continuous outcome, the same general approaches can be applied. However, if pooling two or more arms together, a pooled mean and SD will need to be calculated using the following equations from the Cochrane Handbook:

Reference: Axon, E., Dwan, K., & Richardson, R. (2023) Multiarm studies and how to handle them in a meta-analysis: A tutorial. Cochrane Evidence Synthesis and Methods. Available at publisher's website here.

Review of Time-to-Event Outcomes Analyzed in 50 Systematic Reviews Indicates Lack of Rigorous Reporting

Commonly used in the fields of oncology, cardiology, and others, time-to-event (TTE) outcomes assess not only the occurrence of an event but the amount of time that has lapsed leading up to its occurrence. For individuals in which the event did not occur, their time under observation is included.

TTE outcomes can provide useful insight into, for instance, into how long individuals survive when taking a new drug for advanced cancer. Statistical methods of calculating TTE outcomes include the use of curves and probabilities (Kaplan-Meier curves) and hazard ratios (HRs). 

In addition to being more statistically complex than a simple risk ratio, TTE outcomes are often not reported adequately enough in trials to be used in a meta-analysis without some amount of imputation or exclusion of data, which can introduce error.

In an article published in the July 2023 issue of Journal of Clinical Epidemiology, Goldkuhle and colleagues more closely examined the use of TTE outcomes in meta-analyses included in both Cochrane and non-Cochrane systematic reviews between 2017 and 2020. In the 50 included reviews, a median two TTE outcomes were included in meta-analyses, the most common being comparing the use of biologics and drugs in the treatment of neoplasms. 

However, a lack of clear reporting in the 235 trials informing these systematic reviews could easily lead to incomplete data. For instance, only 82% of the trials included a measure of follow-up duration. Information about missing data was only reported for each trial arm in 134 (57%) of the trials, and about one-third of trials reported no information at all in this respect. 

The authors conclude that trial authors using TTE outcomes should more stridently follow the Consolidated Standards of Reporting Trials (CONSORT) extension to trial outcomes, while specific guidance is needed for the reporting of meta-analysis of TTE outcomes. 

Goldkuhle, M. et al. (2023). Meta-epidemiological review identified variable reporting and handling of time-to-event analyses in publications of trials included in meta-analyses of systematic reviews. J Clin Epidemiol 159: 174-189.

The full-text publication can be accessed here.

McMaster and Guidelines International Network launch Collaboration Toolkit for Guideline Developers

Collaboration between medical professional societies and health organizations can increase the efficiency of development efforts, reduce waste, and help share resources, knowledge, and skills within and across disciplines. However, an international needs assessment of guideline developers found that certain aspects of collaboration were specifically challenging, including difficulties reconciling differences in guideline methodology and the time required for establishing collaboration agreements.

Since then, McMaster University and the Guidelines International Network (G-I-N) have worked on developing a toolkit for organizations looking to inspire, guide, and facilitate their own collaborations. The toolkit, launched at this year's G-I-N conference in Glasgow, Scotland, includes access to helpful resources such as a Memorandum of Understanding template for establishing partnerships and a glossary for making sure all parties are on the same page regarding important terms. A list of collaboration-related publications offers further knowledge and guidance for aspiring collaborators.

Check out the toolkit here and stay posted for update and additions!

EF Scholars Talk High-Speed Ways to Apply Evidence to Practice at Austin GRADE Workshop

 At the most recent GRADE Guideline Development Workshop held in Austin, Texas, our team was joined by two impressive scholars making moves in the field of evidence synthesis and evidence-based medicine. These scholars, Drs. Syed Arsalan Naqvi and Juan Ruiz, were selected to have their workshop registration fee waived based on their application materials, including a description of a current or proposed project to reduce bias in healthcare. They also had the opportunity to present on these topics and field questions from the other workshop attendees.

Syed Arsalan Ahmed Naqvi, M.B.B.S., presented on a current project to apply machine learning to develop an algorithm to automatically assess certainty of evidence in living systematic reviews within a living interactive evidence (LivE) framework. Certainty of evidence assessment requires "repetitive evaluations whenever new studies or updated reports of previously included studies are incorporated into the existing body of evidence," Dr. Naqvi explained. "The manual efforts required to assess certainty of evidence necessitate considerable time and resources, which become even more cumbersome with periodic updates." Dr. Naqvi's project described the proposal for "an adjudication system which leverages an interactive web-based graphical user interface and rule-based algorithms" for assessing certainty in the context of living systematic reviews of randomized controlled trials.

Of his experience at the workshop, Dr. Naqvi said, "“The GRADE guideline workshop was an enriching experience. It was well organized, and precise, focusing on the most important aspects of clinical guideline development. I really enjoyed the small group sessions with the faculty and the participants."

Dr. Juan Ruiz presented his proposal to implement a rapid review and question-and-answer program for hospitalists applying evidence-based medicine in everyday practice. "Doctors have questions every day at the point of care that are not always [addressed], Dr. Ruiz explained. "Factors associated with unanswered medical questions" include "lack of time" and "lack of knowledge and skills to apply evidence-based medicine tools." By training onco-hospitalists in GRADE and creating an accessible database with clinical questions, answers, and recommendations based on evidence-based methodologies, Ruiz hopes to help bridge this gap in the evidence-to-practice pipeline.

"I had the opportunity to participate as a Scholar of the Evidence Foundation in the US GRADE Network workshop in Austin, TX in May 2023," said Dr. Ruiz. "This workshop was focused on non-RCTs, an area of great interest to me since I work analyzing large national databases and electronic medical records of cancer patients. I presented and received valuable feedback for a project to be implemented in a cancer hospital to develop recommendations for daily practice using the GRADE approach. It was a very constructive three-day experience where I was able to learn new skills and share knowledge."

Interested in becoming a scholar? Applications to attend our next fully virtual fall workshop, held October 25-27, 2023, for free close on August 31. See application details here. 

GRADE Guideline Development Workshop - now with INGUIDE!

We at the U.S. GRADE Network are excited to announce that, for the first time ever, we have partnered with INGUIDE, a credentialing system for guideline developers, to offer free access to their Level 1 online training to participants of our Guideline Development Workshop and Guideline Fundamentals for Panel Members and Organization Staff pre-workshop course. 

This announcement comes just in time for our next workshop, held in Austin, Texas May 17-19, for which there are still spots available at our workshop website.

The INGUIDE program, the result of a collaboration between the Guidelines International Network (G-I-N) and McMaster University, provides certification of individuals involved in guideline development on four successive levels, from a Guideline Panel Member to becoming a Guideline Development Credentialing Instructor themselves. 

To obtain access to the INGUIDE Level 1 course, simply register for and attend our 3-day GRADE guideline development workshop in addition to the pre-workshop course on Guideline Fundamentals for Panel Members and Organization Staff. This 3-hour pre-workshop course will cover all of the necessary information about the practical aspects of guideline development, including:

• the importance of trustworthy guidelines
• checklists for the development of guidelines
• the structure and roles of a guideline panel
• the process of grading evidence and formulating recommendations

The main 3-day workshop will cover important aspects of assessing risk of bias, rating the certainty of evidence, and considering other important factors when using GRADE to develop a guideline.

Our pre-workshop course combined with our main three-day in-person course will provide all of the training required for a breeze-through experience online. You will receive free access to the ~2-3 hour online course, comprising four modules, that you can complete at your convenience after the workshop. Upon completion, you will be officially certified as an INGUIDE Level 1 Guideline Panel Member.

Why We Can't Wait for Our Austin Workshop

The U.S. GRADE Network's 18th workshop is just five weeks away! With a focus on assessing certainty of evidence informed by non-randomized studies, this workshop will take place in Austin, Texas - the home of Austin City Limits, the famous Bat Bridge, and some next-level barbecue joints.

We asked some of the workshop facilitators about what they're looking forward to about the trip (besides the chance to talk about GRADE and evidence-based medicine for three days straight!) Here's what they said...

Limited space remains for the workshop. Learn more and register and www.gradeconf.org. 

2022 Recap and What's Next for the U.S. GRADE Network

Thanks to you, 2022 was one of our best years yet.

We were relieved to see you in person, our first time since 2020, at our spring workshop in Chicago last June! Then, we reconvened online for a virtual workshop in November. Overall, we hosted over 100 participants from around the world, stretching from Canada and Spain to Colombia and Sweden. 

Facilitators and participants at the spring 2022 GRADE guideline development workshop in Chicago, IL - the first in-person USGN workshop since March 2020.

We also launched our new two-day virtual Systematic Review workshop offering and welcomed another 48 learners, including a record 13 scholars from across the globe!

We published our takeaways from our early experiences moving our workshop online in response to the Covid-19 pandemic in BMJ Evidence-Based Medicine, which you can read about here.

Finally, we launched the USGN Webinar Series and will host regular free webinars on all things GRADE. For our inaugural session, we were joined by Drs. Gordon Guyatt and Reem Mustafa. You can watch the recording and subscribe to our new YouTube channel here. 

We look forward to seeing you at our 18th Guideline Development Workshop, in person in sunny Austin, Texas, May 17-19, 2023! The workshop will have a special focus on assessing the certainty of evidence informed by non-randomized studies. Registration is now open.

From Risk of Bias to Living Evidence, Fall 2022 EF Scholars Present on Cutting-Edge EBM Projects

During our 17th GRADE Guideline Development Workshop, held virtually, the U.S. GRADE Network had the honor of introducing the three latest recipients of the Evidence Foundation scholarship to the virtual stage. On the third and final morning of the workshop, each scholar presented on their current projects to reduce bias in healthcare and move the field of evidence-based medicine methodology forward. 

Abrar Alshorman, MBBS, presented on her work to examine the tools used to assess risk of bias across various types of evidence. As choosing the wrong risk of bias tool at the start of an evidence synthesis effort might waste time and disrupt workflow, her project aims to guide systematic reviewers in choosing the best tool to use. The resulting list includes risk of bias tools for use in primary studies that were identified in a search of the evidence, including those with applications in randomized, non-randomized, diagnostic test accuracy, prognosis, and qualitative bodies of evidence. 

"The workshop struck the perfect balance between informative and interactive," said Alshorman. "[A]ll sessions have been extremely informative and provided by very knowledgeable presenters. This opportunity offered me insight into the key concepts of the GRADE approach for guideline development. I look forward to incorporating the information I've learned into practice."

Next, Brett Norling, a third-year medical student, discussed his project examining the use of the GRADE framework in systematic reviews in urology. The project reviewed literature published in the top 5 urology journals since 2000, using Perl programming code to identify articles using GRADE. The team then applied a 37-item checklist developed de novo from the "Criteria for Applying or Using GRADE." The results demonstrated that while most (95.8%) of the included reviews correctly applied GRADE on an outcome-level basis, nearly one-third (31.4%) of the 70 reviews summary of findings, evidence profile, or other from of table to present their findings.

"I came to the GRADE Workshop having performed a systematic review and a few other research projects regarding systematic review methodology," said Norling. "I felt that my understanding of GRADE was in the developing stages, though it felt a bit abstract prior to the workshop. Key GRADE principles of evidence assessment came to life as I worked with a small group to assess a body of evidence and arrive at a recommendation. I now feel well equipped to confidently assess evidence and perform systematic reviews with a greater level of autonomy than prior to this workshop."

Finally, Ariadna Alaudell-Rispau, a PhD candidate in the Biomedical Research Methodology and Public Health Doctoral Program at the Autonomous University of Barcelona, presented her proposal for the development of Living Evidence Structures Summaries (LESS). Auladell-Rispau described her project to lead a multi-design, iterative study that allows for a systematic and rigorous development of a "Living Evidence updates reporting model" that "accounts for the periodic and/or constant updates of the evidence syntheses developed under the Living Evidence model, permits the development of valid, reliable, and updated health information to health decision makers, including guideline or [health technology assessment] developers..., [and is] friendly and accessible." The project will include a systematic review, brainstorming meetings, a Delphi consensus process, and finally, testing and assessment of the newly developed model.

We look forward to updating the blog about these projects as they move forward!

ANNOUNCEMENT: Applications for the Spring 2023 GRADE Guideline Development Workshop, held in Austin, Texas, May 17-19, 2023, are due February 28th! Learn more about the application requirements here and read about the workshop details here.

COVID-END Working Groups Call for Living Systematic Reviews and Considerations of Health Equity in Evidence Synthesis and Guideline Efforts

The COVID-19 pandemic was not only associated with a rapid worldwide spread of a virus, but also of large amounts of information across the globe - not all of which was trustworthy or credible. Experts call this an "infodemic." In order to improve the synthesis and dissemination of trustworthy information in a manner that could keep up with the fast pace and ever-changing landscape of knowledge on COVID-19, the COVID-19 Evidence Network to support Decision-making (COVID-END) was established.

In a paper published in this month's issue of Journal of Clinical Epidemiology, McCaul and colleagues describe how the COVID-19 pandemic ushered in an urgent need to rapidly understand the etiology and management strategies for the disease, and to disseminate this information far and wide. However, a lack of collaboration resulting in duplication of work across institutions and countries hampered these efforts. COVID-END, comprising two working groups  dedicated to overseeing the coordination and dissemination of trustworthy evidence syntheses and guidelines, was a result of these unprecedented needs. The effort also included an Equity Task Group that evaluated the impact of evidence synthesis and recommendations on matters of health and socioeconomic disparities arising from or exacerbated by the pandemic.

Figure from McCaul et al. describing the efforts of COVID-END

The goal of the project, in the authors' words, was to support the "evidence supply side" by promoting already available resources and work led by institutions across the globe, both for those involved in evidence synthesis or the formulation of recommendations based on the evidence. The avoidance of effort duplication was highlighted by, for instance, urging guideline developers to first search for existing high-quality and up-to-date guidelines before beginning work on new recommendations. The development and use of living systematic reviews, which are continually updated as new evidence becomes available, is further highlighted as a way to improve the timeliness of evidence syntheses while reducing efforts put into new projects.

McCaul, M., Tovey, D., Young, T., et al. (2022). Resources supporting trustworthy, rapid and equitable evidence synthesis and guideline development: Results from the COVID-19 Evidence Network to support Decision-making (COVID-END). J Clin Epidemiol 151: 88-95. Manuscript available at publisher's website here. 

8 Steps Toward Incorporating Equity in Rapid Reviews and Guidelines

Along with the mass mobilization of systematic reviews of evidence ushered in by the COVID-19 pandemic was also the need to synthesize evidence and disseminate results as rapidly as possible. As part of the process of formulating guidelines based upon rapid reviews, the impact of decisions and policies on equity should be considered. In a newly published paper, Dewidar and colleagues provide specific steps for incorporating stakeholders and improving the consideration of equity in the context of rapid guidelines.

The project was part of work conducted by the Equity Task Force of the global COVID-19 Evidence Network to support Decision-making COVID-END) network. The team was diverse in terms of the gender (70% women), regions (17% from low-middle-income countries), and career stages (40% early career) represented. The resulting guidance was created in line with the steps outlined in the Cochrane Handbook's chapter on equity and followed the PRISMA-Equity (PRISMA-E) extension for reporting. The team then identified published systematic reviews related to COVID-19 that focused on populations experiencing inequities as categorized by the PROGRESS-Plus paradigm - for instance, by Place of Residence (health systems in rural areas and their preparedness for outbreaks), Education (the impact of educational attainment on adherence to COVID-19 health guidelines), and Disability (the impact of COVID-19 on those with disabilities) - for examples that review authors can incorporate equity into their own reviews.

The authors conclude that greater involvement of diverse stakeholders can encourage the consideration of more diverse social factors in the development and interpretation of systematic reviews and resulting guidelines and policies. Rapid reviews also benefit from having a translation plan that includes methods for disseminating findings in a way that is consistent with the goal of reducing inequities. 

Dewidar, O., Kawala, B.A., Antequera, A., et al. (2022). Methodological guidance for incorporating equity when informing rapid-policy and guideline development. J Clin Epidemiol 150: 142-153. Manuscript available at the publisher's website here.