It's Alive! Pt. IV: Results from a Trainee Living Systematic Review Experience

Living systematic reviews (LSRs) continue to be a topic of interest among systematic review and guideline developers, as evidenced by our history posts on the topic here, here, and here. While automation and machine learning have begun to help facilitate what is a generally time- and resource-intensive process to evidence syntheses perpetually up-to-date, some aspects of LSR development still require the human touch. Now, a recently published mixed-methods study discusses the successes and challenges of utilizing a crowdsourcing approach to keep the LSR wheels turning.

The article describes the process of involving trainees in the development of a living systematic review and network meta-analysis (NMA) on drug therapy for rheumatoid arthritis. In their report, the authors posit that evidence-based medicine is a key pillar of learning for trainees, but that they may learn better through an experiential rather than a purely didactic approach; providing the opportunity to participate in a real-life systematic review may provide this experiential learning. 

In short, the team first applied machine learning to sort through an initial database to filter out randomized controlled trials, which was then further assessed by a crowdsourcing platform, Cochrane Crowd. Next, trainees ranging from undergraduate students to practicing rheumatologists and researchers recruited through Canadian and Australian rheumatology mailing lists further assessed articles for eligibility and extracted data from included articles. 

Training included a mix of online webinars, one-on-one trainings, and handbook provisions. Conflicting results were further assessed by an expert member of the team. The authors then elicited both quantitative and qualitative feedback about the trainees' experiences of taking part in the project through a combination of electronic survey and one-on-one interviews. 

Overall, the 21 trainees surveyed rated their training as adequate and experience generally positive. Respondents specifically listed better understanding of PICO criteria, familiarity with outcome measures used in rheumatology, and the assessment of studies' risk of bias as the greatest learning benefits obtained. 

Of the 16 who participated in follow-up interviews, the majority (94%) described a practical and enjoyable experience. Of particular positive regard was the use of task segmentation throughout the project, during which specific tasks (i.e., eligibility assessment versus data extraction) could be "batch-processed," allowing trainees to match the specific time and focus demands to the selected task at hand. Trainees also communicated an appreciation for the international collaboration involved in the review as well as the feeling of meaningfully contributing to the project. 

Notable challenges included issues related to the clarity of communication regarding deadlines and expectations, as well as technical glitches experienced through the platforms used for screening and extraction. Though task segmentation was seen as a benefit, it also included drawbacks: namely, the risk of more repetitive tasks such as eligibility assessment becoming tedious while others that require more focus (i.e., data extraction) may be difficult to integrate into an already-busy daily schedule. To address these issues, the authors suggest improving communications to include regular, frequent updates and deadline reminders, working through technological glitches, and carefully matching tasks to the specific skillsets and availabilities of each trainee.

Lee, C., Thomas, M., Ejaredar, M., Kassam, A., Whittle, S.L., Buchbinder, R., ... & Hazlewood, G.S. (2022). Crowdsourcing trainees in a living systematic review provided valuable experiential learning opportunities: A mixed-methods study. J Clin Epidemiol (in-press). Manuscript available at the publisher's website here.

A New Template for Standardized Wording when Reporting Evidence-to-Decision Considerations in Guidelines

One of the major tenets of GRADE is that certainty of the evidence is just one component of decision-making. Ultimately, decision-makers also need to take into account important factors such as values and preferences, feasibility, and considerations of the impact of a decision on health equity and resource utilization. These factors and others are part of the Evidence-to-Decision (EtD) framework that guides the process from the assessment of certainty of evidence to the final formulation of recommendations in a structured, transparent manner.

Often, multiple teams and individuals involved in the development of a guideline will need to work together to compete the EtD process, which can be a source of confusion. Additionally, until now, no official guidance existed for the use of standardized wording when considering and reporting each EtD framework component. Earlier this year, Piggott and colleagues aimed to address this issue with an article published in the Journal of Clinical Epidemiology.

The project, comprising ten guideline development groups and over 250 recommendations, set out to develop a standardized framework for clear, transparent, and efficient wording when reporting Evidence-to-Decision components within a guideline. This template was then used in two guidelines in development - the European Commission Initiative on Breast Cancer (ECIBC) and the Endocrine Society guidelines on hyperglycemia, hypoglycemia and hypercalcemia. During this process, the authors were able to pilot the wording, receive feedback, and refine the template. The real-life guidelines were also used to provide examples of wording recommendations.

The article includes suggested wording structure and examples for reporting the magnitude and certainty of effect estimates, for conclusions of each portion of the EtD framework, and for justification of recommendations as well as notes on implementation considerations, monitoring and evaluation, and research priorities. 

The authors note that these suggestions are preliminary and may require further refinement. Additionally, current examples of consistent and clear wording of EtDs continues to be lacking, though the dissemination of this guidance may improve future publications. While the suggestions within the article are focused on clinical decisions related to management of conditions, future efforts may expand this to guidelines for diagnostic testing, coverage, and other important areas.

Piggott, T., Baldeh, T., Dietl, B., Wiercoch, W., Nieuwlaat, R., Santesso, N., ... & Schünemann, H. (2022). Standardized wording to improve efficiency and clarity of GRADE EtD frameworks in health guidelines. J Clin Epidemiol (online ahead of print). Manuscript available at the publisher's website here.

   

U.S. GRADE Network Holds Its First Two-Day Systematic Review Workshop

In response to popular demand, the U.S. GRADE Network recently expanded its half-day course on systematic reviews into a two-day virtual workshop. Taking place over February 1-2, 2022, the sessions comprised six total hours of instructional time and question-and-answer sessions with Network faculty in addition to three hours of hands-on activities in a small-group format. Large-group lectures ranged from developing a focused clinical question to conducting meta-analysis and evaluating the quality of systematic reviews. In hands-on sessions of ten participants each, attendees were introduced to a free online screening platform (Rayyan) and tried their hands at assessing risk of bias and conducting meta-analysis in RevMan, a free systematic review and meta-analysis software from Cochrane.

As part of the program, the Evidence Foundation welcomed three recipients of scholarships to attend the workshop free of charge. As part of their applications, the scholars described a current or proposed project for a systematic review. The three recipients, along with their projects, included:

• Bryden Giving, MAOT, OTR/L (Boston University): A traffic light of evidence for occupational therapy interventions supporting autistic children and youth
• Nirjhar Ruth Ghosh, MS (Texas A&M University): Evidence-based practice in the field of nutrition: A systematic review of knowledge, skills, attitudes, behaviors and teaching strategies
• Milton A. Romero-Robles (Universidad Nacional del Santa): Participation, involvement and main barriers in the inclusion of patients with non-communicable diseases in the development of clinical practice guidelines: A systematic review protocol

Be the first to hear about these and other trainings @USGRADEnet on twitter or at www.systematicreview.org.

Note: applications for scholarships to attend the upcoming GRADE Guideline Development Workshop held July 29-July, 2022, in Chicago, Illinois, close March 31. See application details here.

Guideline Development Resource Alert: the G-I-N Public and Patient Toolkit

One of the most common challenges to developing rigorous and high-quality guidelines is the inclusion of the patient and public perspective into the formulation of recommendations. In fact, in a recently published needs assessment of guideline developers worldwide, 81.5% answered with a 5 or greater on a 7-point Likert scale that the incorporation of the patient voice was a relevant need for their organization. 

Now, the Guidelines International Network (G-I-N) has launched a large-scale toolkit aimed to address commonly experienced issues related to patient and public involvement. The result of a combination of international experiences and best practice examples, the toolkit is a one-stop shop spanning the systematic review and guideline development process, from conducting targeted consultation with the public to recruiting and supporting patient panel members to communicating recommendations to the public at-large. As a "living resource," the toolkit will continue to expand and evolve as further information and experience is cultivated.

You can find the freely available toolkit here.

2021 Reflections Before a Big Year Ahead

 From all of us at the U.S. GRADE Network, thanks for making 2021 a year to remember.

• We hosted our 14th and 15th workshops virtually - meaning we had the joy of talking GRADE with 95 attendees from many corners of the world, including Spain, Argentina, Pakistan, and Peru.
• We co-published an international needs assessment of guideline-producing organizations in collaboration with the Guidelines International Network.
• We announced our first ever two-day comprehensive systematic review workshop scheduled to be held virtually this coming February 1-2. Learn more and register here.
• And we continue to look forward to hosting our 16th guideline development workshop, once again live and in person, from June 29-July 1, 2022, in Chicago, Illinois. Registration now open.

GRADE faculty take a moment to pose for a picture with some attendees of the fall 2021 virtual guideline development workshop.

Stay tuned for more.

To stay up-to-date on all future news, events, and research round-ups from us, make sure to follow us on Facebook, twitter, and LinkedIn.

Thanks again for an excellent year, and our best and brightest wishes for a brilliant 2022.

Fall Scholars Dazzle at the 15th GRADE Guideline Development Workshop

During the 15th GRADE Guideline Development Workshop held virtually last month, the Evidence Foundation had the pleasure of welcoming three new scholars with the opportunity to attend the workshop free of charge. As part of the scholarship, each recipient presented to the workshop attendees about their current or proposed project related to evidence-based medicine and reducing bias in healthcare.

Razan Mansour, MD, a postdoctoral research fellow at the University of Kansas Medical Center,  spoke about the challenges of navigating variabilities among published systematic reviews when developing clinical recommendations. Variability may emerge from differences between reviews regarding study inclusion criteria, risk of bias assessment, and the way data are presented. Most sources of variability, Dr. Mansour said, are difficult to explain, but a modified version of the A MeaSurement Tool to Assess systematic Reviews (AMSTAR) 2 tool can help identify common "red flags" and help to prioritize and identify the best systematic reviews off of which to base resulting clinical recommendations. Information from multiple higher-quality systematic reviews may be used, and data from individual studies may need to be extracted to paint a complete picture of the evidence.

Says Dr. Mansour, "Learning about how to rate the certainty of evidence about diagnosis was particularly helpful to me, as this is usually challenging. The subgroup discussions were perfect for one-to-one learning and applying your skills as you go through GDT."

Next, Reena Ragala, MOTR/L, spoke about her project as a Senior Evidence-Based Practice Analyst at the Medical University of South Carolina Value Institute. The Institute was established in 2012 to support MUSC's efforts to build the infrastructure to advance science and discovery through evidence-based practice. Starting in the spring/summer of 2022, Ragala will help lead guideline development "boot camps" though the Institute to train providers within rural health networks in the application of evidence-based guideline methodology. Through this process, Ragala hopes to empower rather than intimidate front-line staff and clinicians new to clinical guideline development.

The fall 2021 Evidence Foundation scholars pose for a virtual group photo with U.S. GRADE Network faculty between workshop sessions. From top-left:  Philipp Dahm, Reem Mustafa, Osama Altayar, Yngve Falck-Ytter, Rebecca Morgan, Carolina Soledad, Reena Ragala, Perica Davitkov, Madelin Siedler, Razaan Mansour, and Shahnaz Sultan.

"When using GRADE for diagnostic tests or guideline development, judgements or recommendations are based on patient outcomes, said Ragala. "GRADE doesn’t just look at the strength/quality of the evidence, but incorporates the feasibility, accuracy, bias, and benefit/harm to the patient to ultimately make recommendations."

Carolina Soledad, MD, MStat, PhD, spoke about the unique opportunities, challenges, and solutions within an aspect of guideline development that has gained great relevance in recent years: reaching consensus within the context of virtual meetings. Dr. Soledad, a junior methodologist within the European Society of Anesthesia and Intensive Care (ESAIC) Guidelines Committee, discussed the fact that while in-person meetings have several strengths -  such as access to nonverbal cues such as body language and facial expression and shared context - virtual meetings also have unique benefits, such as cost savings and a reduced need to focus on logistics. In addition, they have become a necessity in the age of a global pandemic. To help address the unique questions surrounding virtual meetings - such as the optimal length and number of participants  - Dr. Soledad developed a 33-item survey of anesthesiologists and intensivists involved in guideline development. The findings will help formulate best practices for improving the level of communication, engagement, and effectiveness of future virtual meetings.

According to Dr. Soledad, "This workshop gave me a peek into GRADEpro GDT software, and it turned out to be easier to use than I've thought!"

If you are interested in learning more about GRADE and attending the workshop as a scholarship recipient, applications for our upcoming workshop in Chicago, Illinois, are now open. The deadline to apply is March 31, 2022. Details can be found here. 

6 Simple Rules for Creating a Plain Language Summary of a GRADE Guideline Recommendation

While seasoned clinicians and methodheads may consider the nuances of clinical guideline development everyday fare, there is generally a lack of awareness among the public about the implications and use of guidelines. In addition, there is some evidence of public concern that guidelines may be used to ration care as well as public confusion about how they should be applied to an individual's unique needs. The translation of guideline recommendations into plain language, however, may help improve public knowledge and awareness of the applications and implications of guidelines.

In a new paper published in the Journal of Clinical Epidemiology, Santesso and colleagues set out to develop a template for communicating guideline recommendations as well as explore public attitude around guidelines. First, the authors conducted semi-structured focus groups to gather information about general perceptions and opinions regarding guidelines. Then, these insights were used to develop a plain language template which was user-tested. The template was then revised into a final version. During the process, a few key themes emerged, including:

• an upfront and clear description of the population/individuals to whom the guideline applies
• a section detailing topics and questions to bring up with one's health care provider
• definitions surrounding the strength of the recommendation, and further considerations for decision-making around conditional recommendations
• formatting that makes use of bullets and tables rather than blocks of text

These themes informed the development of the final template, which includes six major items:

1. the recommendation, its strength (with a symbol), and an explanation 
2. the population/individuals to whom the recommendation applies
3. rationale for the strength of the recommendation
4. additional considerations when using the recommendation
5. benefits and harms 
6. implications, what a patient can do, and questions or topics to discuss with one's health care provider

Santesso, N., Wiercioch, W., Barbara, A.M., Dietl, H, and Schünemann, H.J. (2021). Focus groups and interviews with the public led to the development of a template for a GRADE plain language recommendation. J Clin Epidemiol, in-press.

Manuscript available at the publisher's website here.