Showing posts with label GRADE. Show all posts
Showing posts with label GRADE. Show all posts

Monday, October 17, 2022

Use of an Evidence-to-Decision Framework is Associated with Better Reporting, More Thorough Consideration of Recommendations

In the guideline development process, a panel should use a defined framework to consider multiple aspects of a clinical decision, including but not limited to the certainty of the underlying evidence, potential impact on resource use, or variability in the values and preferences of patients and other stakeholders. Such frameworks include the GRADE Evidence-to-Decision (EtD) format as well as others such as the "decision-making triangle" and Guidance for Priority-Setting in Health care (GPS Health). 

To better understand the prevalence and use of these various frameworks within guidelines, Meneses-Echavez and colleagues systematically searched for guidelines and related guideline production manuals published between 2003 and May 2020. Items were screened and extracted by two independent authors, with a total of 68 full text documents included and analyzed.

Of these documents, most (93%) reported using a structured framework to assess the certainty of evidence, about half (53%) of which used GRADE or adapted systems based on GRADE (10%). Similarly, 88% of documents reported using a framework to rate the strength of recommendations, with about half (51%) using the GRADE approach. However,  only about two-thirds (66%) of the included documents explicitly stated the process for formulating resulting recommendations. 

Finally, the GRADE framework  was most commonly used for the evidence-to-decision, being cited in 42% of the included articles, with other reported frameworks including NICE (8%), SIGN (8%) and USPSTF (4%). Articles using the GRADE EtD framework reported considering more criteria than those using alternative approaches. The most commonly used criteria across documents included desirable effects (72%), undesirable effects (73%), and the certainty of evidence of effects (73%); the least commonly applied criteria were acceptability (28%), certainty of the evidence of required resources (25%), and equity (16%). 



The use of any EtD framework was associated with a greater likelihood of incorporating perspectives (odds ratio: 2.8; from 0.6-13.8) and subgroup considerations (odds ratio:7.2; from 0.9-57.9), as was the use of GRADE compared to other EtDs (odds ratios: 1.4 and 8.4). These differences also affected whether justifications were reported for each judgment as well as the inclusion of notes to consider for the implementation of recommendations and for monitoring and evaluating recommendations.  

The authors conclude that guidance documents stand to benefit from the more explicit reporting of how recommendations are formulated, from the initial grading of the certainty of underlying evidence to the consideration of how recommendations will affect various criteria such as resource use and equity. These changes, in the words of the authors, may "enhance transparency and credibility, enabling end users to determine how much confidence they can have in the recommendations; facilitate later adaptation to contexts other than the ones in which they were originally developed; and improve usability and communicability of the EtD frameworks." 

Meneses-Echaves JF, Bidonde J, Yepes-Nuñez JJ, et al. (2022).  Evidence to decision frameworks enabled structured and explicit development of healthcare recommendations. J Clin Epidemiol 150:51-62. Manuscript available at publisher's website here.









Thursday, September 1, 2022

When is Imprecision "Very Serious" for a Systematic Review? New GRADE Guidance for Rating Down Two Levels Using a Minimally Contextualized Approach

Within the GRADE framework, the assessment of imprecision of an effect estimate relies on two major aspects: Optimal Information Size (OIS), which considers whether the sample size of the pooled estimate has adequate power to detect a minimally important difference, and the confidence interval (CI) approach, which requires inspecting the width of the 95% confidence interval and determining whether it crosses a meaningful threshold of effect (for instance, ranges from a null or trivial effect on one end to a potentially meaningful effect on the other).

GRADE users who are applying the framework to the assessment of the certainty of evidence in the context of a systematic review but not a practice guideline have, in previous guidance, been instructed to prioritize the OIS rather than the CI approach, which typically requires some judgment about a meaningful effect threshold, or "contextualization." However, given the fact that systematic review authors and readers can also benefit from the application of effect thresholds when assessing imprecision, new guidance has been published to help systematic reviewers apply a "minimally contextualized approach" and consider both reasons for rating down by one, two, or three levels if necessary. 




In addition to providing multiple examples in which rating down by two levels based on the span of the CI may be warranted, the paper also suggests circumstances in which systematic reviewers may rate down by two levels in the absence of concerns about the CI but in the presence of a particularly small information size. Specifically, when the CI does not overlap with threshold(s) of interest and the effect is sufficiently large, authors may still consider rating down for imprecision if the OIS (the sample size needed to detect an effect within one adequately powered trial as determined by a conventional power analysis) is not met. 

For dichotomous outcomes, in this case, one may consider rating down by two levels immediately if the ratio of the upper- to the lower-boundary of the CI is higher than 2.5 for odds ratios or 3.0 for relative risk ratios. If these criteria are not fulfilled, authors should still compare the pooled sample size to the determined OIS. For continuous outcomes, a total sample size that does not exceed 30-50% of the OIS is considered reason for rating down by two levels to "very serious" imprecision.

Finally, if a CI is so wide that it causes authors to be very uncertain about any estimate of effect, it may be acceptable to rate down by three levels for imprecision.

Zeng, L., Brignardello-Petersen, R., Hultcrantz, M., et al. (2022). GRADE Guidance 34: update on rating imprecision using a minimally contextualized approach. J Clin Epidemiol, online ahead of print. Manuscript available at the publisher's web site here. 












Thursday, November 11, 2021

Fall Scholars Dazzle at the 15th GRADE Guideline Development Workshop

During the 15th GRADE Guideline Development Workshop held virtually last month, the Evidence Foundation had the pleasure of welcoming three new scholars with the opportunity to attend the workshop free of charge. As part of the scholarship, each recipient presented to the workshop attendees about their current or proposed project related to evidence-based medicine and reducing bias in healthcare.


Razan Mansour, MD, a postdoctoral research fellow at the University of Kansas Medical Center,  spoke about the challenges of navigating variabilities among published systematic reviews when developing clinical recommendations. Variability may emerge from differences between reviews regarding study inclusion criteria, risk of bias assessment, and the way data are presented. Most sources of variability, Dr. Mansour said, are difficult to explain, but a modified version of the A MeaSurement Tool to Assess systematic Reviews (AMSTAR) 2 tool can help identify common "red flags" and help to prioritize and identify the best systematic reviews off of which to base resulting clinical recommendations. Information from multiple higher-quality systematic reviews may be used, and data from individual studies may need to be extracted to paint a complete picture of the evidence.

Says Dr. Mansour, "Learning about how to rate the certainty of evidence about diagnosis was particularly helpful to me, as this is usually challenging. The subgroup discussions were perfect for one-to-one learning and applying your skills as you go through GDT."

Next, Reena Ragala, MOTR/L, spoke about her project as a Senior Evidence-Based Practice Analyst at the Medical University of South Carolina Value Institute. The Institute was established in 2012 to support MUSC's efforts to build the infrastructure to advance science and discovery through evidence-based practice. Starting in the spring/summer of 2022, Ragala will help lead guideline development "boot camps" though the Institute to train providers within rural health networks in the application of evidence-based guideline methodology. Through this process, Ragala hopes to empower rather than intimidate front-line staff and clinicians new to clinical guideline development.

The fall 2021 Evidence Foundation scholars pose for a virtual group photo with U.S. GRADE Network faculty between workshop sessions. From top-left:  Philipp Dahm, Reem Mustafa, Osama Altayar, Yngve Falck-Ytter, Rebecca Morgan, Carolina Soledad, Reena Ragala, Perica Davitkov, Madelin Siedler, Razaan Mansour, and Shahnaz Sultan.

"When using GRADE for diagnostic tests or guideline development, judgements or recommendations are based on patient outcomes, said Ragala. "GRADE doesn’t just look at the strength/quality of the evidence, but incorporates the feasibility, accuracy, bias, and benefit/harm to the patient to ultimately make recommendations."

Carolina Soledad, MD, MStat, PhD, spoke about the unique opportunities, challenges, and solutions within an aspect of guideline development that has gained great relevance in recent years: reaching consensus within the context of virtual meetings. Dr. Soledad, a junior methodologist within the European Society of Anesthesia and Intensive Care (ESAIC) Guidelines Committee, discussed the fact that while in-person meetings have several strengths -  such as access to nonverbal cues such as body language and facial expression and shared context - virtual meetings also have unique benefits, such as cost savings and a reduced need to focus on logistics. In addition, they have become a necessity in the age of a global pandemic. To help address the unique questions surrounding virtual meetings - such as the optimal length and number of participants  - Dr. Soledad developed a 33-item survey of anesthesiologists and intensivists involved in guideline development. The findings will help formulate best practices for improving the level of communication, engagement, and effectiveness of future virtual meetings.

According to Dr. Soledad, "This workshop gave me a peek into GRADEpro GDT software, and it turned out to be easier to use than I've thought!"

If you are interested in learning more about GRADE and attending the workshop as a scholarship recipient, applications for our upcoming workshop in Chicago, Illinois, are now open. The deadline to apply is March 31, 2022. Details can be found here. 






Friday, September 24, 2021

6 Simple Rules for Creating a Plain Language Summary of a GRADE Guideline Recommendation

While seasoned clinicians and methodheads may consider the nuances of clinical guideline development everyday fare, there is generally a lack of awareness among the public about the implications and use of guidelines. In addition, there is some evidence of public concern that guidelines may be used to ration care as well as public confusion about how they should be applied to an individual's unique needs. The translation of guideline recommendations into plain language, however, may help improve public knowledge and awareness of the applications and implications of guidelines.

In a new paper published in the Journal of Clinical Epidemiology, Santesso and colleagues set out to develop a template for communicating guideline recommendations as well as explore public attitude around guidelines. First, the authors conducted semi-structured focus groups to gather information about general perceptions and opinions regarding guidelines. Then, these insights were used to develop a plain language template which was user-tested. The template was then revised into a final version. During the process, a few key themes emerged, including:

  • an upfront and clear description of the population/individuals to whom the guideline applies
  • a section detailing topics and questions to bring up with one's health care provider
  • definitions surrounding the strength of the recommendation, and further considerations for decision-making around conditional recommendations
  • formatting that makes use of bullets and tables rather than blocks of text

These themes informed the development of the final template, which includes six major items:
  1. the recommendation, its strength (with a symbol), and an explanation 
  2. the population/individuals to whom the recommendation applies
  3. rationale for the strength of the recommendation
  4. additional considerations when using the recommendation
  5. benefits and harms 
  6. implications, what a patient can do, and questions or topics to discuss with one's health care provider
Santesso, N., Wiercioch, W., Barbara, A.M., Dietl, H, and Schünemann, H.J. (2021). Focus groups and interviews with the public led to the development of a template for a GRADE plain language recommendation. J Clin Epidemiol, in-press.

Manuscript available at the publisher's website here. 




















Monday, September 13, 2021

Re-analysis of a systematic review on injury prevention demonstrates that methods do really matter

How much of a difference can methodological decisions make? Quite a bit, argues a new paper published in the Journal of Clinical Epidemiology. A re-analysis of a 2018 meta-analysis on the role of the Nordic hamstring curl (NHE) on injury prevention, the study outlined and then executed several methodological changes within the context of an updated search and found that the resulting magnitude of effect - and strength of recommendations using GRADE - were not quite as dazzling as the original analysis.

Impellizzeri and colleagues noted several suggested changes to the 2018 paper, including:

  • limiting the meta-analysis to higher-level evidence (randomized controlled trials) when available,
  • clarifying the interventions used in the included studies and being cognizant of the effect of co-interventions (for instance, when NHE was used alone versus in combination with other exercises as part of an injury reduction program),
  • being careful not to "double-dip" on events (i.e., injuries) that recur in the same individual when presenting the data as a risk ratio
  • discussing the impact of between-study heterogeneity when discussing the certainty of resulting estimates,
  • presenting the lower- and upper-bounds of 95% confidence intervals for estimates of effect in addition to the point estimates, and
  • taking the limitations of the literature and other important considerations into account when formulating final summaries or recommendations (for instance, using the GRADE framework)
The authors ran an updated systematic search but excluded non-randomized controlled trials or studies that incorporated other exercises with the NHE in the intervention group. Risk of bias was assessed using the Cochrane tool for randomized studies. The overall certainty of evidence as assessed using GRADE was rated "low," although given that concerns regarding risk of bias, inconsistency, and imprecision were noted, the certainty may range to "very low" following the standard GRADE framework. The forest plot of the updated analysis can be seen below.


The results of the updated analysis show that rather than reduce the risk of hamstring injury by 50%, the range of possible effects was too large to draw a conclusion on the effectiveness of this intervention, and only a conditional recommendation can be warranted.

Impellizzeri, F.M., McCall, A., and van Smeden, M. (2021). Why methods matter in a meta-analysis: A reappraisal showed inconclusive injury preventive effect of Nordic hamstring exercise. J Clin Epidemiol, in-press.

The manuscript is available at the publisher's site here.


















Wednesday, August 4, 2021

Correction to guidance for assessing imprecision with continuous outcomes

Systematic review and guideline developers take note: the authors of the 2011 guidance on assessing imprecision within the GRADE framework have recently issued a correction related to the assessment of information size when evaluating a continuous outcome.


Whereas the article stated originally that a sample size of approximately 400 (200 per group) would be required to detect an effect size of 0.2 standard deviations assuming an alpha of 0.05 and a power of 0.8, the correct number is actually 800 (400 per group). 

The full corrigendum can be read here. 

Thursday, July 29, 2021

New GRADE guidance on assessing imprecision in a network meta-analysis

Imprecision is one of the major domains of the GRADE framework and is used to assess whether to rate down the certainty of evidence related to an outcome of interest. In a traditional ("pairwise") meta-analysis which compares two intervention groups, exposures, or tests against one another, two considerations are made: the confidence interval around the absolute estimate of effect, and the optimal information size (OIS). If the bounds of the confidence interval cross a threshold for a meaningful effect, and/or if optimal information size given the sample size in the meta-analysis is not met, then one should consider rating down for imprecision.

In the context of small sample sizes, confidence intervals around an effect may be fragile - meaning they could be changed substantially with additional information. Therefore, the consideration of OIS along with the bounds of the confidence interval helps address this concern when rating the certainty of evidence to develop a clinical recommendation. This is typically done by assessing whether the sample size of the meta-analysis meets that determined by a traditional power analysis for a given effect size.

However, in a network meta-analysis, both direct and indirect comparisons are made across various interventions or tests. Thus, especially if the inclusion of indirect comparisons changes the overall estimate of effect, considering only the sample size involved in the direct comparisons would be misleading. 


A new GRADE guidance paper lays out how to assess imprecision in the context of a network meta-analysis:

  • If the 95% confidence interval crosses a decision-making threshold, rate down for imprecision. Thresholds should be ideally set a priori. It may be considered to rate down by two or even three levels depending on the degree of imprecision and the resulting communication of the certainty of evidence. For example, if imprecision is the only concern for an outcome, rating down by two instead of one level would be the difference between saying that a certain intervention or test "likely" or "probably" increases or decreases a given outcome, versus whether it simply "may" have this effect.
  • If the 95% confidence interval does not cross a decision-making threshold, consider whether the effect size may be inflated. If a point estimate is far away enough from a threshold, even a relatively wide CI may not cross it. Further, relatively large effect sizes from smaller pools of evidence can be reduced with future research. 
    • In the case of a large effect size, consider whether OIS is met. If the number of patients contributing to a NMA does not meet this number, consider rating down by one, two, or three levels depending on the severity of the width of the CI. 
    • If the upper-limit of a confidence interval using relative risk is 3 or more times higher than the lower-limit, OIS has likely not been met. Similarly, upper-to-lower-limit comparisons of odds ratios exceeding 2.5 have likely not met OIS.
  • Alternatively, when the effect size is both modest, plausible, and does not cross a threshold, one likely does not need to rate down for imprecision. 
  • Avoid "double dinging" for imprecision if this limitation has already been addressed by rating down elsewhere.

Brignardello-Peterson R, Guyatt GH, Mustafa RA, et al. (2021). GRADE guidelines 33. Addressing imprecision in a network meta-analysis. J Clin Epidemiol (in-press). 

Manuscript available at the publisher's website here.





Friday, June 25, 2021

Scholars at 14th GRADE Workshop Discuss the Unique Challenges of Sparse Evidence, Guideline Collaborations, and Financial Incentives in Healthcare

During the 14th GRADE Guideline Development Workshop held virtually last month, the Evidence Foundation had the pleasure of welcoming three new scholars with the opportunity to attend the workshop free of charge. As part of the scholarship, each recipient presented to the workshop attendees about their current or proposed project related to evidence-based medicine and reducing bias in healthcare.

This spring's lot of three scholars was nothing short of incredibly impressive. Ifeoluwa Babatunde, a PhD student in clinical research at Case Western Reserve University, discussed the unique challenges of developing a guideline on the management of patients undergoing patent foramen ovale (PFO) closure for the Society for Cardiovascular Angiography and Interventions (SCAI). The synthesis of evidence for this question is hampered by controversies and limited evidence as well as complications due to comorbidities and age differences in the populations of interest. Babatunde discussed her interest in attending the workshop to learn more about the appropriate use of observational and indirect evidence to better answer questions related to PFO closure.

"The GRADE workshop helped me to see systematic review methodology from a deeper and more critical perspective," said Babatunde. "GRADE offers a very comprehensive yet succinct and transparent framework for developing and ascertaining the certainty of evidence in guidelines. Hence I feel better equipped to tackle challenges that arise from creating reviews and guidelines regarding conditions and populations with sparse RCTs."


Next, Dr. Pichamol Jirapinyo, the Director of Bariatric Endoscopy Fellowship at Brigham and Women's Hospital and instructor at Harvard Medical School, discussed her work on an international joint guideline development effort between the American Society for Gastrointestinal Endoscopy (ASGE) and the European Society of Gastrointestinal Endoscopy (ESGE) to produce recommendations for endoscopic and bariatric metabolic therapy (EBMT) in patients with obesity. EBMT is one of several possible management routes for obesity, alongside pharmacological and surgical options. The project will aim to answer several questions, including how patients should be managed before and after EBMT, and regarding the safety and efficacy of both gastric and small bowel EBMT.

“The GRADE workshop provided me a great framework on how to apply GRADE methodology to systematic review and meta-analysis to rigorously develop a guideline," said Dr. Jirapinyo. 'In addition to learning about the GRADE methodology itself, I found the workshop to be tremendously helpful with providing practical tips on how to run a guideline task force successfully and efficiently.”  

Finally, Dr. Lillian Lai, a research fellow in the Department of Urology at the University of Michigan, presented an intriguing discussion of financial incentives in clinical decision-making in urology. The surveillance and management of localized prostate cancer, for instance, has several different options ranging from active surveillance (which is less costly) to prostatectomy (which is more costly). Regardless of the reported health outcomes of these approaches, there is little financial incentive to conduct surveillance as opposed to surgery. The project's goal is to use health services research methods to understand how urologists response to large financial incentives, and then create financial incentives and remove financial disincentives for the promotion of guideline-concordant practices. 

"I gained invaluable knowledge on how to use the GRADE approach to rate the certainty of evidence and strength of recommendations," said Dr. Lai. "Going through the guideline development tool with experts in small groups was particularly useful for me to understand what a guideline recommendation means and entails. This workshop came at a critical time in the backdrop of COVID, and the ever-changing landscape of medicine where patients and providers need to make timely and informed decisions together."

If you are interested in learning more about GRADE and attending the workshop as a scholarship recipient, applications for our upcoming virtual workshop in October are now open. The deadline to apply is July 31, 2021. Details can be found here. 

Wednesday, June 9, 2021

Evidence Foundation scholar spotlight: Georgios Schoretsanitis

Last fall, Dr. Georgios Schoretsanitis attended the 13th (and first-ever virtual) GRADE guideline development workshop as a scholar of the Evidence Foundation. As such, he presented to the rest of the workshop attendees on his work developing guidelines for therapeutic drug monitoring to optimize and tailor treatment for psychotherapeutic medications. Beginning in 2017, a series of recommendations for reference ranges for two commonly prescribed antipsychotic medications was developed, followed this year by an international joint consensus statement on blood levels to optimize antipsychotic treatment in clinical practice.

Dr. Schoretsanitis now has an exciting update on his project.

"My main research interest is therapeutic drug monitoring, also known as TDM, which refers to the quantification and interpretation of medication levels in the blood (plasma or serum) of the patient treated with psychotropic agents," says Dr. Schoretsanitis. "The aim of TDM in clinical practice is to improve treatment response and safety outcomes. Apart from analyzing TDM clinical routine data, I have also been working as a member of the TDM taskforce of the German Association of Neuropsychopharmacology and Phaarmacopsychiatry (Arbeitsgemeinschaft für Neuropsychopharmakologie und Pharmacopsychiatrie; AGNP) involved in systematic reviews of TDM literature, which provide so-called therapeutic reference ranges for medication levels. These ranges may orient clinicians during dose selection. Attending the virtual GRADE workshop in October 2020 provided me much of inspiration, but also knowledge of well-established methodological tools for the assessment of quality of evidence.

Hereafter, in the TDM task force of AGNP, we adopted a GRADE-oriented approach in assessing TDM literature as we are reviewing new TDM evidence on commonly prescribed antipsychotics under the supervision of Prof. Gerhard Gründer, Department of Molecular Neuroimaging, Central Institute of Mental Health, Medical Faculty Mannheim, University of Heidelberg, Mannheim, Germany. This type of approach is more standardized and follows GRADE guidelines. Ultimately, this work will enhance methodological rigidity for the next Consensus guidelines for therapeutic drug monitoring in neuropsychopharmacology [last update 2018; Hiemke et al, Pharmacopsychiatry]. I strongly encourage researchers involved in systematic reviews or assessment of evidence quality to attend the GRADE workshop which enables a major upgrade of related skills and knowledge."



Stay tuned for future updates from other past Evidence Foundation scholars like Dr. Schoretsanitis and the exciting work they are doing to improve the application of GRADE methodology and evidence-based medicine.

If you are interested in learning more about GRADE and attending the workshop as a scholarship recipient, applications for our upcoming virtual workshop in October are now open. The deadline to apply is July 31, 2021. Details can be found here. 


Thursday, April 8, 2021

Digging Deeper: 5 Ways to Help Guide Decision-Making When Research Evidence is "Insufficient"

A key tenet underlying the GRADE framework is that the certainty of available research evidence is a key factor to be considered in the course of clinical decision-making. But what if little to no published research exists off of which to base a recommendation? At the end of the day, clinicians, patients, policymakers, and others will still need to make a decision, and will look to a guideline for direction. Thankfully, there are other options to pursue within the context of a systematic review or guideline that ensures that as much of the available evidence is presented as possible, although it may be from less traditional or direct sources.

A new project conducted by the Evidence-based Practice Center (EPC) Program of the Agency for Healthcare Research and Quality (AHRQ) developed guidance for supplementing a review of evidence when the available research evidence is sparse or insufficient. This guidance was based on a three-pronged approach, including:

  • a literature review of articles that have defined and dealt with insufficient evidence, 
  • a convenience sample of recent systematic reviews conducted by EPCs that included at least one outcome for which the evidence was rated as insufficient, and
  • an audit of technical briefs from the EPCs, which tend to be developed when a given topic is expected to yield little to no published evidence and which often contain supplementary sources of information such as grey literature and expert interviews.
Through this approach, the workgroup identified five key strategies for dealing with the challenge of insufficient evidence:
  1. Reconsider eligible study designs: broaden your search to capture a wider variety of published evidence, such as cohort or case studies.
  2. Summarize evidence outside the prespecified review parameters: use indirect evidence that does not perfectly match the PICO of your topic in order to better contextualize the decision being presented.
  3. Summarize evidence on contextual factors (factors other than benefits/harms): these include key aspects of the GRADE Evidence-to-Decision framework, such as patient values and preferences and the acceptability, feasibility, and cost-effectiveness of a given intervention.
  4. Consider modeling if appropriate, and if expertise is available: if possible, certain types of modeling can help fill in the gaps and make useful predictions for outcomes in lieu of real-life research.
  5. Incorporate health system data: "real-world" evidence such as electronic health records and registries can supplement more mechanistic or explanatory RCTs.



Some of these challenges can be more efficiently addressed up-front, before the scoping of a new review even begins. For instance, identifying topic experts and stakeholders who are familiar with the quantity and quality of available evidence can help a group foresee potential gaps and plan for the need to broaden the scope. Care should be taken to identify the outcomes that are of critical importance to patients, and through this lens, develop strategies and criteria within the protocol that will best meet the needs of the review while tapping into as much evidence as possible. Finally, researchers should avoid using the term "insufficient" when describing the evidence, and instead explicitly state that no eligible studies or types of evidence were available.

Murad MH, Chang SM, Fiordalisi CV, et al. (2021). Improving the utility of evidence synthesis for decisionmakers in the face of insufficient evidence. J Clin Epidemiol, ahead-of-print. 

Manuscript available from publisher's website here.

















Tuesday, March 9, 2021

Expert Evidence: A Framework for Using GRADE When "No" Evidence Exists

To guide the formulation of clinical recommendations, GRADE relies on the use of direct or, if necessary, indirect evidence from peer-reviewed publications as well as the gray literature. However, in some cases, no such evidence may be found even after an extensive search has been conducted. A new paper - part of the informal GRADE Notes series in the Journal of Clinical Epidemiology - relays the results of piloting an "expert evidence" approach and provides key suggestions when using it.

As opposed to simply asking the panel members of a guideline to base their recommendations off of informal opinion, the expert evidence approach systematizes this process by eliciting the extent of their experience with certain clinical scenarios through quantitative survey methods. In this example, at least 50% of the panel members were free of conflicts of interest, with various countries and specialties represented. While members were not required to base their answers off of patient charts, the authors suggest that this can be used to further increase the rigor of the survey. 



As a result of the survey, the recommendations put forward reflected a cumulative 12,000 cases of experience. Because the members felt that at least some recommendation was necessary to help guide care - where the alternative would be to provide no recommendation at all - the guideline helped to fill a gap while indicating the current lack of high-quality published evidence for several clinical questions, which may help guide the production of higher-quality evidence and recommendations in the future. Importantly, by utilizing a survey approach to facilitate the formulation of recommendations, the authors note that it avoided the pitfall of "consensus-based" approaches to guideline development which can often manifest as simply reflecting the opinions of those with the loudest voices. 

Mustafa RA, Cuello Garcia CA, Bhatt M, Riva JJ, Vesely S, Wiercioch W, ... & HJ Schünemann. (2021). How to use GRADE when there is "no" evidence? A case study of the expert evidence approach. J Clin Epidemiol, in-press. 

Manuscript available from the publisher's website here. 


Thursday, February 25, 2021

The Use of GRADE in Systematic Reviews of Nutrition Interventions is Still Rare, but Growing

While the GRADE framework is used by over 100 health organizations to assess the certainty of evidence and guide the formulation of clinical recommendations, its use in the field of nutrition for these purposes is still sparse. A recent examination of all systematic reviews using GRADE in the ten highest-impact nutrition journals over the past five years provides insight and suggestions for moving the field forward in the use of GRADE for evidence assessment in systematic reviews of nutritional interventions.

Werner and colleagues identified 800 eligible systematic reviews, 55 (6.9%) of which used GRADE, and 47 (5.9%) of which rated the certainty of evidence specific to different outcomes. The number of these reviews using GRADE increased year-to-year, from two in 2015 to 23 in 2019. Reviews claiming to use a modification of GRADE were excluded from analysis.

Of the 811 identified cases of downgrading the certainty of evidence, and 31 cases of upgrading. Reviews of randomized controlled trials had a mean number of 1.6 domains downgraded per outcome, while reviews of non-randomized studies had a mean of 2.1. In about 6.5% of upgrading cases, this was done for unclear purposes not in line with GRADE guidance, such as upgrading for low risk of bias, narrow confidence intervals, or very low p-values. Reviews of non-randomized studies were more likely to have outcomes downgraded for imprecision and inconsistency, and less likely to have downgrades for publication bias than those of randomized studies. 

The authors conclude that while the use of GRADE in systematic reviews of nutritional interventions has grown over recent years based on this sample, continued education and training of nutrition researchers and experts can help improve the spread and quality of the application of GRADE to assess the certainty of evidence in this discipline.

Werner SS, Binder N, Toews I, et al. (2021). The use of GRADE in evidence syntheses published in high-impact-factor nutrition journal: A methodological survey. J Clin Epidemiol, in-press.

Manuscript available here. 











Wednesday, January 20, 2021

Help for Choosing Among Multiple Interventions Using GRADE

It is not uncommon for a health guideline to compare two or more interventions against one another. However, while sophisticated statistical approaches such as network meta-analyses allow us to compare these interventions head-to-head in terms of specified health outcomes, they do not take other important aspects of clinical decision-making into account, such as patient values and preferences, resource use, and equity considerations. A new paper from Piggott and colleagues aims to provide initial suggestions for using the GRADE evidence to decision (EtD) framework when choosing which of multiple interventions to recommend.

The authors identified a need for more direction when undertaking a multiple intervention comparison (MC) approach while working on recently released guidelines for the European Commission Initiative on Breast Cancer in which multiple screening intervals were compared against one another. Based on this experience, the group drafted a flexible yet transparency-minded framework to help guide similar efforts in the future, which was then added as a module in GRADE's official guideline development software, GRADEpro. 



The new module was pilot-tested for feasibility with several additional guidelines. The module allows the user to select and then compare multiple pairwise comparisons against one another (for instance, with one column for "Intervention 1 vs. Comparator 1" and "Intervention 2 vs. Comparator 2"). A five-star system is used to judge various components of the EtD, such as cost effectiveness, for each individual intervention and comparator, whereas a column on the right-hand side allows the user to input the relative importance of these components in decision-making.


Finally, the user can review all judgments across interventions and summatively recommend the most favorable intervention(s) overall.

Piggott T, Brozek J, Nowak A, et al. (2021). Using GRADE evidence to decision frameworks to choose from multiple interventions. J Clin Epidemiol 130:117-124.

Manuscript available from the publisher's website here.












Monday, November 16, 2020

Evidence Foundation Welcomes Four Scholars in First Virtual Workshop

In late October, the U.S. Grade Network held its thirteenth GRADE Guideline Development Workshop. Like any of the twelve workshops before it, there was much learning, discussion, and networking to be shared. However, unlike any workshop in the past, it was fully online.

Among the 45 attendees who participated in offices and living rooms from Brazil to Cyprus were four participants who attended the workshop free of charge as recipients of the Evidence Foundation scholarship. During a virtual Evening with the Fall 2020 Evidence Foundation Scholars, these four bright minds presented briefly on a proposal or current project designed to reduce bias in healthcare.

Dr. Stavros Antoniou, Chair of the European Association for Endoscopic Surgery Guidelines Subcommittee, discussed the tripartite Guideline Assessment Project (GAP) aimed at developing an extension of the AGREE II tool for surgical guidelines. In an exploratory analysis published earlier in 2018 (GAP I), Antoniou and colleagues assessed 67 surgical guidelines and reported that development of more than one guideline per year, the presence of a guideline committee, and the use of GRADE was associated with higher scores in AGREE II. Second, the group explored the reliability, internal consistency, and unidimensionality of the AGREE II tool when applied to surgical guidelines (GAP II). The group is now in the process of using the Delphi process to identify and finalize items for the surgical extension based on stakeholder input, pilot-testing the instrument, and assessing its validity (GAP III). Of the workshop. Dr. Antoniou noted, "participating in the GRADE Guideline Workshop as a scholar was an inspirational experience. It was fascinating to be trained by world-renowned experts, who have embraced us with true interest and conveyed their passion with quality in guideline development."

Jung Min Han, PharmD, MS, manages the development of guidelines for the American Academy of Dermatology. Her presentation reviewed her current project to update the organization's 2016 guidelines on the management of acne vulgaris using the GRADE framework. Ms. Han discussed the plan to organize two working groups, one to review and update the nine clinical questions from the previous guidelines, and the other to add additional new questions as needed. An updated search would then be run for the first set of questions to identify any newly published evidence since the original guidelines were developed; simultaneously, a novel systematic search would be conducted for the second group of questions. New recommendations would then be drafted following the GRADE methodology. Ms. Han stated, "The GRADE Workshop has trained me to confidently use GRADE in different scenarios where head-to-head data from randomized controlled trials are not available. The workshop was very well-structured with a concrete theme and a mix of lectures, small and large group discussions, meet the experts Q&A sessions, and real-world examples that challenged trainees in many ways."

Dr. Georgios Schoretsanitis of Zucker Hillside Hospital in Glen Oaks, New York presented on his work developing guidelines for therapeutic drug monitoring to optimize and tailor treatment for psychotherapeutic medications. Beginning in 2017, a series of recommendations for reference ranges for two commonly prescribed antipsychotic medications was developed, followed this year by an international joint consensus statement on blood levels to optimize antipsychotic treatment in clinical practice. "For long I have been interested in conducting systematic reviews and meta-analyses," said Dr. Schoretsanitis. "Attending the GRADE Guideline Workshop organized by the US GRADE Network gave me exactly what I was looking for: a unique chance to essentially deepen my knowledge on major methodological aspects during stimulating lectures by experts that have set the tone in the field. It was an intense experience far beyond acquiring knowledge, which I highly suggest to every methodologist."

Dr. Zeinab Hosseini, a Saskatchewan Health Research post-doctoral fellow at the University of Saskatchewan, discussed her work examining the impact of exercise interventions on osteoporosis. Because gender and sex affect the prognosis and management of the disease, guidelines that consider these differences are needed, she said. As part of her research under the advisement of Dr. Phil Chilibeck, she hopes to contribute further understanding in the field related to gender- and sex-specific considerations for exercise recommendations in patients with osteoporosis, and to help inform future guideline recommendations on this topic. "The US GRADE Network Workshop was an amazing opportunity for me as a post-doctoral fellow in health proving insight on how to think as a health researcher from early stages of research up to knowledge translation and dissemination and how to provide evidence-based recommendations to inform the public considering situations where the literature is scarce," said Dr. Hosseini. "There are top women and men scientists on the training panel who respond to questions using their experiences as member on different panels, which I think is unique."

The USGN facilitators pose for a virtual group photo with the four fall 2020 Evidence Foundation scholars. Click to enlarge.

The Evidence Foundation thanks all four scholars for attending and contributing their engagement and expertise to our 2020 fall workshop.

If interested in applying for a scholarship to future GRADE workshops, more details can be found here: https://evidencefoundation.org/scholarships.html. Please note the deadline for applications to our next workshop in Chicago, Illinois will be February 28, 2021.












Friday, October 30, 2020

U.S. Guideline-Producing Organizations Show Some Promise, Room for Improvement in their Application of GRADE

As many as one-third of guideline-producing health organizations in the United States report using the GRADE framework, but exactly how closely these organizations follow the key tenets of GRADE - such as using evidence summaries of each identified outcome to inform the overall certainty of evidence, and linking this certainty to a strength of recommendation - is a matter of debate.

In study by Dixon and colleagues published earlier this year in the Journal of Clinical Epidemiology, the authors set out to evaluate the use of GRADE in U.S.-based guidelines published between 2011 and 2018 and available in the National Guidelines Clearinghouse. Assessing up to three of the most recent guidelines from each of 135 identified U.S.-based organizations, the authors used several criteria to examine how closely each of the 67 resulting guidelines adhered to core GRADE concepts, including:

  • defining the certainty of evidence,
  • explicitly considering the GRADE domains when assessing the certainty of evidence, and
  • consistently defining the strength of resulting recommendations as strong or weak/conditional.
While most (89.6%) defined the certainty of evidence in a matter consistent with GRADE, only 10.4% explicitly reported examining certainty through all 8 GRADE criteria. Only 13.4% of guidelines assessing the certainty of evidence sourced from non-randomized trials reported assessing the potential reasons to upgrade the certainty of evidence (i.e., large magnitude of effect, dose-response gradient, and residual confounding). Finally, only about half (53.7%) provided an evidence profile or summary of findings table describing the assessments, and while reporting of the certainty of evidence and the balance between desirable and undesirable effects was most common (100% and 97%, respectively), explicit consideration of resource use and patients' values and preferences were also fairly common (73.1% and 77.6%, respectively) .The use of GRADE in line with the authors' established criteria appeared to grow somewhat more frequent over time, indicating a general trend toward proper use of GRADE.

Figure from Dixon et al. shows the relative reporting frequency of the various GRADE criteria for assessing certainty of evidence in years 2011-14 versus 2015-18, suggesting a trend for improved reporting over time. Click to enlarge.

The authors conclude that continued training of guideline developers and dissemination of education on the appropriate application of GRADE should further improve adherence, including the explicit consideration of all eight domains for assessing the certainty of evidence and of all aspects that inform the translation of this evidence into clinical recommendations.

Dixon C, Dixon PE, Sultan S, Mustafa R, Morgan RL, Murad MH, Falck-Ytter Y, and Dahm P. (2020). Guideline developers in the United States were inconsistent in applying criteria for appropriate Grading of Recommendations, Assessment, Development and Evaluation use. J Clin Epidemiol 124:193-199.

Manuscript available at the publisher's website here.


Thursday, July 30, 2020

Research Revisited: "Quality of Evidence is a Key Determinant for Making a Strong GRADE Guidelines Recommendation" (2015)

This month in 2015, Djulbegovic and colleagues published a paper that examined the impact of quality of evidence, balance between benefits and harms, patient values and preferences, and resource use (the four GRADE factors) on the strength of resulting clinical recommendations.

The four major GRADE factors that drive clinical recommendations 






















The authors circulated a survey among 18 members of a guideline panel of the American Association of Blood Banking (AABB) who had recently convened to develop guidelines for the use of prophylactic versus therapeutic platelet transfusion in patients with thrombocytopenia. Using the panel members’ assessments of the GRADE factors with regards to the evidence that had been presented and their resulting recommendations for strong or weak recommendations, a logistic regression was conducted in order to examine the relative impact of each of the four GRADE factors.

The guideline panel had reviewed the evidence for ten key questions. Overall, the consistency of judgments across panel members was good (Cronbach’s alpha = 0.86). Those questions with a high quality of evidence were 4.5 times more likely to result in a strong recommendation (p < 0001), whereas none of the three remaining GRADE factors were significantly associated with the strength of the resulting recommendations. Moreover, the model suggested that in cases where the quality of evidence was high, there was a 90% chance of the resulting recommendation being strong; when the quality of evidence was very low, this chance dropped to 10%.

The figure from Djulbegovic shows the associations between increasing quality of evidence and strength of resulting recommendations

The authors concluded that the quality of evidence was far and away the most important contributing factor to the resulting strength of recommendations, at least within the studied guideline panel. However, it’s important to note that patient values and preferences, the balance between benefits and harms, and issues of resource use should all be involved in the process of moving from evidence to decisions, but the relative nebulousness around these considerations and a lack of determined structures for eliciting the data they require (such as the use of a patient panel or survey) likely make them less impactful on the overall strength of recommendation. 

The Evidence-to-Decision framework, which makes these additional considerations more explicit in the formulation of recommendations, was introduced just a year later. It’s worth wondering whether the relative impact of the remaining three GRADE factors has changed since the introduction and adoption of this framework – perhaps presenting an opportunity to revisit this research.

Djulbegovic, B., Kumar, A., Kaufman, R.M., Tobian, A., and Guyatt, G.H. Quality of evidence is a key determinant for making a strong GRADE guidelines recommendation. J Clin Epidemiol 68(7): 727-732. 

Manuscript is available at the publisher's website here. 

Tuesday, June 2, 2020

Research Shorts: Use of GRADE for the Assessment of Evidence about Prognostic Factors

In addition to questions of interventions and diagnostic tests, GRADE can also be used to assess the certainty of evidence when it comes to prognostic factors. In part 28 of the Journal of Clinical Epidemiology’s GRADE series published earlier this year, Foroutan and colleagues provide guidance for applying GRADE to a body of evidence of prognostic factors.

The Purpose of Prognostic Studies

GRADE may be applied to a body of evidence, separated by individual prognostic factors instead of outcomes, for one of two reasons. The first is a non-contextualized setting, such as when the certainty of evidence surrounding prognostic factors is being evaluated for application within research planning and analysis (e.g., determining which factors are best to use when stratifying for randomization). The second is a contextualized setting, when the certainty of evidence surrounding prognostic factors is used to help inform clinical decisions.

Establishing the Certainty of Evidence

Unlike when grading the certainty of evidence of an intervention, when assessing prognostic evidence, the overall certainty for observational studies starts out as HIGH. This is because the patient population is likely to be more representative studies than in RCTs, when eligibility criteria may place artificial restrictions on the characteristics of patients. Certainty may then be rated down based on the five traditional domains:
  • Risk of bias tools and instruments such as QUality In Prognosis Studies (QUIPS) and Prediction model Risk Of Bias ASsessment Tool (PROBAST) may be helpful here. When teasing out the effect of each potential factor, consider utilizing some form of multivariate analysis that accounts for dependence between several different prognostic factors.
  • Inconsistency can be examined via visual tests of the variability between individual point estimates and the overlap of confidence intervals; statistical tests such as i2 are likely to be less helpful, as they can often be inflated when large studies lead to particularly narrow Cis. As always, potential explanations for any observed heterogeneity should be considered a priori.
  • Imprecision will depend on whether the setting is contextualized, in which case it will depend on the relationship between the confidence interval and the previously set clinical decision threshold, or non-contextualized, in which case the threshold will most likely represent the line of no effect.
  • Indirectness should be based on a comparison of the PICOs for the clinical question at hand, and those addressed in the meta-analyzed studies.
  • Publication bias can be assessed via visually exploring a funnel plot or the use of appropriately applied statistical tests.
Foroutan F, Guyatt G, Zuk V, Vandvik PO, Alba AC, Mustafa R, Vernooij R et al. GRADE guidelines 28: Use of GRADE for the assessment of evidence about prognostic factors: Rating certainty in identification of groups of patients with different absolute risks. J Clin Epidemiol 121; 62-70.

Manuscript available from the publisher's website here. 




Friday, May 22, 2020

Research Shorts: Assessing the Certainty of Diagnostic Evidence, Pt. II: Inconsistency, Imprecision, and Publication Bias

Earlier this week, we discussed a recent publication of the GRADE series in Journal of Clinical Epidemiology that provides guidance for assessing risk of bias and indirectness across a body of evidence of diagnostic test accuracy. In this post, we’ll follow-up with continued guidance (published in Part II) for the rest of the GRADE domains.

Inconsistency

Unexplained inconsistency should be evaluated separately for the findings on test specificity and test sensitivity. When a meta-analysis is available, both the visual and quantitative markers of inconsistency can be used in a similar fashion to a meta-analysis of intervention studies. If differences between studies related to any of the PICO elements is suspected as an explanation for observed heterogeneity, exploration via subgroup analyses may be appropriate.

Imprecision

Again, imprecision of a test’s sensitivity and specificity should be evaluated separately. As with assessments of interventional evidence, evaluation of imprecision across a body of test accuracy studies entails the consideration of the width of the confidence interval as well as the number of events (specifically, the number of patients with the disease and the number of positive tests for sensitivity, and the number of patients without the disease and the number of negative tests for specificity).

In contextualized settings, when one end of the confidence interval may lead to the use of the testing strategy while the other end would not, then imprecision is likely present. It may be helpful to set priori a threshold through which a confidence interval should not cross in order for the test to have sufficient value.

Publication Bias

The use of traditional funnel plot assessments (e.g., Egger’s or Begg’s test) on a body of test accuracy studies is more likely to result in undue suspicion of publication bias than when applied to a body of therapeutic studies. While other sophisticated statistical assessments are available (e.g., Deeks’ test, trim and fill), systematic review and health technology assessment (HTA) authors may choose to base a judgment of publication bias on the knowledge of the existence of unpublished studies. If studies published by for-profit entities or those with precise estimates claiming high test accuracy despite small sample sizes exist, publication bias may also be suspected.

Upgrading the Certainty of Evidence ("Rating Up")

As with an assessment of interventional evidence, there may be reasons to upgrade the certainty of evidence in the face of highly convincing links between the use of a test and the likelihood and/or magnitude of an observed outcome. The diagnostic test accuracy equivalent of a dose-response gradient – the Receiving Operator Characteristic, or ROC curve – may be used to assess this potential upgrader.

Schünemann H, Mustafa RA, Brozek J, Steingart KR, Leeflang M, Murad MH, Bossuyt P, et al. GRADE guidelines 21 pt. 2. Test accuracy: inconsistency, imprecision, publication bias, and  other domains for rating the certainty of evidence and presenting it in evidence profiles and summary of findings tables. J Clin Epidemiol  2020 Feb 10. pii: S0895-4356(19)30674-2. doi: 10.1016/j.jclinepi.2019.12.021. [Epub ahead of print].

Manuscript available here on the publisher's site.