Reviews of Screening Interventions Often Fail to Include Relevant Harms

With any given intervention comes a set of both potential desirable as well as undesirable effects, and proper consideration should be given to both in the context of clinical decision-making. However, our knowledge about potential undesirable effects (or "harms") of an intervention depends on the availability of the evidence, just as it does with the potential benefits. A recent systematic review of reviews for screening interventions suggests that the way evidence for harms is synthesized may not follow the same rigor and depth as for an intervention's potential desirable effects, limiting our ability to throughly weigh the two against one another when making clinical decisions to inform screening behaviors.

In the January 2021 issue of Journal of Clinical Epidemiology, Johanssen and colleagues systematically searched and screened 47 Cochrane reviews, making note of those that reported including potential harms as outcomes within the search strategy, even if no available evidence was ultimately found. Overdiagnosis was only included in 15% of the 39 reviews in which the Johanssen and colleagues deemed it a potentially relevant outcome; overtreatment was mentioned in 16% of eligible reviews. The inclusion of secondary harm outcomes in potentially eligible reviews ranged from 7% (incidental findings) to 91% (all-cause mortality). While psychosocial consequences was discussed as a potential outcome in a majority (64%) of eligible reviews, the data for this outcome were often not synthesized. 

Overall, reviews were less likely to meta-analyze or assess the risk of bias for evidence around harms than for benefits. Two-thirds (67%) of summary of findings tables, however, did not include any harms as outcomes; further, 42% of abstracts and 58% of plain language summaries did not mention any harms. 

The authors conclude that these findings demonstrate a need for a "broad collaboration" to develop reporting guidelines and core outcome sets that will ensure the more thorough and rigorous reporting of harms outcomes in screening studies. Through a consensus process involving a diverse set of stakeholders including clinicians, methodologists, policymakers, and medical ethicists, improved standards can be set for the reporting of all outcomes of screening interventions that are of potential relevance to patients.

Johansson M, Borys F, Peterson H, et al. 2021. Addressing the harms of screening - A review of outcomes in Cochrane reviews and suggestions for next steps. J Clin Epidemiol 129:68-73.

Manuscript available from the publisher's website here. 

Evidence Foundation Scholar Update: Christian Kershaw

Dr. Christian Kershaw, a health policy analyst with CGS Administrators, LLC, attended the fall 2019 GRADE Guideline Development Workshop free of charge as an Evidence Foundation scholar. As part of the scholarship, Dr. Kershaw submitted and then presented on a proposal for reducing bias in healthcare, focusing on how to build and lead cross-functional teams (blog post here). We followed up with Dr. Kershaw at one year post-workshop to see what's happened since her attendance.

"In my work as a health policy analyst for a Medicare Fee-for-Service contractor, I use my research background to help with the evaluation of scientific literature on products being considered for coverage by Medicare. My team was created because of a call for transparency in how Medicare makes coverage decisions. I received the Evidence Foundation scholarship to the Fall 2019 GRADE conference and attended along with my team members to learn GRADE methodology. Our goal was to determine if implementation of GRADE methodology would standardize our literature evaluation process for coverage decisions," said Dr. Kershaw.

"At the conference I presented on the benefits of establishing cross-functional teams. By joining forces with team members with a heterogeneous set of skills and backgrounds, we can leverage individual strengths and encourage innovation to reach a common goal. In my work, I collaborate with MDs, RNs, and policy experts with a well-rounded knowledge base of clinical standards, Medicare processes, and coverage policies. After learning GRADE methodology, we implemented the use of GRADE to improve the transparency and standardization of our process for writing coverage policies. our team has now completed two GRADE workshops, and we are constantly working to improve our use of this methodology. We have found that the use of GRADE helps our cross-functional team improve our ability to systematically make coverage determinations based on scientific evidence."

Stay tuned for future updates from other past Evidence Foundation scholars like Dr. Kershaw and the exciting work they are doing to improve the application of GRADE methodology and evidence-based medicine.

If you are interested in learning more about GRADE and attending the workshop as a scholarship recipient, applications for our upcoming workshop next May are now open. The deadline to apply is February 28, 2021. Details can be found here. 

No Single Definition of a Rapid Review Exists, but Several Common Themes Emerge

"The only consensus around [a rapid review] definition," write Hamel and colleagues in a review published in the January 2021 issue of the Journal of Clinical Epidemiology, "is that a formal definition does not exist."

In their new review, Hamel et al. sifted through 216 rapid reviews and 90 methodological articles published between 2017 and 2019 to better understand the existing definitions and use of the term "rapid review," identifying eight common themes among them all.

The figure below from the publication shows the relative usage of these themes throughout the relevant identified articles.

In summary of all definitions examined in the review, the authors suggest the following broad definition of a rapid review: "a form of knowledge synthesis that accelerates the process of conducting a traditional systematic review through streamlining or omitting a variety of methods to produce evidence in a resource-efficient manner."

To complicate matters further, Hamel and colleagues also found that reviews meeting these general criteria may not always go by the term "rapid." For instance, the term "restricted review" fits many of these same parameters, but is not necessarily defined by the amount of time from inception to publication. However, the lack of an as-yet agreed-upon definition of a "rapid review" may ultimately hamper authors and potential end-users of these products, as the accepted legitimacy of such reviews may depend upon a common understanding of their standards and methodological frameworks. In addition, the range of rigor and specific protocols continues to vary widely between products labeled as "rapid reviews." Until there is a broader consensus of the definition of a rapid review and what, exactly, it entails, this working definition and associated themes provide insight into the current state of the art.

Check out our related post on the two-week systematic review here.

Hamel C, Michaud A, Thuku M, Skidmore B, Stevens A, Nussbaumer-Streit B, and Garritty C. (2020). Defining rapid reviews: A systematic scoping review and thematic analysis of definitions and defining characteristics of rapid reviews. J Clin Epidemiol 129: 74-85.

Manuscript available from the publisher's website here. 

Assessing the Reliability of Recently Developed Risk of Bias Tools for Non-Randomized Studies

Risk of bias is one of the five domains to be considered when assessing the certainty of evidence across a body of studies, and is the only domain which must first be assessed on the individual study level. While several risk of bias assessment tools exist for non-randomized studies (NRS; or observational trials), two of the most recently introduced are the Risk of Bias in Non-Randomized Studies of Interventions (ROBINS-I, developed in 2016) and the Risk of Bias instrument for NRS of Exposures (ROB-NRSE, developed in 2018). Assessment of the risk of bias in a systematic review off of which a guideline is based should ideally be conducted independnelty by at least two reviewers. Given this scenario, how likely is it that the two reviewers' assessments will agree sufficiently with one another?

In a recently published paper by Jeyaraman and colleagues, a multi-center group of collaborators assessed both the inter-rater reliability (IRR) and interconsensus reliability (ICR) of these tools based on a previously published cross-sectional study protocol. The seven reviewers had a median of 5 years of experience assessing risk of bias, and two pairs of reviewers assessed risk of bias using each tool. IRR was used to assess reliability within pairs, while ICR assessed reliability between the pairs. The time burden was also assessed by recording the amount of time required to assess each included study and to come to a consensus. For the overall assessment of bias, IRR was rated as "Poor" (Gwet's agreement coefficient of 0%) for the ROBINS-I tool and "slight" (11%) for the ROB-NRSE tool, whereas the ICR was rated as "poor" for both ROBIN-I (7%) and ROB-NRSE (0%). The average evaluator time burden was over 48 minutes for the ROBINS-I tool and almost 37 minutes for the ROB-NRSE.

Click to enlarge.

Click to enlarge.

The authors note that overall, ROBINS-I tended to have a better IRR as well as ICR, both of which may be due in part to poorer reporting quality in exposure studies. In addition, simplification of related guidance documents for applying the tool and increased training for reviewers looking to use the ROBINS-I and ROB-NRSE tools to assess risk of bias in non-randomized studies may improve agreement considerably while cutting down on the time required to apply the tool correctly to each individual study.

Jeyaraman MM, Rabbani R, Copstein L, Robson RC, Al-Yousif N, Pollock M, ... & Abou-Setta AM. (2020). Methodologically rigorous risk of bias tools for nonrandomized studies had low reliability and high evaluator burden. J Clin Epidemiol 128:140-147.

Manuscript available from the publisher's web site here.