The Cochrane Risk of Bias (RoB) tool for randomized controlled trials (RCTs) is commonly used in both Cochrane and non-Cochrane systematic reviews as a standardized way to assess and report the risk of bias within a study or a body of evidence. The tool comprises seven domains, each representing a potential source of bias within the design or execution of an RCT. Judgments for each domain (for instance, allocation concealment, or selective outcome reporting) are made between whether the study possessed a low, high, or unclear risk of bias from that source.
A new review of non-Cochrane systematic reviews (NCSRs) published in this month’s edition of the Journal of Clinical Epidemiology reports that the use of the Cochrane RoB tool in these reviews is incomplete or inadequate in most cases. Within 508 eligible systematic reviews that used the original (2011) Cochrane RoB tool published through 3 July 2018, the majority (85%) reported the analysis of risk of bias; within these papers, about half (53%) used the Cochrane tool specifically, leaving a total of 269 reviews for further analysis.
 |
| A non-negligible minority of studies included in the review by Puljak et al. either did not include certain domains of the Cochrane RoB tool, or did not report which domains were used. Only 40% of the reviews analyzed RoB through all seven domains. Click to enlarge. |
Less than two-thirds (60%) of the 269 included reviews used all seven domains of the Cochrane tool, report Puljak and colleagues, and only 16 of the included reviews (5.9%) reported both a judgment and a comment explaining each judgment either within the manuscript or in a supplementary file. Within these 15 reviews, the proportion of inadequate judgments (either those in which the comment was not in line with the judgment or in which there was no supporting comment) ranged from 25% (Other Bias domain) to 65% (Selective Reporting Bias domain). The reviews “rarely” included full tables illustrating the RoB judgments for the different domains.
The authors’ findings highlight that both a judgment (low/high/unclear risk of bias) as well as a comment explaining the judgment within each domain should be included in systematic reviews that report use of the Cochrane RoB tool.
Puljak, L., Ramic, I., Naharro, C.A., Brezova, J., Lin, Y.C., Surdila, A.A,.... & Salvado, M.S. Cochrane risk of bias tool was used inadequately in the majority of non-Cochrane systematic reviews. J Clin Epidemiol, 2020; 123: 114-119.
Manuscript available from publisher's website here.
