Diagnostic Test Accuracy Meta-Analyses Are Often Missing Information Required for Reproducibility

Reproducibility of results is considered a key tenet of the scientific process. When results of a study are reproduced by others using the same protocol, there is less chance that the original results observed were due human or random error. Testing the reproducibility of evidence syntheses (e.g., meta-analyses) is just as important as for individual trials.

In a paper published earlier this month, Stegeman and Leeflang undertook the task of testing the reproducibility of meta-analyses of diagnostic test accuracy. The authors identified 51 eligible meta-analyses published in January 2018. In 19 of these, sufficient information was provided in the text of the study to reproduce the 2x2 tables of the individual studies included; in the remaining 32, only estimates were provided in the text. In 17 of these 32, the authors located primary data to attempt reproducibility. When attempting to reproduce the meta-analyses of the 51 identified papers, reproducibility was only achieved 28% of the time; none of the 17 papers for which 2x2 tables were not provided were reproducible.

Click to enlarge.

Only 14 (27%) of the 51 articles provided full search terms. In nearly half (25) of the included reviews, at least one of the full texts of included references could not be located; in 12, at least one title or abstract could not be located. Overall, of the 51 included reviews, only one was deemed fully reproducible by providing a full protocol, 2x2 tables, and the same summary estimates as the authors.

The authors conclude with a call for increased prospective registration of protocols and improved reporting of search terms and methods. The application of the 2017 PRISMA statement for diagnostic test accuracy is a helpful tool for any aspiring author of a diagnostic test accuracy meta-analysis to improve the reporting and reproducibility of results.

Stegeman I. and Leeflang M.M.G. (2020). Meta-analyses of diagnostic test accuracy could not be reproduced. J Clin Epidemiol 127:161-166.

Manuscript available at the publisher's website here. 

Practical Tips for Finding and Assessing Patient Survey Data

 An essential part of translating a body of evidence into a clinical recommendation within the GRADE framework is the consideration of patients' values and preferences. Not only should the likely treatment preferences and values placed on outcomes among the patient population be considered; if there is likely a great amount of variability within these, this may also influence the ultimate strength of recommendation.

Guideline panels and public health decision-makers may use self-reported patient survey data to better understand the range of patient values and preferences when formulating recommendations or policies. However, like all sources of evidence, patient surveys may be at risk for specific sources of bias which can ultimately affect the results. What should decision-makers look out for when applying patient survey data to a recommendation for care? In a recently published paper, Santesso and colleagues propose a practical guide for finding, interpreting, and applying patient data to better inform healthcare decision-making.

Click to enlarge.

Because 97% of published surveys have been found to use the words "survey" or "questionnaire" in the title, the authors suggest using these terms in title, abstract, and topic fields when conducting a search for relevant data. When assessing the risk of bias of a given survey, decision-makers should ask whether the population was adequately representative of the patient population in question, taking care to consider the use of random sampling and the potential impact of nonresponse. A survey should also be assessed for whether it measures the intended constructs adequately. Survey authors should report the variability around reported measures whenever possible, and these data can be used to judge the overall variability in patient values and preferences. Finally, decision-makers should take care to discern how directly the survey data applies to the patient population in question; the table of survey respondent characteristics is a useful place from which to draw judgments of directness.

Using these helpful and practical points of guidance, guideline panel members and clinical decision-makers can better inform their retrieval, critical appraisal, and application of patient survey data to important healthcare questions, ultimately resulting in more informed guidelines and policies.

Santesso N, Akl E, Bhandari M, Busse JW, Cook DJ, Greenhalgh T, Muti P, Schünemann H, and Guyatt G. (2020). A practical guide for using a survey about attitudes and behaviors to inform health care decision making. J Clin Epidemiol 128:93-100.

Manuscript available from the publisher's website here. 

Evidence Foundation Welcomes Four Scholars in First Virtual Workshop

In late October, the U.S. Grade Network held its thirteenth GRADE Guideline Development Workshop. Like any of the twelve workshops before it, there was much learning, discussion, and networking to be shared. However, unlike any workshop in the past, it was fully online.

Among the 45 attendees who participated in offices and living rooms from Brazil to Cyprus were four participants who attended the workshop free of charge as recipients of the Evidence Foundation scholarship. During a virtual Evening with the Fall 2020 Evidence Foundation Scholars, these four bright minds presented briefly on a proposal or current project designed to reduce bias in healthcare.

Dr. Stavros Antoniou, Chair of the European Association for Endoscopic Surgery Guidelines Subcommittee, discussed the tripartite Guideline Assessment Project (GAP) aimed at developing an extension of the AGREE II tool for surgical guidelines. In an exploratory analysis published earlier in 2018 (GAP I), Antoniou and colleagues assessed 67 surgical guidelines and reported that development of more than one guideline per year, the presence of a guideline committee, and the use of GRADE was associated with higher scores in AGREE II. Second, the group explored the reliability, internal consistency, and unidimensionality of the AGREE II tool when applied to surgical guidelines (GAP II). The group is now in the process of using the Delphi process to identify and finalize items for the surgical extension based on stakeholder input, pilot-testing the instrument, and assessing its validity (GAP III). Of the workshop. Dr. Antoniou noted, "participating in the GRADE Guideline Workshop as a scholar was an inspirational experience. It was fascinating to be trained by world-renowned experts, who have embraced us with true interest and conveyed their passion with quality in guideline development."

Jung Min Han, PharmD, MS, manages the development of guidelines for the American Academy of Dermatology. Her presentation reviewed her current project to update the organization's 2016 guidelines on the management of acne vulgaris using the GRADE framework. Ms. Han discussed the plan to organize two working groups, one to review and update the nine clinical questions from the previous guidelines, and the other to add additional new questions as needed. An updated search would then be run for the first set of questions to identify any newly published evidence since the original guidelines were developed; simultaneously, a novel systematic search would be conducted for the second group of questions. New recommendations would then be drafted following the GRADE methodology. Ms. Han stated, "The GRADE Workshop has trained me to confidently use GRADE in different scenarios where head-to-head data from randomized controlled trials are not available. The workshop was very well-structured with a concrete theme and a mix of lectures, small and large group discussions, meet the experts Q&A sessions, and real-world examples that challenged trainees in many ways."

Dr. Georgios Schoretsanitis of Zucker Hillside Hospital in Glen Oaks, New York presented on his work developing guidelines for therapeutic drug monitoring to optimize and tailor treatment for psychotherapeutic medications. Beginning in 2017, a series of recommendations for reference ranges for two commonly prescribed antipsychotic medications was developed, followed this year by an international joint consensus statement on blood levels to optimize antipsychotic treatment in clinical practice. "For long I have been interested in conducting systematic reviews and meta-analyses," said Dr. Schoretsanitis. "Attending the GRADE Guideline Workshop organized by the US GRADE Network gave me exactly what I was looking for: a unique chance to essentially deepen my knowledge on major methodological aspects during stimulating lectures by experts that have set the tone in the field. It was an intense experience far beyond acquiring knowledge, which I highly suggest to every methodologist."

Dr. Zeinab Hosseini, a Saskatchewan Health Research post-doctoral fellow at the University of Saskatchewan, discussed her work examining the impact of exercise interventions on osteoporosis. Because gender and sex affect the prognosis and management of the disease, guidelines that consider these differences are needed, she said. As part of her research under the advisement of Dr. Phil Chilibeck, she hopes to contribute further understanding in the field related to gender- and sex-specific considerations for exercise recommendations in patients with osteoporosis, and to help inform future guideline recommendations on this topic. "The US GRADE Network Workshop was an amazing opportunity for me as a post-doctoral fellow in health proving insight on how to think as a health researcher from early stages of research up to knowledge translation and dissemination and how to provide evidence-based recommendations to inform the public considering situations where the literature is scarce," said Dr. Hosseini. "There are top women and men scientists on the training panel who respond to questions using their experiences as member on different panels, which I think is unique."

The USGN facilitators pose for a virtual group photo with the four fall 2020 Evidence Foundation scholars. Click to enlarge.

The Evidence Foundation thanks all four scholars for attending and contributing their engagement and expertise to our 2020 fall workshop.

If interested in applying for a scholarship to future GRADE workshops, more details can be found here: https://evidencefoundation.org/scholarships.html. Please note the deadline for applications to our next workshop in Chicago, Illinois will be February 28, 2021.