Skip to Main Content

Systematic Reviews and Meta Analysis

A resource for finding data sources, filters, and standards to support systematic searches of the biomedical literature.
  • Risk of Bias - In a systematic review or meta-analysis, the Risk of Bias (RoB) process is used to assess the methodological qualities of the included studies after the full text selection and data extraction has been completed. Analysis at the level of a group or body of studies can often verify and quantify the direction and magnitude of bias caused by methodological problems. It is unnecessary to conduct for most Scoping Reviews.
  • Quality Assessment -  Aims to assess the "risks to rigor" present in a primary qualitative study by examining a study's methodological strengths and limitations, including research conduct. A Quality Assessment is more common in older Systematic Reviews, and not required for most reviews. However, Qualitative Evidence Syntheses (QES) do require a Quality Assessment. The criteria for assessing the quality of included studies should be determined by the team, but Cochrane recommends the CASP checklist for Qualitative Studies as a general-purpose tool that can be used for Quality Assessment. Additional tools like CONQUAL (Munn 2014) and GRADE-CERQUAL (Lewin et al 2015) have been developed to assess confidence in qualitative evidence by looking at several domains each.
  • Critical Appraisal - Assesses the validity, results, and clinical applicability of a study for applying Evidence-Based Medicine (EBM). Methodological assessment may be part of the appraisal, but is not required. Critical Appraisal typically assumes there is a patient that the study will need to consider in the local clinical environment. Commonly used critical appraisal tools include the Critical Appraisal Skills Programme (CASP) checklists.

Important Note:  Some people and even organizations may use these terms interchangeably, however each has a specific intention and purpose in Evidence-Based Medicine (EBM). For instance, the JBI Critical Appraisal tools are technically a suite Risk of Bias (RoB) assessment tools. Additionally certain types of critical appraisal or quality assessment may be appropriate as a RoB tool, depending on its application.

Risk of Bias Methods

Each risk of bias tool has benefits and drawbacks to use. Here are some considerations for conducting a risk of bias assessment:

  • Typically a Risk of Bias tool is designed to assess bias for a single type of study (such as a Randomized Controlled Trial)
  • Some Risk of Bias tools can be adapted to other types of studies, such as ROBINS-I, which is used to assess Non-randomized studies of intervention (NSRI)
  • For observational or non-experimental studies, it may be necessary to use an additional tool to assess risk of bias, such as the JBI critical appraisal tools, which features targeted RoB assessments for a variety of study types.
  • Risk of Bias assessment should be completed by at least two separate people and reported as a table (AHRQ)
  • For Risk of Bias of RCTs and NSRIs, prior to undertaking the RoB assessment, the team will need to select which specific results from the included trials to assess (Cochrane)
    • Trials usually contribute multiple results to a systematic review, several risk-of-bias assessments may be needed for each trial, although it is unlikely to be feasible to assess every result for every trial in the review (Cochrane)
    • An approach that focuses on the main outcomes of the review (the results contributing to the review’s ‘Summary of findings’ table) may be the most appropriate approach (Cochrane)
  • Each article should be assessed based on the domains of the RoB tool, then the team should make a judgement about the overall risk-of-bias and categorize thusly:
    • Low risk-of-bias - The trial is judged to be at low risk of bias for all domains for this result.
    • Some Concerns - The trial is judged to raise some concerns in at least one domain for this result, but not to be at high risk of bias for any domain.
    • High Risk of Bias - The trial is judged to be at high risk of bias in at least one domain for this result.
      The trial is judged to have some concerns for multiple domains in a way that substantially lowers confidence in the result. (cochrane)
  • If your review features multiple study types:
    • Use 1 tool for all studies:  Offers direct comparison and consistent data. However, rigidity of tools may overlook study-specific domains.
    • Use different tools for each type of study:  Each tool has a benefit to use and will contain domains specific to the type of study. However, the data won't provide easy comparisons due to inconsistencies.
  • It is inadvisable remove articles from inclusion due to the results from a Risk of Bias assessment, as that will introduce bias into your review and skew results.

Reporting the Risk of Bias

Risk of Bias Report example from a systematic review appendixAccording to PRISMA 2020, [Item 11] the manuscript of the review should clearly name the name of the tool and version used. If modifications were made, it should be indicated as well. Report the methods and steps used to assess bias in the body of the manuscript.

The Risk of Bias itself should be reported in a table. Most frequently, reviews use a red/green/yellow "traffic light" approach to coding bias. While others may incorporate symbols (*, +, -, ?) either separately or in addition to the traffic light coding to indicate concerns about levels of bias. Other reports may use numerical values to indicate levels of bias, but these have a tendency to be more vague about the results of the RoB assessment.

Additionally, including study data from the review authors in the appendix adds further transparency to the process.

         Countway Three Shield Logo   

Countway Library
10 Shattuck St, Boston MA 02115 | (617) 432-2136

Harvard Library | Privacy Policy | Harvard Digital Accessibility Policy
Copyright © 2020 President and Fellows of Harvard College. All rights reserved.