Systematic review

Characteristics

A systematic review aims to provide a complete, exhaustive summary of current literature relevant to a research question. The first step in conducting a systematic review is to perform a thorough search of the literature for relevant papers. The Methodology section of a systematic review will list all of the databases and citation indexes that were searched such as Web of Science, Embase, and PubMed and any individual journals that were searched. The titles and abstracts of identified articles are checked against pre-determined criteria for eligibility and relevance to form an inclusion set. This set will relate back to the research problem. Each included study may be assigned an objective assessment of methodological quality preferably by using methods conforming to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (the current guideline) or the high quality standards of Cochrane collaboration.

Systematic reviews often, but not always, use statistical techniques (meta-analysis) to combine results of eligible studies, or at least use scoring of the levels of evidence depending on the methodology used. An additional rater may be consulted to resolve any scoring differences between raters. Systematic review is often applied in the biomedical or healthcare context, but it can be applied in any field of research. Groups like the Campbell Collaboration are promoting the use of systematic reviews in policy-making beyond just healthcare.

A systematic review uses an objective and transparent approach for research synthesis, with the aim of minimizing bias. While many systematic reviews are based on an explicit quantitative meta-analysis of available data, there are also qualitative reviews which adhere to standards for gathering, analyzing and reporting evidence. The EPPI-Centre has been influential in developing methods for combining both qualitative and quantitative research in systematic reviews. The PRISMA statement suggests a standardized way to ensure a transparent and complete reporting of systematic reviews, and is now required for this kind of research by more than 170 medical journals worldwide.

Recent developments in systematic reviews include realist reviews, and the meta-narrative approach. These approaches try to overcome the problems of methodological and epistemological heterogeneity in the diverse literatures existing on some subjects.

Stages

The main stages of a systematic review are:

1. Defining a question and agreeing an objective method.
2. A search for relevant data from research that matches certain criteria. For example, only selecting research that is good quality and answers the defined question.
3. 'Extraction' of relevant data. This can include how the research was done (often called the method or 'intervention'), who participated in the research (including how many people), how it was paid for (for example funding sources) and what happened (the outcomes).
4. Assess the quality of the data by judging it against criteria identified at the first stage.
5. Analyse and combine the data (using complex statistical methods) which give an overall result from all of the data. This combination of data can be visualised using a blobbogram (also called a forest plot). The diamond in the blobbogram represents the combined results of all the data included. Because this combined result uses data from more sources than just one data set, it's considered more reliable and better evidence, as the more data there is, the more confident we can be of conclusions.

Once these stages are complete, the review may be published, disseminated and translated into practice after being adopted as evidence.

Cochrane Collaboration

The Cochrane Collaboration is a group of over 37,000 specialists in healthcare who systematically review randomised trials of the effects of prevention, treatments and rehabilitation as well as health systems interventions. When appropriate, they also include the results of other types of research. Cochrane Reviews are published in The Cochrane Database of Systematic Reviews section of the Cochrane Library. The 2015 impact factor for The Cochrane Database of Systematic Reviews was 6.103, and it was ranked 12th in the “Medicine, General & Internal” category. There are six types of Cochrane Review:

1. Intervention reviews assess the benefits and harms of interventions used in healthcare and health policy.
2. Diagnostic test accuracy reviews assess how well a diagnostic test performs in diagnosing and detecting a particular disease.
3. Methodology reviews address issues relevant to how systematic reviews and clinical trials are conducted and reported.
4. Qualitative reviews synthesize qualitative and quantitative evidence to address questions on aspects other than effectiveness.
5. Prognosis reviews address the probable course or future outcome(s) of people with a health problem.
6. Overviews of Systematic Reviews (OoRs) are a new type of study in order to compile multiple evidence from systematic reviews into a single document that is accessible and useful to serve as a friendly front end for the Cochrane Collaboration with regard to healthcare decision-making.

The Cochrane Collaboration provides a handbook for systematic reviewers of interventions which "provides guidance to authors for the preparation of Cochrane Intervention reviews." The Cochrane Handbook outlines eight general steps for preparing a systematic review:

1. Defining the review question(s) and developing criteria for including studies
2. Searching for studies
3. Selecting studies and collecting data
4. Assessing risk of bias in included studies
5. Analysing data and undertaking meta-analyses
6. Addressing reporting biases
7. Presenting results and "summary of findings" tables
8. Interpreting results and drawing conclusions

The Cochrane Handbook forms the basis of two sets of standards for the conduct and reporting of Cochrane Intervention Reviews (MECIR - Methodological Expectations of Cochrane Intervention Reviews)

The Cochrane Collaboration logo visually represents how results from some systematic reviews can be explained. The lines within illustrate the summary results from an iconic systematic review showing the benefit of corticosteroids, which 'has probably saved thousands of premature babies'.

The Campbell Collaboration

The Campbell Collaboration is one of a number of groups promoting evidence-based policy in the social sciences. The Campbell Collaboration "helps people make well-informed decisions by preparing, maintaining and disseminating systematic reviews in education, crime and justice, social welfare and international development.^[2] It is a sister initiative of Cochrane. The Campbell Collaboration was created in 2000 and the inaugural meeting in Philadelphia, USA, attracted 85 participants from 13 countries.

Strengths and weaknesses of systematic reviews

While systematic reviews are regarded as the strongest form of medical evidence, a review of 300 studies found that not all systematic reviews were equally reliable, and that their reporting can be improved by a universally agreed upon set of standards and guidelines. A further study by the same group found that of 100 systematic reviews monitored, 7% needed updating at the time of publication, another 4% within a year, and another 11% within 2 years; this figure was higher in rapidly changing fields of medicine, especially cardiovascular medicine. A 2003 study suggested that extending searches beyond major databases, perhaps into grey literature, would increase the effectiveness of reviews.

Roberts and colleagues highlighted the problems with systematic reviews, particularly those conducted by the Cochrane Collaboration, noting that published reviews are often biased, out of date and excessively long. They criticized Cochrane reviews as not being sufficiently critical in the selection of trials and including too many of low quality. They proposed several solutions, including limiting studies in meta-analyses and reviews to registered clinical trials, requiring that original data be made available for statistical checking, paying greater attention to sample size estimates, and eliminating dependence on only published data.

Some of these difficulties were noted early on as described by Altman: "much poor research arises because researchers feel compelled for career reasons to carry out research that they are ill equipped to perform, and nobody stops them." Methodological limitations of meta-analysis have also been noted. Another concern is that the methods used to conduct a systematic review are sometimes changed once researchers see the available trials they are going to include. Bloggers have described retractions of systematic reviews and published reports of studies included in published systematic reviews.

Systematic reviews are increasingly prevalent in other fields, such as international development research. Subsequently, a number of donors – most notably the UK Department for International Development (DFID) and AusAid – are focusing more attention and resources on testing the appropriateness of systematic reviews in assessing the impacts of development and humanitarian interventions.