Writing · Advanced
Plan and report a systematic or scoping review
Plan selection, extraction, assessment and synthesis for a systematic or scoping review.
17 min read · Oct 26, 2026
Key points
- Match review type to question.
- Distinguish records, reports and studies.
- Use appropriate reporting guidance.
Contents
Plan selection, extraction, assessment and synthesis for a systematic or scoping review.
A systematic review is a research project whose evidence comes from existing studies. Its credibility depends on decisions about what to include, how to find it, how to assess it and how to combine it. A scoping review also uses explicit methods, but usually asks what kinds of evidence or concepts exist and how they are distributed. Neither becomes rigorous simply by having many references, a flow diagram or statistical software.
This tutorial explains the complete path from question to manuscript. All study counts and numerical demonstrations below are synthetic teaching examples. They do not describe an actual review, and the sample search expression has not been run as a research search.
Choose the review type before building the search
Use the question to choose the method. “Does a defined feedback intervention improve writing scores compared with usual teaching?” may suit a systematic review of effects. “How is feedback literacy defined and measured across university courses?” may suit a scoping review. “How can competing theories of feedback explain contradictory findings?” may suit the narrative approach in Article 26.
For an effectiveness review, PICO identifies the population, intervention, comparator and outcomes. Add relevant settings, follow up times and designs. For a scoping review, PCC identifies the population, concept and context. These are question building aids, not rules that every discipline must squeeze into identical wording. The JBI manual explains how scoping eligibility should follow the review objective and may include diverse evidence sources when justified. S503
For example, an engineering review could evaluate whether one thermal management approach reduces peak device temperature under comparable loading conditions. A science review could compare evidence for a mechanism in specified experimental systems. A social science review could examine educational interventions or synthesise qualitative accounts of institutional trust. Methods must fit the evidence type. An intervention meta analysis template does not automatically suit a synthesis of theories, interview findings or simulation architectures.
Understand the difference between conduct and reporting
A protocol specifies what you intend to do. A reporting guideline identifies information readers need to evaluate the completed work. A risk of bias tool helps assess ways an estimate could be systematically distorted. These instruments serve different functions.
PRISMA 2020 provides reporting resources, including checklists, explanation documents and selection flow templates. PRISMA ScR addresses scoping reviews. Use the extension that matches your review and consult its full explanation. Completing a checklist does not prove that the search was adequate, screening accurate or synthesis valid. S06 S38
This distinction matters when writing. “The review followed PRISMA” is not a sufficient methods description. Readers still need your eligibility criteria, information sources, actual screening process, extraction rules, appraisal method and synthesis decisions. A beautifully completed checklist cannot recover studies omitted by an unsuitable search.
Write a protocol that resolves predictable decisions
Prepare the protocol before screening results in detail. Define the question and rationale, eligibility criteria, search sources, screening procedure, extraction variables, appraisal, synthesis groups and planned sensitivity analyses. Decide how you will handle several reports of the same study, repeated outcomes, missing information and amendments. Allocate responsibilities and realistic time for retrieval and checking.
For a fictional review of peer feedback in university writing courses, specify whether the intervention must involve student to student feedback, whether teacher feedback is a comparator or an exclusion, which writing outcomes count, and whether both randomised and nonrandomised designs are eligible. Decide whether a course grade measures the same thing as an independently rated writing task. These decisions determine the evidence base; leaving them until results are known can make inclusion depend on whether findings suit your expectations.
Register the protocol in an appropriate registry when the review is eligible, or place a dated protocol in a suitable repository. Eligibility varies by registry and should be checked directly. Registration records an intention; it does not certify methodological quality. If the protocol changes, retain the earlier version and explain what changed, when and why. A correction to an impossible search restriction is different from changing the primary outcome because its results were disappointing.
Develop a search that can find the evidence you need
Identify sources from the discipline and question. Biomedical databases, education indexes, psychology databases, engineering libraries, trial registries, theses and technical reports have different coverage. Do not assume a general search engine captures everything, or that a fixed number of databases guarantees completeness. Work with a librarian or information specialist when possible. Cochrane's search chapter explains sensitive retrieval, source selection and the distinction between studies and their reports for intervention reviews; adapt its scope carefully outside that setting. S20
Begin with concept groups and test whether known relevant papers can be retrieved. A hypothetical text word starting point is:
("peer feedback" OR "peer assessment" OR "peer review")
AND
(writing OR composition)
AND
(university OR undergraduate OR college)
This is a teaching expression, not a validated strategy. “Peer review” also retrieves journal reviewing, while relevant educational papers may use vocabulary absent from the expression. Database subject headings and field codes can improve retrieval. An outcome block may exclude eligible studies whose abstracts omit the outcome, so do not add every PICO element automatically. Revise through testing and documented reasoning rather than reducing the results until screening feels convenient.
For each source retain the database, platform, complete search expression, search date, filters, coverage limits and record count. Save the exports and document deduplication. Search reporting guidance in PRISMA S explains why naming a database and providing a few keywords is insufficient for an update or replication. S501
Language or date limits may be justified, but they can also restrict the conclusions. Explain practical constraints plainly. “We included English language reports because translation resources were unavailable” is more informative than silently omitting other languages. Update the search near submission when appropriate and report the final search date rather than the date the introduction was written.
Screen with rules, records and a disagreement process
Pilot the eligibility rules on a varied sample, including borderline cases. Resolve ambiguities before screening the full collection. At title and abstract screening, uncertain records commonly need retrieval rather than exclusion based on missing details. At the full report stage, record a specific reason for each exclusion and use a consistent hierarchy when more than one reason applies.
For systematic intervention reviews, Cochrane specifies at least two people independently making final inclusion decisions from full reports, with a process to resolve disagreements. Independence means each makes an initial judgement before comparing decisions. A single reviewer shortcut is a departure from this benchmark, not an equivalent procedure. S20
For a resource limited student project, report exactly who screened which stages and how decisions were checked. Do not claim that two reviewers independently screened all records if one person screened and a supervisor checked a sample. A transparent limitation is preferable to an invented safeguard, but the limitation may still affect the confidence readers place in the review.
Keep three units distinct. A record is an entry retrieved from a database. A report is a document describing research. A study is the underlying investigation. One study may produce a conference abstract, a journal article and a follow up paper. Including all three reports can be useful; counting their shared participants three times is not.
The original teaching flow illustrates the bookkeeping. Of 180 records, 40 duplicates are removed, leaving 140 for screening. After excluding 110, 30 reports are sought. Two cannot be retrieved, leaving 28 assessed reports. Eighteen are excluded with reasons, so ten reports remain, representing eight distinct studies.
The diagram is a simplified illustration. Use the appropriate official template for a real report, including additional identification routes where relevant. Unavailable reports belong in the flow; they should not disappear from the arithmetic.
Extract what the synthesis actually requires
Pilot an extraction form before applying it to every study. Record the study and report identifiers, setting, design, sample, intervention or exposure, comparator, outcome definition, time point, analysis population and numerical results. Preserve source locations and distinguish numbers reported by authors from quantities calculated by reviewers. A missing standard deviation should be recorded as missing, not silently replaced with a guessed value. Cochrane requires at least two people to extract outcome data independently, with a defined disagreement process. Another person checking critical numbers is useful in a constrained project, but it is not equivalent to duplicate independent extraction. Report the actual procedure and any departure from the intended standard. S526
Do not exclude an otherwise eligible study solely because results for a measured outcome are unreported or unusable. Distinguish eligibility for the review from availability for a particular meta analysis; retain the study in the review, seek missing information where possible, and report the limitation. S20
In the fictional writing review, an extraction row might describe a randomised classroom comparison, the number of independent classes, the writing rubric, who rated the work, the follow up period and an estimate accounting for clustering. The number of students and the number of classes are both relevant. Copying a student level standard error from an analysis that ignored class allocation may import the original study's error into the synthesis.
For qualitative or conceptual syntheses, extract contextual and interpretive information appropriate to the method. A quotation, an author's analytic theme and a reviewer's interpretation are different kinds of material. A scoping extraction form may chart definitions, methods, settings and evidence gaps without estimating a combined intervention effect. Keep the extraction aligned with the question rather than collecting every available detail.
Assess risk of bias at the relevant level
Risk of bias concerns systematic distortion, not merely incomplete reporting or a small sample. A precise estimate can be biased. A poorly reported method may leave its bias risk unclear, but uncertainty about the method is not proof that a particular flaw occurred.
Choose tools designed for the study type and outcome. RoB 2 addresses results from randomised trials and has versions for particular trial designs. Its domain judgments should be supported by information from the study, not converted into an arbitrary total score. A tool designed for a randomised intervention trial is not automatically suitable for an observational association, an interview study or an engineering benchmark. S504
In a hypothetical educational trial, missing final writing assessments may matter if students struggling most were more likely to leave. An unblinded assessor might also influence a subjective score. Explain how these possibilities affect a particular result. For engineering comparisons, examine whether test sets were reused during tuning, baselines received comparable optimisation, and failed trials were omitted. Use a justified appraisal framework rather than relabelling every performance checklist as a validated bias tool.
Scoping reviews do not always require formal appraisal; whether it is needed depends on their aim. If you omit it, avoid presenting a map of publications as a judgement that the interventions are effective or the evidence is trustworthy. If you perform appraisal, explain the tool and its role in interpretation. The current JBI guidance discusses this decision explicitly. S521
Decide whether a meta analysis answers a coherent question
A meta analysis combines numerical estimates, not conclusions such as “positive” or “negative”. Before pooling, ask whether the populations, interventions, outcomes, time points and effect definitions address a meaningful shared question. Distinguish independent studies from several correlated results in one study. Multiple outcomes or treatment groups may require selection rules or methods that account for dependence.
Choose an effect measure that readers can interpret. A mean difference uses the outcome's original units. A standardised mean difference may be appropriate for compatible constructs measured on different scales. Risk ratios, odds ratios and risk differences answer different questions; they cannot be substituted without explanation. Decide the direction so that every included effect uses the same interpretation. S21
A model assuming a common underlying effect and a model allowing effects to vary answer different inferential questions. In a common effect inverse variance calculation, more precise estimates receive greater weight. A random effects model includes an estimate of variation between study effects. Choosing the model should follow the scientific target and assumptions, not which produces a smaller p value. Neither model repairs fundamentally incompatible studies. S519
A small numerical demonstration
Suppose two entirely synthetic, independent studies estimate the same mean difference in the same units. Their estimates are 2.0 and 4.0, with standard errors 1.0 and 2.0. Under a simplified common effect model, the inverse variance weights are 1/1² = 1 and 1/2² = 0.25. The pooled estimate is (1 × 2 + 0.25 × 4) / 1.25 = 2.4. Its standard error is sqrt(1 / 1.25) = 0.894. A normal approximation gives a 95% interval of about 0.65 to 4.15.
The arithmetic explains why simply averaging 2 and 4 to obtain 3 ignores their different precision. It does not establish that these fictional studies should be pooled. With only two studies, estimating variation across studies is especially uncertain; the calculation is not a template for selecting a real model or assuming all reviews need a pooled estimate.
Interpret heterogeneity and missing evidence carefully
Heterogeneity means variation among studies, including differences in populations, methods and underlying effects. The statistic I² describes estimated inconsistency relative to sampling variation; it is not the percentage of studies that are biased or the percentage of the pooled effect that is false. Its interpretation depends on the precision and number of studies. Tau squared, written τ², estimates variance between underlying effects on the model's effect scale. S520
A prediction interval, when justified and estimable, addresses variation in an underlying effect in a new comparable study rather than uncertainty only in the pooled mean. Do not interpret a narrow confidence interval around the mean as proof that every setting will show the same result. Explore plausible explanations specified in the protocol and label additional explorations. A difference between one subgroup's significant result and another's nonsignificant result is not itself a statistical test of subgroup difference.
Publication bias is one form of missing evidence: availability may depend on the result. Selective outcome reporting within available studies is another concern. Funnel plot asymmetry can have several explanations, and symmetry does not establish absence of bias. Formal asymmetry tests often have little power with few studies; their suitability depends on the effect measure and data. Cochrane discusses these limitations and cautions against treating an asymmetry test as a simple pass or fail assessment. S518
Synthesis remains necessary when pooling is unsuitable
Group studies using a stated rationale, display estimates and uncertainty where available, and explain how design differences affect interpretation. Do not replace a meta analysis with a tally of statistically significant studies. SWiM provides reporting guidance for intervention reviews using other synthesis methods, including explaining grouping, metrics and how conclusions were reached. S502
For example, a fictional review might say: “We examined classroom interventions separately from laboratory exercises because their settings and outcomes addressed different questions. The classroom studies used incompatible writing rubrics, and several lacked information required to calculate comparable effect estimates. We therefore present their estimates in the original units and compare design, implementation and uncertainty without calculating a pooled effect.” The statement explains a decision. It must then be followed by the actual structured comparison, not a list of abstracts.
For an intervention evidence body, GRADE separates confidence in the overall evidence for each outcome from bias judgments about individual studies. Considerations include risk of bias, inconsistency, indirectness, imprecision and missing evidence. Do not label a conclusion “high certainty” simply because several papers were included. Use the full framework with appropriate expertise and explain the judgments. S517
Write the manuscript so another team could update it
The Introduction explains the unresolved question and why an existing review does not already answer it. The Methods describes the protocol, eligibility, information sources, searches, selection, extraction, appraisal and synthesis. The Results reports the selection flow, study characteristics, assessments and findings in the planned groups. The Discussion answers the question while separating limitations of the included evidence from limitations introduced by the review itself.
Provide complete searches, extraction definitions, important exclusions, assessment details and analysis materials in suitable supplements or repositories. Report software and relevant versions. If automation or AI assisted screening or extraction, describe the actual task, checking process and limitations. Do not claim that a model retrieved all evidence or accurately extracted numerical results unless those claims have been evaluated.
A scoping review's results may include a concept map, distribution of study designs and a table of operational definitions. Counts describe the retrieved literature, not automatically the prevalence of a phenomenon in society. Ten publications using a method do not mean that ten independent research groups used it, and several reviews citing the same primary study do not create additional primary evidence.
Practice and model answer
Task: Your review has 12 included reports. Two articles and one conference abstract describe the same experiment. One colleague proposes pooling all 12 report estimates and excluding papers without p < 0.05. Explain the problems and repair the plan.
Model answer: First map reports to underlying studies and identify overlapping participants and outcomes. Use complementary reports to obtain complete information, but do not treat repeated estimates from the same experiment as independent. Apply the prespecified eligibility rules without selecting on statistical significance. Extract effect sizes and uncertainty, assess bias, and decide which estimates address a compatible question. Document any protocol amendment before synthesis and report its rationale. If information remains insufficient, use a transparent structured synthesis rather than inventing numbers or silently discarding uncertainty.
Before starting your own review, have another person apply your eligibility rules to three borderline reports. Explain disagreements and revise ambiguous rules. Use Template 9 for the protocol and Template 3 for the search and evidence record.
Sources
S06: Page MJ and colleagues. PRISMA 2020 statement and official reporting resources.
Downloads
- Calculation record: review_engineering_calculations.json (1 KB)
- About the teaching data: README.txt (8 KB)
Sign in to download these files.