Мазмұнға өту

Жазу · Жоғары

Plan and report a systematic or scoping review

Plan selection, extraction, assessment and synthesis for a systematic or scoping review.

17 мин оқу · 2026 ж. 26 қаз.

Бұл бет әлі аударылмаған, сондықтан ағылшын нұсқасын оқып отырсыз.

Негізгі тұжырымдар

  • Match review type to question.
  • Distinguish records, reports and studies.
  • Use appropriate reporting guidance.
Мазмұны

Plan selection, extraction, assessment and synthesis for a systematic or scoping review.

A systematic review is a research project whose evidence comes from existing studies. Its credibility depends on decisions about what to include, how to find it, how to assess it and how to combine it. A scoping review also uses explicit methods, but usually asks what kinds of evidence or concepts exist and how they are distributed. Neither becomes rigorous simply by having many references, a flow diagram or statistical software.

This tutorial explains the complete path from question to manuscript. All study counts and numerical demonstrations below are synthetic teaching examples. They do not describe an actual review, and the sample search expression has not been run as a research search.

Use the question to choose the method. “Does a defined feedback intervention improve writing scores compared with usual teaching?” may suit a systematic review of effects. “How is feedback literacy defined and measured across university courses?” may suit a scoping review. “How can competing theories of feedback explain contradictory findings?” may suit the narrative approach in Article 26.

For an effectiveness review, PICO identifies the population, intervention, comparator and outcomes. Add relevant settings, follow up times and designs. For a scoping review, PCC identifies the population, concept and context. These are question building aids, not rules that every discipline must squeeze into identical wording. The JBI manual explains how scoping eligibility should follow the review objective and may include diverse evidence sources when justified. S503

For example, an engineering review could evaluate whether one thermal management approach reduces peak device temperature under comparable loading conditions. A science review could compare evidence for a mechanism in specified experimental systems. A social science review could examine educational interventions or synthesise qualitative accounts of institutional trust. Methods must fit the evidence type. An intervention meta analysis template does not automatically suit a synthesis of theories, interview findings or simulation architectures.

Understand the difference between conduct and reporting

A protocol specifies what you intend to do. A reporting guideline identifies information readers need to evaluate the completed work. A risk of bias tool helps assess ways an estimate could be systematically distorted. These instruments serve different functions.

PRISMA 2020 provides reporting resources, including checklists, explanation documents and selection flow templates. PRISMA ScR addresses scoping reviews. Use the extension that matches your review and consult its full explanation. Completing a checklist does not prove that the search was adequate, screening accurate or synthesis valid. S06 S38

This distinction matters when writing. “The review followed PRISMA” is not a sufficient methods description. Readers still need your eligibility criteria, information sources, actual screening process, extraction rules, appraisal method and synthesis decisions. A beautifully completed checklist cannot recover studies omitted by an unsuitable search.

Write a protocol that resolves predictable decisions

Prepare the protocol before screening results in detail. Define the question and rationale, eligibility criteria, search sources, screening procedure, extraction variables, appraisal, synthesis groups and planned sensitivity analyses. Decide how you will handle several reports of the same study, repeated outcomes, missing information and amendments. Allocate responsibilities and realistic time for retrieval and checking.

For a fictional review of peer feedback in university writing courses, specify whether the intervention must involve student to student feedback, whether teacher feedback is a comparator or an exclusion, which writing outcomes count, and whether both randomised and nonrandomised designs are eligible. Decide whether a course grade measures the same thing as an independently rated writing task. These decisions determine the evidence base; leaving them until results are known can make inclusion depend on whether findings suit your expectations.

Register the protocol in an appropriate registry when the review is eligible, or place a dated protocol in a suitable repository. Eligibility varies by registry and should be checked directly. Registration records an intention; it does not certify methodological quality. If the protocol changes, retain the earlier version and explain what changed, when and why. A correction to an impossible search restriction is different from changing the primary outcome because its results were disappointing.

Develop a search that can find the evidence you need

Identify sources from the discipline and question. Biomedical databases, education indexes, psychology databases, engineering libraries, trial registries, theses and technical reports have different coverage. Do not assume a general search engine captures everything, or that a fixed number of databases guarantees completeness. Work with a librarian or information specialist when possible. Cochrane's search chapter explains sensitive retrieval, source selection and the distinction between studies and their reports for intervention reviews; adapt its scope carefully outside that setting. S20

Begin with concept groups and test whether known relevant papers can be retrieved. A hypothetical text word starting point is:

("peer feedback" OR "peer assessment" OR "peer review")
AND
(writing OR composition)
AND
(university OR undergraduate OR college)

This is a teaching expression, not a validated strategy. “Peer review” also retrieves journal reviewing, while relevant educational papers may use vocabulary absent from the expression. Database subject headings and field codes can improve retrieval. An outcome block may exclude eligible studies whose abstracts omit the outcome, so do not add every PICO element automatically. Revise through testing and documented reasoning rather than reducing the results until screening feels convenient.

For each source retain the database, platform, complete search expression, search date, filters, coverage limits and record count. Save the exports and document deduplication. Search reporting guidance in PRISMA S explains why naming a database and providing a few keywords is insufficient for an update or replication. S501

Language or date limits may be justified, but they can also restrict the conclusions. Explain practical constraints plainly. “We included English language reports because translation resources were unavailable” is more informative than silently omitting other languages. Update the search near submission when appropriate and report the final search date rather than the date the introduction was written.

Screen with rules, records and a disagreement process

Pilot the eligibility rules on a varied sample, including borderline cases. Resolve ambiguities before screening the full collection. At title and abstract screening, uncertain records commonly need retrieval rather than exclusion based on missing details. At the full report stage, record a specific reason for each exclusion and use a consistent hierarchy when more than one reason applies.

For systematic intervention reviews, Cochrane specifies at least two people independently making final inclusion decisions from full reports, with a process to resolve disagreements. Independence means each makes an initial judgement before comparing decisions. A single reviewer shortcut is a departure from this benchmark, not an equivalent procedure. S20

For a resource limited student project, report exactly who screened which stages and how decisions were checked. Do not claim that two reviewers independently screened all records if one person screened and a supervisor checked a sample. A transparent limitation is preferable to an invented safeguard, but the limitation may still affect the confidence readers place in the review.

Keep three units distinct. A record is an entry retrieved from a database. A report is a document describing research. A study is the underlying investigation. One study may produce a conference abstract, a journal article and a follow up paper. Including all three reports can be useful; counting their shared participants three times is not.

The original teaching flow illustrates the bookkeeping. Of 180 records, 40 duplicates are removed, leaving 140 for screening. After excluding 110, 30 reports are sought. Two cannot be retrieved, leaving 28 assessed reports. Eighteen are excluded with reasons, so ten reports remain, representing eight distinct studies.

A simplified selection flow shows 180 records becoming ten included reports representing eight studies, with exclusions and unavailable reports accounted for.

The diagram is a simplified illustration. Use the appropriate official template for a real report, including additional identification routes where relevant. Unavailable reports belong in the flow; they should not disappear from the arithmetic.

Extract what the synthesis actually requires

Pilot an extraction form before applying it to every study. Record the study and report identifiers, setting, design, sample, intervention or exposure, comparator, outcome definition, time point, analysis population and numerical results. Preserve source locations and distinguish numbers reported by authors from quantities calculated by reviewers. A missing standard deviation should be recorded as missing, not silently replaced with a guessed value. Cochrane requires at least two people to extract outcome data independently, with a defined disagreement process. Another person checking critical numbers is useful in a constrained project, but it is not equivalent to duplicate independent extraction. Report the actual procedure and any departure from the intended standard. S526

Do not exclude an otherwise eligible study solely because results for a measured outcome are unreported or unusable. Distinguish eligibility for the review from availability for a particular meta analysis; retain the study in the review, seek missing information where possible, and report the limitation. S20

In the fictional writing review, an extraction row might describe a randomised classroom comparison, the number of independent classes, the writing rubric, who rated the work, the follow up period and an estimate accounting for clustering. The number of students and the number of classes are both relevant. Copying a student level standard error from an analysis that ignored class allocation may import the original study's error into the synthesis.

For qualitative or conceptual syntheses, extract contextual and interpretive information appropriate to the method. A quotation, an author's analytic theme and a reviewer's interpretation are different kinds of material. A scoping extraction form may chart definitions, methods, settings and evidence gaps without estimating a combined intervention effect. Keep the extraction aligned with the question rather than collecting every available detail.

Assess risk of bias at the relevant level

Risk of bias concerns systematic distortion, not merely incomplete reporting or a small sample. A precise estimate can be biased. A poorly reported method may leave its bias risk unclear, but uncertainty about the method is not proof that a particular flaw occurred.

Choose tools designed for the study type and outcome. RoB 2 addresses results from randomised trials and has versions for particular trial designs. Its domain judgments should be supported by information from the study, not converted into an arbitrary total score. A tool designed for a randomised intervention trial is not automatically suitable for an observational association, an interview study or an engineering benchmark. S504

In a hypothetical educational trial, missing final writing assessments may matter if students struggling most were more likely to leave. An unblinded assessor might also influence a subjective score. Explain how these possibilities affect a particular result. For engineering comparisons, examine whether test sets were reused during tuning, baselines received comparable optimisation, and failed trials were omitted. Use a justified appraisal framework rather than relabelling every performance checklist as a validated bias tool.

Scoping reviews do not always require formal appraisal; whether it is needed depends on their aim. If you omit it, avoid presenting a map of publications as a judgement that the interventions are effective or the evidence is trustworthy. If you perform appraisal, explain the tool and its role in interpretation. The current JBI guidance discusses this decision explicitly. S521

Decide whether a meta analysis answers a coherent question

A meta analysis combines numerical estimates, not conclusions such as “positive” or “negative”. Before pooling, ask whether the populations, interventions, outcomes, time points and effect definitions address a meaningful shared question. Distinguish independent studies from several correlated results in one study. Multiple outcomes or treatment groups may require selection rules or methods that account for dependence.

Choose an effect measure that readers can interpret. A mean difference uses the outcome's original units. A standardised mean difference may be appropriate for compatible constructs measured on different scales. Risk ratios, odds ratios and risk differences answer different questions; they cannot be substituted without explanation. Decide the direction so that every included effect uses the same interpretation. S21

A model assuming a common underlying effect and a model allowing effects to vary answer different inferential questions. In a common effect inverse variance calculation, more precise estimates receive greater weight. A random effects model includes an estimate of variation between study effects. Choosing the model should follow the scientific target and assumptions, not which produces a smaller p value. Neither model repairs fundamentally incompatible studies. S519

A small numerical demonstration

Suppose two entirely synthetic, independent studies estimate the same mean difference in the same units. Their estimates are 2.0 and 4.0, with standard errors 1.0 and 2.0. Under a simplified common effect model, the inverse variance weights are 1/1² = 1 and 1/2² = 0.25. The pooled estimate is (1 × 2 + 0.25 × 4) / 1.25 = 2.4. Its standard error is sqrt(1 / 1.25) = 0.894. A normal approximation gives a 95% interval of about 0.65 to 4.15.

The arithmetic explains why simply averaging 2 and 4 to obtain 3 ignores their different precision. It does not establish that these fictional studies should be pooled. With only two studies, estimating variation across studies is especially uncertain; the calculation is not a template for selecting a real model or assuming all reviews need a pooled estimate.

Interpret heterogeneity and missing evidence carefully

Heterogeneity means variation among studies, including differences in populations, methods and underlying effects. The statistic I² describes estimated inconsistency relative to sampling variation; it is not the percentage of studies that are biased or the percentage of the pooled effect that is false. Its interpretation depends on the precision and number of studies. Tau squared, written τ², estimates variance between underlying effects on the model's effect scale. S520

A prediction interval, when justified and estimable, addresses variation in an underlying effect in a new comparable study rather than uncertainty only in the pooled mean. Do not interpret a narrow confidence interval around the mean as proof that every setting will show the same result. Explore plausible explanations specified in the protocol and label additional explorations. A difference between one subgroup's significant result and another's nonsignificant result is not itself a statistical test of subgroup difference.

Publication bias is one form of missing evidence: availability may depend on the result. Selective outcome reporting within available studies is another concern. Funnel plot asymmetry can have several explanations, and symmetry does not establish absence of bias. Formal asymmetry tests often have little power with few studies; their suitability depends on the effect measure and data. Cochrane discusses these limitations and cautions against treating an asymmetry test as a simple pass or fail assessment. S518

Synthesis remains necessary when pooling is unsuitable

Group studies using a stated rationale, display estimates and uncertainty where available, and explain how design differences affect interpretation. Do not replace a meta analysis with a tally of statistically significant studies. SWiM provides reporting guidance for intervention reviews using other synthesis methods, including explaining grouping, metrics and how conclusions were reached. S502

For example, a fictional review might say: “We examined classroom interventions separately from laboratory exercises because their settings and outcomes addressed different questions. The classroom studies used incompatible writing rubrics, and several lacked information required to calculate comparable effect estimates. We therefore present their estimates in the original units and compare design, implementation and uncertainty without calculating a pooled effect.” The statement explains a decision. It must then be followed by the actual structured comparison, not a list of abstracts.

For an intervention evidence body, GRADE separates confidence in the overall evidence for each outcome from bias judgments about individual studies. Considerations include risk of bias, inconsistency, indirectness, imprecision and missing evidence. Do not label a conclusion “high certainty” simply because several papers were included. Use the full framework with appropriate expertise and explain the judgments. S517

Write the manuscript so another team could update it

The Introduction explains the unresolved question and why an existing review does not already answer it. The Methods describes the protocol, eligibility, information sources, searches, selection, extraction, appraisal and synthesis. The Results reports the selection flow, study characteristics, assessments and findings in the planned groups. The Discussion answers the question while separating limitations of the included evidence from limitations introduced by the review itself.

Provide complete searches, extraction definitions, important exclusions, assessment details and analysis materials in suitable supplements or repositories. Report software and relevant versions. If automation or AI assisted screening or extraction, describe the actual task, checking process and limitations. Do not claim that a model retrieved all evidence or accurately extracted numerical results unless those claims have been evaluated.

A scoping review's results may include a concept map, distribution of study designs and a table of operational definitions. Counts describe the retrieved literature, not automatically the prevalence of a phenomenon in society. Ten publications using a method do not mean that ten independent research groups used it, and several reviews citing the same primary study do not create additional primary evidence.

Practice and model answer

Task: Your review has 12 included reports. Two articles and one conference abstract describe the same experiment. One colleague proposes pooling all 12 report estimates and excluding papers without p < 0.05. Explain the problems and repair the plan.

Model answer: First map reports to underlying studies and identify overlapping participants and outcomes. Use complementary reports to obtain complete information, but do not treat repeated estimates from the same experiment as independent. Apply the prespecified eligibility rules without selecting on statistical significance. Extract effect sizes and uncertainty, assess bias, and decide which estimates address a compatible question. Document any protocol amendment before synthesis and report its rationale. If information remains insufficient, use a transparent structured synthesis rather than inventing numbers or silently discarding uncertainty.

Before starting your own review, have another person apply your eligibility rules to three borderline reports. Explain disagreements and revise ambiguous rules. Use Template 9 for the protocol and Template 3 for the search and evidence record.

Sources

S06: Page MJ and colleagues. PRISMA 2020 statement and official reporting resources.

S38: Tricco AC, Lillie E, Zarin W, et al. 2018. PRISMA extension for scoping reviews: checklist and explanation. Annals of Internal Medicine 169(7):467 to 473. Official resource.

S503: Pollock D, Peters MDJ, Tricco AC, et al. 2026. Scoping reviews. In: JBI Manual for Evidence Synthesis. Current online chapter.

S20: Lefebvre C, Glanville J, Briscoe S, et al. Cochrane Handbook, Chapter 4: Searching for and selecting studies. Current online edition, accessed 11 November 2026.

S501: Rethlefsen ML, Kirtley S, Waffenschmidt S, et al. 2021. PRISMA S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Systematic Reviews 10:39.

S504: Sterne JAC, Savović J, Page MJ, et al. 2019. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ 366:l4898. Official tools and design specific versions.

S521: JBI Manual for Evidence Synthesis. Section 10.2.8: Critical appraisal or risk of bias assessment. Current online guidance, accessed 11 November 2026.

S21: Deeks JJ, Higgins JPT, Altman DG, et al. Cochrane Handbook, Chapter 10: Analysing data and undertaking meta analyses. Current online edition, accessed 11 November 2026.

S519: Hedges LV, Vevea JL. 1998. Fixed and random effects models in meta analysis. Psychological Methods 3(4):486 to 504.

S520: Higgins JPT, Thompson SG, Deeks JJ, Altman DG. 2003. Measuring inconsistency in meta analyses. BMJ 327:557 to 560.

S518: Cochrane Handbook, Chapter 13: Assessing risk of bias due to missing evidence in a meta analysis. Current online edition, accessed 11 November 2026.

S502: Campbell M, McKenzie JE, Sowden A, et al. 2020. Synthesis without meta analysis (SWiM) in systematic reviews: reporting guideline. BMJ 368:l6890.

S517: Cochrane Handbook, Chapter 14: Completing summary of findings tables and grading the certainty of the evidence. Current online edition, accessed 11 November 2026.

S526: Cochrane Handbook, Chapter 5: Collecting data. Current online edition, accessed 11 November 2026.

Файлдар

Бұл файлдарды жүктеп алу үшін кіріңіз.