Reproducibility · Intermediate
Write a protocol and justify your sample size
Record the protocol and justify the amount of evidence the study needs.
11 min read · Oct 26, 2026
Key points
- Specify outcomes before analysis.
- Distinguish power and precision planning.
- Report constraints and amendments honestly.
Contents
Record the protocol and justify the amount of evidence the study needs.
A protocol records what you intend to do before the results make some choices more attractive than others. It connects the research question to recruitment, measurements, analysis and interpretation. A good protocol is detailed enough that another researcher can distinguish following the plan from making a new decision.
Sample size belongs in this plan because it affects what the study can teach. A large sample does not correct a biased design, and a small sample is not automatically useless. The right question is whether the proposed information can answer the intended question with useful precision, power or qualitative depth.
Give the protocol a clear structure
Begin with the question, contribution and target population or system. Specify the design, eligibility criteria, sampling or recruitment route, relevant permissions, allocation, measurements and timing. Identify independent units and repeated observations. Then explain the analysis and the conditions under which the study will stop or change.
A primary outcome is the outcome given priority for the main planned inference. Secondary outcomes address additional planned questions. Exploratory analyses investigate patterns that may generate new questions. Several primary outcomes are possible, but their joint interpretation and any multiplicity strategy must be justified. Do not use the label “primary” for whichever outcome later produces the smallest p value.
The outcome definition should include the measurement, unit, timing, scoring or aggregation rule, and treatment of unusual events. “Assess growth” is incomplete. “Estimate the difference in dry mass at the specified endpoint” is more useful, but still requires a procedure for obtaining the mass and handling specimens that cannot be measured.
Replace vague promises with operational decisions
“We will use appropriate statistical tests” does not tell readers what you planned. A stronger statement identifies the target quantity, calculation, uncertainty method and assumptions. It also explains what will happen if a planned model cannot be fitted or an assumption is clearly unsuitable.
For the hypothetical controller study, a useful planning statement is: “The primary estimand is the mean difference in positioning error across the defined distribution of layouts, with each layout evaluated using both controllers. We will calculate A minus B within layouts and estimate its mean with an uncertainty interval appropriate to the paired design. Completion failures will be reported separately under the prespecified rule.” This is still a framework; a real protocol needs the exact scenario distribution and analysis details.
Record exclusions before outcomes are inspected where possible. Define missing data handling, planned covariates, subgroup comparisons and sensitivity analyses. A sensitivity analysis asks whether conclusions change under plausible alternative assumptions or decisions. It should probe a genuine uncertainty, not provide many opportunities to choose a preferred result.
Use Template 2 for the study protocol and Template 4 for the analysis details.
Understand preregistration and Registered Reports
Preregistration places a dated plan in a registry before the relevant analysis. It helps distinguish decisions made without knowledge of the outcomes from those informed by them. A preregistration of an existing dataset should disclose what the researchers already know about the data. A dated document does not erase prior access. S07
A Registered Report is a publication format in which a journal evaluates a study plan before the main results are available, according to that journal’s procedures. S110 Neither preregistration nor a Registered Report substitutes for ethics approval, methodological expertise or honest execution.
Plans can change. Preserve the original version, date the amendment, explain the reason and state whether relevant results were already known. If equipment fails or a measurement proves unusable, transparent revision is more defensible than pretending the final procedure was planned from the start. Report both planned and exploratory work clearly.
Choose the sample size rationale that fits the aim
Lakens describes several ways of justifying sample size, including precision, statistical power, an available population and resource constraints. The central task is to explain the information the sample can provide for the intended inference. A conventional number copied from another paper does not supply that explanation. S100
| Aim | Useful planning question | Main inputs |
|---|---|---|
| Estimate a quantity precisely | How narrow must the uncertainty interval be? | Variability, desired precision, confidence level and design |
| Detect a meaningful effect | How often should the planned test detect a specified effect? | Effect size, variability, significance level, power and design |
| Describe a finite accessible population | Can all eligible units be measured? | Population definition, coverage and missingness |
| Assess feasibility | Which practical uncertainty must be resolved? | Recruitment, measurement success or another feasibility target |
| Conduct qualitative inquiry | What information is needed for this approach and question? | Sampling logic, diversity, analytic depth and available material |
A census of the accessible population does not automatically establish generalisation to other populations or future conditions. A feasibility study should address feasibility directly rather than using a small convenience sample to make an ambitious efficacy claim.
Learn the statistical planning terms
Effect size describes the magnitude of a difference or relationship. It can be expressed in original units, such as millimetres, or standardised relative to variability. A smallest effect of interest is the smallest effect that would matter for the scientific or practical decision. It needs justification from consequences, theory, measurement or prior evidence.
Alpha, commonly written α, is the planned Type I error threshold for a specified testing procedure. A Type I error is rejecting a true null hypothesis. Power is the probability that the procedure rejects its null under a specified alternative and assumptions. Beta, β, is the corresponding Type II error probability, so power is 1 minus β. Power is not the probability that a hypothesis is true.
Precision concerns uncertainty around an estimate. For a simple independent mean, the standard error is the standard deviation divided by the square root of sample size. This explains why halving standard error usually requires about four times as many independent observations under otherwise unchanged conditions. It does not mean that four times as many technical repeats replace four times as many independent units. S14
Worked example: plan an interval for an instrument measurement
Suppose a fictional instrument project wants to estimate a mean with an approximate 95% confidence interval extending no more than 4 units on either side. Prior information suggests a standard deviation of 10 units. These numbers are constructed for teaching, not recommended inputs for an actual study.
A normal approximation gives n ≈ (1.96 × SD ÷ h)², where h is the desired half width. Substitution gives (1.96 × 10 ÷ 4)² = 24.01, which rounds upward to 25 independent observations. Rounding downward would fail the stated approximation.
If the standard deviation must be estimated, the t critical value depends on sample size. Using the planning standard deviation of 10, the conventional t interval has a planned half width of about 4.04 units at n = 26 and 3.96 units at n = 27. Thus 27 is the first integer meeting this plug in calculation. The actual interval width remains random because the observed standard deviation will vary. This is a planning approximation, not a guarantee of the realised precision.
Suppose the team additionally expects 15% of planned measurements to be unusable for prespecified technical reasons. A simple inflation gives 27 ÷ 0.85 = 31.76, rounded upward to 32 planned units. This adjustment does not repair informative missingness: if failures preferentially occur at extreme values, the observed mean may still be biased. Explain both the numerical inflation and the reason for expected loss.
A useful Methods statement would report the target half width, assumed variability, interval method, calculated number, anticipated loss and final target. Simply writing “a sample size calculation was performed” hides the decisions readers need.
Worked example: plan a comparison rather than one mean
A fictional social science study plans two independent groups of equal size. It considers a difference of 5 score units worth detecting and assumes an SD of 10 units in each group. The standardised difference is therefore 0.5. A normal approximation for 80% power with a two sided alpha of 0.05 is n per group ≈ 2 × (1.96 + 0.84)² ÷ 0.5², giving about 63 per group after rounding upward. The factor 2 represents the variance contributed by both equally sized groups. This adapts the normal critical value planning approach for a mean to the variance of a difference between independent means. S111
This approximate result is not an exact t test calculation and is not suitable unchanged for classroom allocation, repeated observations, unequal allocation or substantial missingness. The inputs themselves also need evidence. If the plausible SD is larger than expected, more participants may be needed. If the meaningful effect is smaller, the required sample increases.
For paired measurements, planning uses variability of the differences, which depends on the relationship between measurements within pairs. For clustered studies, the number and size of clusters and their within cluster similarity matter. For a regression or a complex engineering simulation, simulation based planning may better represent the actual model. Describe the model and assumptions used in that simulation, including how uncertainty in the inputs was explored.
Examine more than one plausible scenario
Do not treat a noisy pilot effect estimate as a reliable prediction of the future effect. Pilot data may help assess measurement variability or feasibility, but an unusually large observed effect can produce an unrealistically small planned sample.
Construct a planning table using several plausible variabilities or effect sizes. For the precision example, a larger anticipated SD means a wider interval at the same n. Ask whether the study remains informative in the less optimistic scenario. If resource constraints limit recruitment, state the achievable precision or detectable effects under those constraints.
Distinguish planning based on a meaningful target effect from observed power calculated using the effect already observed in the completed study. The latter should not replace interpretation of the estimate and uncertainty interval. A nonsignificant result does not establish that the effect is negligible. A claim of equivalence needs justified equivalence margins and an appropriate design and analysis.
Set stopping and amendment rules
Specify whether data collection ends at a target sample, a date, an information level or another justified rule. A fixed deadline may be reasonable for a student project, but readers should know whether the stopping decision depended on the result.
Repeatedly checking an ordinary p value and stopping when it crosses 0.05 changes the error properties of the procedure. If interim analysis is needed, obtain advice and use a valid sequential design with its corresponding reporting. Do not quietly add observations after an inconvenient result and present the analysis as if the final sample had always been fixed.
For qualitative studies, explain how sampling decisions will be reviewed in relation to the analytic approach and question. Avoid a universal numerical rule or an unsupported assertion that a particular interview count guarantees saturation. An evolving sampling strategy can be rigorous when the rationale and decisions are documented.
Practice a protocol review
Exercise. A fictional proposal states: “We will recruit as many students as possible, measure several outcomes and use whichever test fits. If results are not significant, we will collect more data.” Identify the missing decisions.
Model answer. The proposal lacks a defined recruitment target or deadline, primary estimand, outcome priorities, model, multiplicity approach, missing data plan and valid stopping rule. The team should state its feasible resources, calculate what information those resources provide, specify the main comparison and define amendments prospectively. Additional data collection is not inherently wrong; making it depend on repeated ordinary significance checks without a valid design is the problem.
Before approving your own protocol, ask a peer to identify the primary result using only the document. If several incompatible answers fit, the plan needs greater specificity. A useful protocol makes later writing easier because the Methods, Results and limitations already have a coherent foundation.
Sources
S07: Center for Open Science. Preregistration. Official guidance, accessed 11 November 2026.