Мазмұнға өту

Қайталанымдылық · Бастапқы

Design a study with appropriate controls and replication

Understand controls, randomisation, masking and the difference between independent and repeated observations.

11 мин оқу · 2026 ж. 26 қаз.

Бұл бет әлі аударылмаған, сондықтан ағылшын нұсқасын оқып отырсыз.

Негізгі тұжырымдар

  • Identify the independent unit.
  • Explain the role of each control.
  • Represent batches and repeated observations.
Мазмұны

Understand controls, randomisation, masking and the difference between independent and repeated observations.

A strong paper gives readers a reason to believe that its evidence answers its question. Clear prose cannot repair a design in which treatment is inseparable from measurement date, a survey omits the population of interest, or repeated observations are mistaken for independent evidence. Design is the structure that connects a question to a defensible comparison.

This tutorial explains how to identify the relevant units, choose controls, reduce bias and describe repeated or clustered observations. All worked examples are hypothetical. Use them to recognise design problems, not as instructions to conduct an unsupervised experiment.

Draw the study before selecting a statistical test

Start by drawing or listing the entities in the project: participants, classrooms, cultures, animals, components, simulation layouts or published studies. Identify which entities receive an intervention, which are sampled, which are measured repeatedly, and which share an environment. Then identify the comparison that answers the research question.

A design specifies how observations are obtained and related. An analysis uses that structure to estimate or test something. Choosing a familiar statistical test first and arranging the data to fit it reverses this logic. A spreadsheet with two columns does not tell you whether the columns contain independent groups, matched pairs or repeated measurements from the same unit.

For observational social science, draw the recruitment route as well. Students recruited through a voluntary research club may differ from students who never join. For computational work, draw the route from available cases to development, validation and final evaluation sets. Selection can influence interpretation even when no person is assigned to a treatment.

Distinguish the units that people call “the sample”

The experimental unit is the entity independently assigned to an experimental condition. The observational unit is the entity measured. The sampling unit is the entity selected during sampling. These may coincide, but often they do not. The quantity of information relevant to a claim depends on the assignment, sampling and dependence structure, not simply on the number of rows in a file.

If treatment is assigned to each culture dish and several cells are measured within each dish, the cells share an assignment and environment. If a teaching intervention is assigned to classrooms, students share the classroom assignment. If a material component is measured ten times, those readings describe repeatability for that component; they are not ten newly manufactured components. Lazic and colleagues explain why distinguishing units is essential to valid replication in biological experiments. S101

Technical replication usually refers to repeated measurement of the same underlying material or unit. Independent replication adds distinct units relevant to the inference. Terminology differs between fields, so describe exactly what was repeated, how it was assigned and what sources of variation it represents.

Three independent cultures each contain four measured cells. The twelve measurements belong to three clusters.

The illustration has twelve cell observations within three cultures. Reporting only “n = 12” hides the structure. A useful description identifies both counts. If cultures receive the intervention, a culture summary or an appropriate hierarchical analysis may be suitable, depending on the question. Neither strategy creates treatment replication when there is only one treated culture.

Recognise pseudoreplication

Pseudoreplication occurs when dependent observations or subsamples are used as if they provided independent replication for the comparison of interest. The resulting uncertainty can be too small, making evidence appear stronger than the design supports.

Suppose a fictional study treats three dishes and leaves three control dishes untreated, then measures ten cells per dish. There are 60 cell measurements but only six independently assigned dishes. The paper should preserve dish identifiers and report the distribution of observations across dishes. If dish means are the planned outcome, the comparison concerns three dish summaries per group. A hierarchical model can use information at both levels, but its assumptions and the small number of dishes still matter.

Now suppose the six dishes were all prepared from one donor. They may be independent treatment assignments under the chosen laboratory conditions, yet evidence about variation across donors remains limited. “Independent” is therefore a relationship to the design and inference, not a permanent property that a sample possesses in every context.

Choose a control for a specific alternative explanation

A control is useful because it makes a comparison interpretable. Explain what alternative explanation each control addresses.

Control or comparator Main purpose Example question it helps answer
Negative control Assess background or an expected absence of response Does the assay produce signal without the target?
Positive control Check that the system can detect an expected response Can the measurement detect a response under known conditions?
Vehicle control Separate the carrier from the active treatment Could the solvent explain the difference?
Sham procedure Address procedural effects where ethically justified Could handling or a procedure explain the outcome?
Active comparator Compare against an existing intervention Does the new approach improve on current practice?
Engineering baseline Provide a credible reference method Does the proposed method outperform a properly configured alternative?

A control group is not automatically suitable because it lacks the main intervention. If treatment participants receive more attention, equipment access or assessment time, those differences may be relevant. Whether to equalise them depends on the question: are you testing a specific component or the whole intervention package?

For engineering comparisons, give the baseline a fair configuration and tuning process. Comparing an extensively tuned new method with an unsuitable default can exaggerate improvement. Report the information and computational budget available to each method.

Separate random sampling from random allocation

Random sampling concerns which units enter a sample. Random allocation concerns which condition sampled units receive. Random sampling can support inference to a sampling frame; random allocation can support a treatment comparison by reducing systematic differences between conditions on average. Neither guarantees that a small realised sample is perfectly balanced.

A trial can randomly allocate volunteers without randomly sampling the wider population. Its internal treatment comparison and its transfer to other populations are separate questions. Conversely, a randomly sampled survey can describe a population well while remaining unable to identify a causal intervention effect without additional assumptions.

Describe the allocation process sufficiently for readers to understand it. “Participants were randomly divided” is weaker than identifying the sequence generation method, allocation ratio, relevant restrictions and who implemented assignment. In some designs, blocking or stratification helps distribute important characteristics across conditions. These choices should be planned and reflected in the analysis.

Use concealment and masking where they solve a problem

Allocation concealment prevents knowledge of upcoming assignments from influencing enrolment. Blinding, also called masking, limits knowledge of conditions during intervention delivery, measurement or analysis. These practices address different opportunities for bias.

A classroom teacher may need to know which teaching activity is delivered, while assessors can still score anonymised work without knowing the condition. An engineer may know which algorithm produced a file, while an independent evaluator can judge labelled outputs without that information. If masking is impossible, identify the unmasked stage and use objective procedures where appropriate.

Do not write “the study was double blind” without saying who was blinded. Do not claim masking merely because file names were replaced after the analyst had already inspected labelled outcomes. Describe what actually happened and any risk that remains.

Balance batches and block known sources of variation

A batch effect is variation associated with a shared processing occasion, instrument, operator or other grouping. Confounding occurs when differences relevant to the outcome are mixed with the comparison of interest. If every control is measured on Monday and every treated sample on Tuesday, treatment and date cannot be separated from those observations alone.

A block groups units that share an important background condition. When appropriate, compare conditions within each block, such as testing both materials on each production day. Randomise the order within the design where feasible. NIST describes blocking as a way to account for identifiable nuisance variation while preserving the comparison of interest. S106

Blocking is not a magic correction applied after data collection. Plan what will vary together, record it, and use an analysis that respects it. More observations within a completely confounded design will not reveal which confounded factor caused the difference.

Understand pairing, repetition and interaction

A paired design links observations through the same unit or a justified match. In a controller comparison, each layout can be evaluated using both controllers, so the layout provides a pair. The difference within each layout is the relevant outcome for a simple paired comparison. Arbitrarily pairing unrelated observations after seeing their values does not create a paired design.

Repeated measurements over time also share a unit. They can show trajectories, but their dependence and timing matter. Measuring a participant repeatedly does not increase the number of independently recruited participants. For simulation research, repeated random seeds within a layout assess run variability, while new layouts assess variation across scenarios.

A factorial design varies more than one factor. It can address an interaction, meaning that the effect of one factor changes with the level of another. Consider hypothetical component output values: material A has means of 10 and 12 units at low and high temperature, while material B has means of 14 and 21. The temperature difference is 2 for A and 7 for B. The difference between those differences is 5 units. These invented means illustrate interaction; without replication and uncertainty they do not establish a real effect.

Three design audits across disciplines

Natural science audit. A plant project uses one chamber for each light duration. Measuring 40 plants in each chamber increases observations but leaves chamber and duration confounded. Possible redesigns might independently replicate chambers or assign conditions across repeated chamber periods with suitable control of carryover and timing. Which redesign is valid depends on the biological system and equipment. The Methods should describe that design rather than imply 80 independent assignments.

Social science audit. A school assigns one classroom to a new activity and one to usual teaching. Class membership, teacher and condition may be inseparable. Adding more students within those two classrooms does not solve the missing classroom replication. A feasible exploratory report may still describe what happened, but it must avoid claiming a general treatment effect unsupported by the assignment structure.

Engineering audit. Two controllers are evaluated in twelve independently generated layouts, with ten runs per controller in each layout. The protocol defines the primary outcome as mean error across runs within each layout. There are 240 runs and twelve paired layout summaries. The Methods reports both counts, the scenario generation procedure, seeds, evaluation order and failures. This lets readers distinguish performance variation between layouts from variation between runs.

These audits show why a design diagram often reveals problems faster than a long statistical Methods paragraph.

Write the design so readers can reconstruct it

A useful design paragraph states the system or population, selection procedure, allocation, controls, unit counts, repeated observations, masking, timing and exclusions. Follow it with a table or schematic if several levels are involved. Define the primary comparison before naming the test.

Reporting guidance can help identify omissions. ARRIVE applies to animal research. STROBE addresses reporting for specified observational designs. CONSORT 2025 addresses randomised trials. These resources guide transparent reporting; they do not confer ethics approval or establish that a study was well designed. S03 S04 S05

Practice. A fictional team photographs five fields from each of four specimens and describes “n = 20 independent specimens”. Rewrite the description and identify what is still missing.

Model answer. “Four specimens were measured, with five image fields per specimen, giving 20 fields.” The report still needs to explain how specimens were obtained, whether specimens share a donor or batch, how fields were selected, where treatment was assigned, and how the nested observations were analysed. The arithmetic alone does not identify the correct inferential unit.

Use Template 2 to record these decisions. Before collecting data, ask a colleague to mark every independent assignment or sampling unit and every repeated measurement. Resolve disagreements while the design can still be changed.

Sources

S101: Lazic, S. E., Clarke-Williams, C. J., and Munafò, M. R. (2018). What exactly is ‘N’ in cell culture and animal experiments? PLOS Biology, 16(4), e2005282. DOI: 10.1371/journal.pbio.2005282.

S106: NIST/SEMATECH. Randomized block designs. e Handbook of Statistical Methods, section 5.3.3.2. Accessed 11 November 2026.

S03: ARRIVE. The ARRIVE guidelines 2.0. Official guidance for reporting animal research, accessed 11 November 2026.

S04: STROBE Initiative. STROBE Statement and scope. Official guidance, accessed 11 November 2026.

S05: EQUATOR Network. CONSORT 2025 Statement: updated guideline for reporting randomised trials. Official guideline record, accessed 11 November 2026.