Статистика · Средний
Handle missing data, outliers and analytical uncertainty
Document missing observations, unusual values and sensitivity to analytical choices.
14 мин чтения · 26 окт. 2026 г.
Эта страница ещё не переведена, поэтому вы читаете английскую версию.
Главное
- Investigate before excluding.
- State missingness assumptions.
- Resample the correct unit.
Содержание
Document missing observations, unusual values and sensitivity to analytical choices.
Real datasets contain incomplete records, unusual observations and decisions that are not resolved by a software default. A strong paper makes those issues visible and explains how they affect the answer. A clean looking table is not a sufficient reason to delete a row, replace a value or choose an analysis.
The goal is to preserve the connection between the research question and the data actually available. This tutorial explains missingness, defensible corrections, imputation, sensitivity analysis and resampling. All numerical scenarios below are synthetic teaching examples.
Preserve the original records and document every change
Keep an unchanged copy of the raw data. Create a separate analysis dataset using a script or documented sequence of transformations. Record what changed, why it changed, who made the decision and whether the decision occurred before or after outcome inspection. An audit trail should allow a reader to distinguish a verified correction from an analytical choice.
Start by inspecting units, allowed ranges, duplicate identifiers and impossible combinations. An age of 200 years might be a transcription error; a very large sensor reading might be a valid failure. A correction requires evidence, such as an original record or calibration log. Do not change a value merely because it is inconvenient or differs from a group average.
Use explicit missing codes. Zero is a measured value in many datasets, not a universal code for missingness. A blank, “not applicable”, “not measured” and “below detection limit” can represent different processes. Keep those distinctions in the data dictionary and explain how the analysis treats them.
Count missingness at the level of the question
Report the number missing for each important variable, group and time point, together with the relevant denominator. Also report the number of complete records for each model. Different analyses can silently use different samples if software drops rows with missing values.
For a survey, distinguish students who never responded from respondents who skipped one question. For a biological experiment, distinguish a sample that was never collected from an assay that failed after collection. For engineering, distinguish an absent log file from a run that failed to complete. A failed run may itself be an outcome. Excluding it from a performance comparison can favour a system that succeeds only in easier conditions.
Create a missingness summary before testing the main hypothesis. Examine whether missingness varies with recorded baseline characteristics or group assignment, and consult collection notes. These checks can challenge assumptions, but observed data alone generally cannot determine the values that were never measured.
Understand MCAR, MAR and MNAR
Missing completely at random, or MCAR, means that the missingness process does not depend on observed or unobserved values in the relevant model. A truly random loss of sample labels unrelated to sample properties could be an illustration. Do not assume that instrument failure is MCAR: failure might occur more often at high concentrations or under particular conditions.
Missing at random, or MAR, means that after conditioning on relevant observed information, missingness does not additionally depend on the missing values themselves. For example, suppose follow up responses depend on recorded baseline confidence, and within levels of the observed information they do not depend on the unobserved follow up confidence. That is a possible MAR assumption. It does not mean that values are missing randomly in ordinary language.
Missing not at random, or MNAR, permits dependence on the missing values after conditioning on the observed information. Students with lower unrecorded follow up confidence might be less likely to respond even after accounting for recorded baseline confidence. The distinction is about the process and conditioning information, not a label assigned by one diagnostic test. Rubin’s foundational paper sets out conditions under which the missingness process can be ignored for particular forms of inference. S311
Treat these assumptions as claims requiring justification. A test that fails to detect differences between responders and nonresponders does not prove MAR or MCAR. It cannot compare values that are absent. Explain plausible reasons for missingness and evaluate how conclusions change under alternatives.
Work through a simple sensitivity calculation
Imagine a fictional programme enrols forty students. Thirty provide follow up scores with a mean of 70 points on a scale from zero to one hundred. Ten scores are missing, so the response fraction is 75%. The observed mean describes the thirty responders. It is not automatically the mean for all forty students.
If the ten missing scores averaged 50 points, the mean for all students would be:
(30 × 70 + 10 × 50) / 40 = 65 points
If they averaged 90 points, the overall mean would be:
(30 × 70 + 10 × 90) / 40 = 75 points
These are assumptions, not recovered measurements. The calculation shows how the target mean depends on unobserved outcomes. It does not supply confidence intervals or account for sampling uncertainty. A formal analysis would combine a defensible outcome model, missingness assumptions and uncertainty estimation.
A useful teaching sentence is: “The observed mean was 70 points among thirty of forty constructed students. Changing the assumed mean of the ten missing scores from 50 to 90 points changed the full group mean from 65 to 75 points. The example therefore shows that the enrolled group mean is sensitive to assumptions about missing outcomes.”
This reasoning is often more informative than a vague claim that missing values were “few”. The consequences depend on their distribution and relationship to the question, not only their percentage.
Choose a missing data method deliberately
Complete case analysis uses records with all variables needed for the analysis. It is simple and can be valid under particular conditions, but it can lose precision and produce bias. MCAR is a sufficient condition for some common uses, not a universal necessary condition for every regression estimand. State the justification for the specific analysis rather than declaring complete cases always valid or always invalid.
Replacing every missing value with the observed mean typically distorts variation and relationships, and treats the replacements as more certain than they are. Carrying the last observation forward assumes a particular outcome path that may be implausible. Neither should be a reflexive response to empty cells. Sterne and colleagues explain the potential and limitations of multiple imputation and the importance of reporting the missing data process. S37
Multiple imputation creates several completed datasets by drawing plausible missing values from a specified model. Each completed dataset is analysed using the intended analysis model. Estimates and their uncertainty are then combined, preserving both uncertainty within completed datasets and variation among their results. It does not reveal what the missing observations actually were.
A defensible imputation workflow requires choosing variables, respecting data types and allowable ranges, preserving relevant nonlinear terms and interactions, and accounting for clustering when needed. Include useful observed predictors of missingness and incomplete values when appropriate. Check the plausibility of generated values and convergence of the imputation process. White, Royston and Wood discuss these implementation decisions and their pitfalls. S310
Do not average completed datasets into one table and analyse that table as if observed. Do not average the p values from the separate analyses. The pooling step must combine estimates and uncertainty using a method appropriate to the analysis. The mice methods paper describes imputation, diagnostics and pooling workflows. S317
When reporting multiple imputation, identify the imputation model, variables, number of imputations, relevant iteration and diagnostic settings, pooling method and assumptions. Explain how the imputation and analysis models are compatible. The appropriate number of imputations depends on the missing information and desired numerical stability; a small fixed number is not suitable for every project.
Recognise censoring and structural absence
A value below an assay’s detection limit is known to lie within a range. It is not simply an unknown value with no information. A device that has not failed by the end of observation provides a lower bound on its lifetime, which is a form of censoring. These situations can require specialised likelihoods or survival methods rather than ordinary imputation of unrestricted values.
A question that does not apply to a participant creates a structural absence. For example, a question about hours in paid employment may not apply to someone reporting no employment, depending on the questionnaire design. Decide whether the correct value is zero, not applicable or undefined from the measurement definition. Do not make that decision solely to simplify an analysis.
Explain each process in Methods. If the available information cannot identify an outcome, say so and limit the claim. A transparent limitation is more defensible than a fabricated precision from an inappropriate replacement rule.
Investigate outliers without treating them as mistakes
An outlier is an observation that appears unusual relative to a specified distribution or model. Whether it is unusual depends on the context. A box plot flag identifies a point for inspection; it is not an instruction to delete it. NIST distinguishes identifying unusual observations, checking for errors and using methods that accommodate valid extremes. S314
Consider ten synthetic sensor errors in millimetres:
2, 3, 3, 4, 4, 4, 5, 5, 6, 24
The mean is 6.00 mm and the median is 4.00 mm. Removing the value 24 changes the mean to 4.00 mm. A 10% trimmed mean, removing one observation from each tail before averaging, is 4.25 mm. These are different summaries, not interchangeable versions of the same answer.
Suppose the sensor log confirms that the 24 mm observation occurred during normal operation. It may represent an important failure mode. The primary mean should include it if the research question concerns average performance under those conditions. A median or trimmed mean can describe typical or central performance, but cannot erase the practical relevance of a rare failure.
If an original calibration record instead proves that 24 arose from a known unit conversion error, correct it through the documented workflow. If the value is ambiguous, present the main analysis under the justified rule and a sensitivity analysis showing the alternative. Explain the uncertainty rather than selecting whichever result looks better.
Design sensitivity analyses around a scientific concern
A sensitivity analysis asks how an answer changes under a plausible alternative assumption or decision. A robustness check usually asks whether the main interpretation persists under reasonable analytical variations. Neither is a search for an analysis that restores significance.
Start with the threat: missing outcomes, influential observations, a processing threshold, a model form or an unmeasured confounder. Specify a reasonable alternative and explain why it is plausible. Keep the outcome definition and target quantity clear. Changing from a mean to a median changes the estimand, so agreement between them is not automatically evidence that one mean estimate is robust.
For the synthetic forty student example, a simple sensitivity parameter could shift the assumed mean of missing scores twenty points below the observed mean. Because one quarter of the scores are missing, that shift lowers the overall mean by five points. In a formal imputation analysis, analogous shifts can explore departures from MAR, but the implementation must propagate uncertainty and respect the scale’s limits.
For engineering, examine performance when failed runs remain in the evaluation and under an appropriate alternative failure summary. For biology, inspect whether a documented assay quality threshold drives the result. For social science, compare conclusions under plausible missing outcome assumptions or coding choices. Report the planned primary analysis alongside these alternatives, including disagreements.
A useful limitations paragraph explains the consequence: “The direction of the estimated association was unchanged, but its magnitude depended on the missing outcome assumption.” An unhelpful paragraph merely states “Sensitivity analyses were performed” without showing what changed or why.
Bootstrap the unit that contributes independent information
The bootstrap approximates uncertainty by repeatedly drawing samples with replacement from the observed data and recalculating a statistic. In the ordinary bootstrap for independent observations, each resampled dataset has the original sample size. Some units can appear several times and others not at all. The resulting distribution reflects variation under that resampling model. NIST introduces the idea and cautions that it is not appropriate for every statistic or distribution. S33
For paired controller values, resample complete layout pairs, preserving the relationship between A and B. Resampling the A and B columns independently destroys that relationship. SciPy’s bootstrap documentation explicitly distinguishes paired resampling from independent sample resampling. S312
For repeated student observations, a suitable cluster bootstrap might resample students with all of their records. Time series may require blocks or a different procedure because neighbouring values are dependent. The appropriate scheme depends on the sampling and target inference. A small number of clusters cannot be transformed into a large independent sample by drawing many bootstrap resamples.
Report the statistic, resampling unit, number of resamples, interval method and random seed or reproducibility settings. Percentile, basic and bias corrected intervals are different procedures. Increasing the number of resamples reduces computational noise; it does not repair an unrepresentative sample, inappropriate resampling model or inadequate number of independent units.
Permute according to a justified null model
A permutation test compares the observed statistic with values generated by rearrangements justified under a null hypothesis or randomisation design. Exchangeability means that the rearranged observations or labels can be treated as interchangeable under the relevant assumptions. It is not permission to shuffle any column.
For a paired randomised comparison, a valid scheme may exchange treatment labels within pairs according to the original allocation restrictions. For a blocked design, the allowed rearrangements may need to remain within blocks. For independent groups, unrestricted label permutation typically assumes a stronger exchangeability condition than simply equal means when distributions differ. SciPy documents distinct permutation structures for independent samples, paired samples and association questions. S313
State the null, statistic, allowed rearrangements and whether the calculation was exact or based on a random subset. A procedure called “nonparametric” still needs a defensible relationship between the rearrangements and the study design.
Report the decisions where readers can evaluate them
Methods should state missingness handling and exclusion rules. Results should report how many observations were available, corrected and excluded, with reasons and relevant denominators. Present the primary result and consequential sensitivity results. The Discussion should explain what uncertainty remains and how it limits the claim.
In a review, record missing outcome data and selective reporting in the included studies. A study that does not report an outcome is not a study showing no effect on that outcome. Missing study results and missing participant measurements are different problems, and both can affect a synthesis.
Use Template 16 for the decision trail and Template 4 for planned handling and sensitivity analyses. The records should make it possible to understand the choices without trusting the authors’ memory.
Practice and model answers
Exercise 1. An algorithm crashes in difficult scenarios, and the paper calculates mean error only for completed runs. What might this conceal?
Model answer. The evaluated sample may exclude precisely the scenarios in which performance is poor. Define completion and failure outcomes, retain the relevant denominator and explain how performance is assessed across all planned evaluations.
Exercise 2. A researcher draws one million bootstrap resamples from four independent animals and describes the analysis as having a very large sample. What is wrong?
Model answer. Resamples are computations from the same four animals. They do not add biological information or guarantee reliable population inference. The number of independent animals remains four.
Exercise 3. A suspicious value changes the p value from above to below 0.05 when deleted. What should determine its handling?
Model answer. The measurement evidence, quality rule and target question should determine handling. The p value change is a reason to disclose sensitivity, not a scientific reason for exclusion.
Sources
S33: NIST/SEMATECH. e Handbook of Statistical Methods. Section 1.3.3.4, Bootstrap Plot.
S314: NIST/SEMATECH. e Handbook of Statistical Methods. Section 1.3.5.17, Detection of Outliers.
Файлы
- Calculation record: statistics_tutorial_calculations.json (5 KB)
- Python script that reproduces the examples: reproduce_statistics_tutorials.py (4 KB)
- Calculation record: computation_environment.json (1 KB)
- About the teaching data: README.txt (8 KB)
Войдите, чтобы скачать эти файлы.