Мазмұнға өту

Статистика · Орташа

Choose an analysis that matches the design

Choose a model from the question, outcome and dependence structure.

13 мин оқу · 2026 ж. 26 қаз.

Бұл бет әлі аударылмаған, сондықтан ағылшын нұсқасын оқып отырсыз.

Негізгі тұжырымдар

  • Identify the estimand and design.
  • Understand assumptions and diagnostics.
  • Use comparisons that answer the question.
Мазмұны

Choose a model from the question, outcome and dependence structure.

A statistical test is a procedure for answering a particular question under particular assumptions. It is not a general detector of scientific truth. Choosing one by searching for a familiar command, or by trying several until a small p value appears, disconnects the calculation from the study.

Begin with the question, define the outcome, identify the independent unit and decide what comparison matters. Then choose a model or procedure capable of estimating that quantity. This article offers a decision process, worked examples and a reporting framework. It is a starting guide, not a substitute for specialist advice on complex designs.

Translate the question into an analysis problem

An analysis problem should identify the estimand, meaning the target quantity; the observations available; and their dependence structure, meaning which observations share a source or are otherwise related.

For natural science, a question might be: “What is the mean difference in plant dry mass between two watering conditions, with pots independently assigned to conditions?” If several plants share a pot, the pot may be the experimental unit. For social science: “How does mean confidence differ between students who did and did not attend mentoring, after adjustment for specified baseline characteristics?” Because attendance was not randomised, this is initially an association question. For engineering: “What is the average change in positioning error when the same layouts are evaluated using two controllers?” This is a paired comparison.

Write the analysis target before examining the outcome. State what a positive result means, which group is the reference, what time point is primary and whether the analysis was planned or exploratory. One dataset can support several questions, but each requires its own clear definition.

Identify the outcome type and the unit

A continuous outcome can take values along a numerical scale, such as length or voltage. A binary outcome has two categories, such as pass or fail. A count records events, such as collisions, and may need an exposure denominator. An ordinal outcome has ordered categories whose distances are not necessarily equal, such as an individual agreement item. A validated questionnaire total may need different treatment from one item.

A paired design links observations from the same unit or a prespecified matched pair. A clustered design contains groups of related observations, such as students within classes. Repeated measures follow a unit across times or conditions. These structures are not cosmetic details. They affect uncertainty, interpretation and the number of independent contributions to the analysis.

Question and data structure Possible starting method What you must check
Mean difference between independent groups Welch t procedure or regression Independence, target mean and influence of extreme values
Mean difference within matched pairs Paired t procedure Independent pairs and distribution of their differences
Several groups or predictors Regression or an appropriate ANOVA model Coding, contrasts, residual pattern and dependence
Probability of a binary event Binomial model, logistic regression or a design specific comparison Outcome coding, sparse cells and interpretation of odds versus risks
Events per period of exposure Poisson or negative binomial model Exposure duration, dispersion and dependence
Ordered response categories Ordinal model or a justified rank method Ordering assumptions and the question being estimated
Repeated or clustered observations Mixed model, generalised estimating equations or a suitable summary analysis Hierarchy, cluster count and covariance assumptions
Time until an event Survival analysis Time origin, censoring, competing events and model assumptions

This table provides candidates, not automatic answers. A model’s distribution and link function, the mathematical relationship between predictors and the modelled mean, must match the outcome and interpretation. For example, logistic regression uses the log odds of a binary event; a Poisson model with a log link expresses effects multiplicatively on the mean count or rate. S316

Understand an independent mean comparison

Welch’s t procedure compares means of independent groups without imposing equal population variances. Its estimated standard error is the square root of the sum of each group’s variance divided by its sample size. Its degrees of freedom are calculated from those contributions and need not be an integer. The method still requires an appropriate sampling model and does not solve dependence, strong contamination or selection bias. S15

Consider a wholly synthetic plant example. Dry masses in grams are:

Condition A: 12, 14, 11, 13, 15, 12, 14, 13

Condition B: 15, 19, 13, 18, 22, 16, 20, 17, 14, 21

Assume, solely for the demonstration, that every value represents an independently assigned pot with one plant and that a mean comparison is appropriate. Condition A has n = 8, mean 13.00 g and SD 1.31 g. Condition B has n = 10, mean 17.50 g and SD 3.03 g. The B minus A difference is 4.50 g. The Welch standard error is 1.063 g, the degrees of freedom are 12.80 and the 95% CI is 2.20 to 6.80 g. The t statistic is 4.23 and the two sided p value is approximately 0.00101.

A useful results sentence is: “In the synthetic example, mean dry mass was 4.50 g higher in condition B than A, with a Welch 95% confidence interval from 2.20 to 6.80 g; n = 10 and n = 8 independent pots, respectively.” If this were real research, describe randomisation, cultivation conditions, measurement and exclusions in Methods before interpreting the contrast. The numerical procedure alone would not establish those facts.

Why not first test whether the variances differ and then choose a t test? A preliminary variance test adds another uncertain decision and can miss meaningful differences in small samples. Select the analysis using the design, estimand and plausible variance structure, then assess its adequacy. A variance assumption should be justified, not accepted simply because a preliminary test failed to reject it.

Preserve the information in pairing

A paired t procedure operates on the difference within each pair. In the supplied twelve layout controller example, the mean A minus B difference is 3.58 mm with a 95% CI from 2.95 to 4.22 mm. The analysis contains twelve paired contrasts, not twenty four unrelated controller values. Pairing often reduces irrelevant between unit variation, but its benefit depends on the relationship within pairs. S28

Suppose a student has a confidence score before and after a workshop. The paired analysis of change can describe the average change among those students. Without an appropriate comparison or stronger design, it cannot isolate workshop effects from maturation, concurrent events or repeated testing. Correctly handling pairing is necessary, but it does not by itself establish causality.

Do not create pairing by sorting two independent groups and matching their ranks. The relationship must arise from the study design. For genuine matched designs, state the matching variables and how the matched structure was incorporated into the analysis.

Check assumptions with both reasoning and diagnostics

An assumption is a condition under which an analysis has the advertised interpretation or performance. Some assumptions concern how data were collected; others concern the statistical model. No single diagnostic can certify all of them.

Independence is mainly a design issue. Multiple readings from the same sensor or cells from the same culture may share influences. An ordinary normality test cannot detect or correct that structure. If a study contains few independent clusters, treating cluster adjustment as a routine software option can give unreliable uncertainty. Report the number of clusters as well as the number of rows.

A residual is an observed value minus its fitted value. Plot residuals against fitted values to inspect curvature and changing spread. Plot them against time or collection order when drift is plausible. Inspect influential observations, which are points whose inclusion substantially changes an estimate. A high R squared value is not a sufficient model check. S29

For a paired t procedure, normality concerns the distribution of the paired differences for exact small sample inference, not whether each original condition separately passes a test. In larger samples, some departures may be tolerable, but severe skew, heavy tails and unusual influential observations still deserve attention. A normality test with few observations has little ability to detect problems. With many observations, it may detect minor departures that do not matter for the target analysis. Report the relevant evidence and sensitivity checks rather than a ritual list of assumption p values.

Know what rank methods do and do not answer

A nonparametric method does not mean a method without assumptions. Rank procedures replace numerical magnitudes with order information, which can be useful but changes what is being tested or estimated.

The Mann Whitney procedure is not universally a test of whether group medians differ. A median or location shift interpretation needs additional distributional conditions. The Wilcoxon signed rank procedure for paired observations also has a symmetry assumption for its usual location interpretation. R’s official documentation explicitly distinguishes the estimated location parameter from a simple difference in sample medians. S304

Imagine two engineering devices with similar typical errors, but one occasionally produces severe failures. A rank method and a mean comparison may emphasise different properties. The scientific question should decide whether average cost, median error, failure probability or another quantity is primary. Switching procedures changes more than a p value.

A transformation can also change the estimand. Modelling a positive outcome after taking logarithms often makes ratios or multiplicative differences natural to interpret. Exponentiating the mean of log values gives a geometric mean, not generally the arithmetic mean. Explain the transformed scale and do not silently describe its coefficient as an original scale mean difference.

Use contrasts and interactions to answer specific questions

An omnibus test asks about a collection of parameters, such as whether all group means are equal. A contrast addresses a specific comparison, such as condition B versus A. A significant omnibus result does not identify which contrast differs or how much. Plan scientifically meaningful contrasts and decide how the family of comparisons will be handled.

An interaction means that one association or effect changes with another variable on the model’s chosen scale. Test that difference directly. “Significant among younger students but not among older students” is not sufficient evidence that the groups differ. Gelman and Stern explain this common error. S204

Here is a synthetic calculation using independent normal approximation estimates. The estimated effect in group A is 4 points with SE 1.8; in group B it is 3 points with SE 2.2. Their separate two sided p values against zero are approximately 0.026 and 0.173. Yet the estimated difference between effects is only 1 point. Its SE is square root of (1.8² + 2.2²) = 2.84, and its 95% interval runs from negative 4.57 to positive 6.57 points. The direct comparison is very imprecise. A difference between threshold labels concealed that fact.

When estimates are correlated, their covariance must enter the difference calculation. Do not apply the independent estimate formula indiscriminately to coefficients from one model.

Distinguish absence of evidence from equivalence

A large p value for a test of no difference does not establish that conditions are equivalent. Equivalence testing asks whether differences larger than prespecified meaningful bounds can be excluded with the chosen procedure. Noninferiority asks whether a new option is not worse than a comparator by more than a justified margin. Both need design and margin decisions before looking at results. S303

For a synthetic equivalence illustration, suppose the estimate is 0.20 units with SE 0.20 and 39 degrees of freedom. A 90% t interval is approximately negative 0.137 to positive 0.537 units. If justified equivalence bounds were negative 1 to positive 1, this interval falls within them and corresponds to the usual two one sided tests procedure at alpha 0.05. The bounds must be defensible for the application; selecting them after seeing this interval would weaken the intended confirmatory claim.

Recognise when a different inferential framework is appropriate

Bayesian analysis combines a likelihood, describing the data model, with a prior distribution, representing assumptions about parameters, to obtain a posterior distribution. A credible interval summarises posterior uncertainty conditional on that model and prior. It is not a frequentist confidence interval with a new name. Report the prior, likelihood, fitting diagnostics and sensitivity to plausible alternatives. Prior and posterior predictive checks assess whether the model generates plausible data patterns. S318

Do not select a framework simply because one produces a preferred conclusion. The choice should support the question and available information. The same expectation of transparent assumptions applies to frequentist, Bayesian and resampling procedures.

Write the statistical analysis section before the results

A good analysis paragraph answers what was estimated, at what level, using which method, with which uncertainty measure and planned comparisons. It states missing data handling, exclusions, relevant diagnostic procedures and software versions. “Data were analysed using statistics software” provides almost none of this information.

For the synthetic independent plant example, an appropriate model paragraph is: “We defined the primary contrast as mean dry mass in condition B minus condition A. Each constructed record represented one independent pot. We used a two sided Welch t procedure and report the mean difference with its 95% confidence interval. No records were missing or excluded. Individual values and group distributions accompany the estimate. This teaching calculation assumes independent observations and a defensible mean model.”

In a real paper, describe only checks and procedures you actually performed. If a diagnostic led to an unplanned change, record its timing and rationale and report the sensitivity of the result. Use Template 4 to make the plan explicit.

Practice and model answers

Exercise. You test two controllers in six layouts, with fifty runs per controller per layout. Can a comparison treat all six hundred runs as independent?

Model answer. Not automatically. Runs share layouts and may share seeds or other conditions. Define whether the target is a contrast across layouts, runs, or both. A suitable hierarchical model or a prespecified summary per layout may be appropriate. The analysis must retain the paired controller comparison and account for the relevant levels of dependence.

Exercise. A review reports that four studies were significant and three were not. Does that establish a difference between sets of studies?

Model answer. No. Extract effect estimates, uncertainty, design and populations. Different sample sizes can produce different significance labels for similar effects. A defensible comparison or synthesis addresses the estimates and their heterogeneity rather than counting labels.

Sources

S15: NIST/SEMATECH. e Handbook of Statistical Methods. Section 1.3.5.3, Two Sample t Test for Equal Means.

S28: NIST/SEMATECH. e Handbook of Statistical Methods. Section 7.3.1.1, Analysis of paired observations.

S29: NIST/SEMATECH. e Handbook of Statistical Methods. Section 4.4.4, How can I tell if a model fits my data?

S303: Lakens D. 2017. Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta Analyses. Social Psychological and Personality Science 8(4), pages 355 to 362. DOI: 10.1177/1948550617697177.

S304: R Core Team. R stats documentation: Wilcoxon Rank Sum and Signed Rank Tests. Accessed 11 November 2026.

S204: Gelman A and Stern H. 2006. The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant. The American Statistician 60(4), pages 328 to 331. DOI: 10.1198/000313006X152649. Author manuscript linked.

S316: R Core Team. R stats documentation: Family Objects for Models. Accessed 11 November 2026.

S318: Gelman A, Vehtari A, Simpson D and colleagues. 2020. Bayesian Workflow. arXiv:2011.01808. Methodological preprint.

Файлдар

Бұл файлдарды жүктеп алу үшін кіріңіз.