Статистика · Орташа
Understand p values, Bonferroni, Holm and false discovery rates
Interpret p values and compare Bonferroni, Holm and BH using a complete worked example.
13 мин оқу · 2026 ж. 26 қаз.
Бұл бет әлі аударылмаған, сондықтан ағылшын нұсқасын оқып отырсыз.
Негізгі тұжырымдар
- Define the comparison family.
- Distinguish family error from FDR.
- Report estimates alongside adjusted evidence.
Мазмұны
Interpret p values and compare Bonferroni, Holm and BH using a complete worked example.
A p value is meaningful only when the reader knows the hypothesis, test statistic, model assumptions and analysis decisions that produced it. A threshold does not tell you whether the research was well designed, whether the effect matters or whether the conclusion will replicate.
This tutorial explains a single test, then shows why a collection of tests requires additional planning. The numerical examples are synthetic. They teach error control and reporting without claiming discoveries about real outcomes.
Build the logic of one hypothesis test
A null hypothesis specifies a reference claim, such as a population mean difference of zero. An alternative hypothesis describes the departure the procedure is designed to detect. A test statistic summarises the data in a way relevant to that comparison. For a paired t test, it compares the observed mean difference with zero relative to its estimated standard error.
A p value is the probability, under the specified null model and assumptions, of obtaining a test statistic at least as extreme as the observed one according to the test’s definition of extremeness. The definition includes the null hypothesis and assumptions about sampling, dependence and the procedure. It does not refer merely to “chance” in an unspecified sense. The ASA statement stresses that interpretation requires statistical and scientific context. S300
If p = 0.03, you have not calculated a 3% probability that the null is true, a 97% probability that the alternative is true, or a 3% probability of replication failure. Those are different probabilities. A p value also does not measure effect size: a tiny effect can be estimated precisely in a large dataset, while a substantial effect can remain uncertain in a small one.
A two sided test considers departures in either direction. A one sided test uses a specified direction. Decide the direction before examining the data and justify it from the research question. Choosing a favourable direction afterwards changes the procedure and invalidates the advertised interpretation. An unexpected effect in the other direction may still be scientifically important even if it is outside a planned one sided rejection rule.
Distinguish errors, power and importance
A Type I error is rejection of a true null hypothesis. A procedure’s significance level, alpha, controls or bounds this error probability under its conditions. A Type II error is failure to reject a false null for a specified alternative. Power is the probability of rejecting the null at that alternative under the planned design and assumptions. Power depends on the effect, variability, sample size, dependence and decision rule.
None of these quantities is the probability that a specific published finding is true. That requires additional information and a different inferential calculation. A confidence interval and effect estimate give readers more useful context than a threshold label alone. Greenland and colleagues discuss these distinctions and common misinterpretations. S205
Suppose a synthetic controller comparison estimates a 3.58 mm mean reduction with a 95% CI from 2.95 to 4.22 mm. Whether that reduction matters depends on tolerances, cost and application. Its very small p value in the teaching calculation cannot supply those practical criteria. If a different study had a wide interval covering both a meaningful benefit and a meaningful harm, “no significant effect” would conceal the unresolved uncertainty.
Why many tests change the problem
Imagine twenty independent tests, every null hypothesis true, each conducted at alpha 0.05. The probability that none rejects is 0.95²⁰. Therefore the probability of at least one rejection is 1 minus 0.95²⁰, approximately 64.2%. Independence and the assumption that all twenty nulls are true are essential to this calculation. Real tests can be dependent, so do not quote 64.2% as a universal rate.
Multiplicity can arise from several outcomes, several treatment contrasts, several time points, many genes, many survey items or many benchmark tasks. It can also arise through choices: trying several exclusion rules, different transformations or alternative subgroup definitions and reporting the most attractive result. A correction applied only to the final displayed table does not necessarily account for that larger search.
Researcher degrees of freedom are analytical or collection choices that can affect results. When those choices are guided by a desired significance threshold but not disclosed, readers cannot evaluate the actual procedure. Simmons and colleagues demonstrated how such flexibility can inflate false positive findings. S320
Use a protocol to distinguish planned confirmatory analyses from exploration. Exploration is useful for generating ideas, but label it and seek fresh evidence for claims selected because they looked promising. Repeatedly collecting more observations until an ordinary p value crosses 0.05 also changes the error rate. Planned sequential designs can address repeated looks; an unplanned stopping rule cannot be ignored.
Define the family before adjusting it
A comparison family is the set of hypotheses addressed by a chosen error control objective. Define it using the scientific claims, not file boundaries. Five primary outcomes all supporting one broad claim may belong to one family. A prespecified hierarchy of primary and secondary questions can require a different strategy. The correct choice is contextual and should be explained.
For example, a hypothetical school project evaluates five prespecified questionnaire outcomes to assess a mentoring programme. If any positive outcome will be used to claim programme benefit, treating each as an unrelated single test is difficult to defend. In a gene screening project, thousands of features may instead form a discovery family. An engineering paper that compares many algorithms across tasks must explain the claims made across those comparisons, including how dependent measurements are handled.
Before analysis, write the list of hypotheses, reference groups, directions, primary claims and adjustment method. State whether subgroups were planned. Do not make the family smaller by deleting tests that failed to produce favourable results. Template 4 includes a place to document these choices.
Bonferroni controls the chance of any false rejection
The familywise error rate, or FWER, is the probability of at least one false rejection in a family. Bonferroni compares each of m valid p values with alpha / m. Equivalently, multiply each p value by m and cap the adjusted value at one. It provides FWER control without requiring independence, but it can be conservative. S16
For five hypotheses at family alpha 0.05, the individual threshold is 0.01. A raw p value of 0.012 becomes an adjusted p value of 0.060. The underlying estimated effect has not changed. The procedure changes the inferential decision in view of the family.
FWER control may fit a small set of confirmatory claims where any false positive is consequential. It does not protect against an invalid original model, an incorrect independence assumption within a test or selective reporting of an undisclosed larger collection.
Holm uses an ordered sequence
Holm’s procedure also controls FWER under general dependence when the individual tests are valid. Sort the p values from smallest to largest. Compare the smallest with alpha / m, the next with alpha / (m minus 1), and continue. Stop at the first failure; do not reject that hypothesis or any later hypothesis using this rule. Holm is at least as powerful as the basic Bonferroni method. S41 S16
For the synthetic sorted p values 0.003, 0.012, 0.024, 0.041, 0.200, the first comparison is 0.003 ≤ 0.010, so it passes. The second is 0.012 ≤ 0.0125, so it passes. The third is 0.024 ≤ 0.0167, which fails. The procedure therefore rejects only the first two hypotheses.
To obtain Holm adjusted values, multiply the sorted p values by 5, 4, 3, 2 and 1, respectively; then take the cumulative maximum moving from smallest to largest and cap at one. This monotonicity step matters. Simply multiplying by a changing factor without it is not the general adjustment algorithm.
BH controls a different error quantity
The false discovery rate, or FDR, is the expected proportion of false rejections among all rejections across repeated applications, defining that proportion as zero when there are no rejections. It concerns the selected set rather than the probability of any false rejection. The Benjamini and Hochberg procedure, commonly called BH, controls FDR under independence and certain positive dependence conditions. S42
To apply BH at target q, sort m p values. Find the largest rank k satisfying p at rank k ≤ (k / m) × q. Reject every hypothesis at or below that rank. This is a step up rule. You do not stop at the first failure when checking ranks in ascending order.
For the five synthetic values at q = 0.05, the rank thresholds are 0.01, 0.02, 0.03, 0.04 and 0.05. The largest passing rank is three because 0.024 is below 0.03, while the fourth and fifth values fail their thresholds. BH therefore rejects the first three hypotheses.
For adjusted BH values, begin with m × p / rank, take the cumulative minimum from the largest rank back towards the smallest, and cap at one. Returning the adjusted values to the original test order is essential when working with real datasets. The Benjamini and Yekutieli modification offers a more conservative guarantee under arbitrary dependence; specialised data structures may also justify other procedures. S43
FDR control at 5% does not promise that exactly 5% of the discoveries in your particular experiment are false. It also does not assign a 5% probability of being false to each selected feature. These are different claims. A BH adjusted p value should not be silently called a posterior probability or treated as identical to a q value estimated using another method. Storey’s q value approach has its own definition and estimation framework. S321
Compare the complete synthetic family
All five tests are included in every adjustment. The values are teaching inputs, not results from the controller dataset or a real experiment.
| Test | Raw p | Bonferroni | Holm | BH |
|---|---|---|---|---|
| A | 0.003 | 0.015 | 0.015 | 0.015 |
| B | 0.012 | 0.060 | 0.048 | 0.030 |
| C | 0.024 | 0.120 | 0.072 | 0.040 |
| D | 0.041 | 0.205 | 0.082 | 0.05125 |
| E | 0.200 | 1.000 | 0.200 | 0.200 |
At 0.05, Bonferroni rejects A; Holm rejects A and B; BH rejects A, B and C. The difference does not show that one method discovered the “true answer”. The methods use different rules, and FWER and FDR address different error objectives. Choose the objective and method before comparing outcomes.
In an actual paper, add each effect estimate and an appropriately defined interval to the table. This teaching table isolates the adjustment calculations; it is not a complete results table. Adjusting p values does not automatically turn ordinary individual confidence intervals into simultaneous intervals, which are designed to cover a collection of parameters jointly. Label the interval type and avoid implying a guarantee you did not calculate.
Write a transparent Methods and Results account
A useful Methods paragraph for the five outcome example is: “We treated the five prespecified questionnaire outcomes as one comparison family. We applied Holm adjustment to their two sided p values to control the familywise error rate at 0.05. We report all five effect estimates, individual 95% confidence intervals, and raw and adjusted p values. The individual intervals are not simultaneous intervals. Analyses introduced after inspection are identified as exploratory.”
The paragraph names the family, objective, procedure, direction and interval limitation. It does not say that adjustment guarantees correct conclusions. A Results paragraph should describe the size and uncertainty of effects before discussing the chosen decision rule. Replace “Only two outcomes worked” with a statement about the estimated changes, their uncertainty and which comparisons met the prespecified rule.
Report p values to useful precision. If a program displays 0.000, use an appropriate inequality or scientific notation rather than reporting a probability of exactly zero. Avoid “almost significant” or “a trend towards significance” as substitutes for describing an imprecise estimate. A p value of 0.051 does not mark a scientific boundary different in kind from 0.049.
Apply the same reasoning when writing reviews
A review should not classify papers simply as positive or negative according to p < 0.05. Studies may differ in sample size, outcome definition, adjustment, precision and risk of bias. A large p value in a small study can coexist with an estimate similar to that in a larger study. Extract the effect and its uncertainty, identify the comparison family if relevant, and describe whether the analysis was planned.
If a review itself conducts many subgroup or moderator analyses, multiplicity applies to those analyses too. A systematic search does not remove the need for an analysis plan. A meta analysis based on selected published results can also inherit selective reporting from the underlying literature.
Practice and model answers
Exercise 1. A student tests ten outcomes but reports only the two with raw p < 0.05, then adjusts those two. What is wrong?
Model answer. The adjustment omits the selection process and the other tested outcomes. Define and report the full relevant family. Keeping only favourable tests before adjustment does not provide the intended error control.
Exercise 2. A BH procedure at 5% selects twenty features. Can the paper assert that exactly one is false?
Model answer. No. FDR is an expectation over repeated applications, not a guarantee about the realised number of false discoveries. Report the procedure, family, assumptions and effect estimates, and consider independent validation.
Exercise 3. Recalculate the Holm and BH columns in the synthetic table, including the monotonicity steps. Explain in one sentence why BH can reject more hypotheses while targeting a different error quantity.
Sources
Файлдар
- Synthetic teaching data: multiple_testing_example.csv (1 KB)
- Calculation record: verified_calculations.json (1 KB)
- Python script that reproduces the examples: reproduce_examples.py (3 KB)
- Calculation record: statistics_tutorial_calculations.json (5 KB)
- Python script that reproduces the examples: reproduce_statistics_tutorials.py (4 KB)
- Calculation record: computation_environment.json (1 KB)
- About the teaching data: README.txt (8 KB)
Бұл файлдарды жүктеп алу үшін кіріңіз.