P-Values Explained: What a Statistical Test Does and Does Not Say

Start with the question and the null model

A statistical test needs a defined outcome, a null hypothesis, a test statistic, and assumptions about how the data were generated. Those assumptions can include random sampling or assignment, independence, and a suitable probability model.

A p-value is the probability, under that null model and its assumptions, of obtaining a test statistic at least as extreme as the observed one in the direction or directions specified by the test.

The American Statistical Association cautions against interpreting p-values as the probability a hypothesis is true or using a threshold alone to make a scientific or business conclusion. ^[1]

Worked example: a two-sided z-test

Suppose an illustrative experiment estimates a treatment difference of 4 units with a standard error of 2 units. Under a null hypothesis of no difference, the z-statistic is 4 ÷ 2 = 2.

For a two-sided standard-normal test, the p-value is approximately 0.0455. This says that, if the null model and assumptions were correct, a statistic at least as far from zero as ±2 would occur about 4.55% of the time.

It does not say there is a 4.55% probability that the null hypothesis is true. It also does not say there is a 95.45% probability that the treatment works, or that the effect will replicate at the estimated size.

An approximate 95% interval based on the same normal calculation is 4 ± 1.96 × 2, or about 0.08 to 7.92 units. That broad range is important context even though the p-value is slightly below 0.05.

Statistical significance is not practical importance

A tiny effect can produce a small p-value in a very large sample. A potentially meaningful effect can produce a large p-value in a small or noisy sample.

Report the effect in units that matter to the decision: dollars per customer, percentage points of conversion, hours saved, or another relevant quantity. Include uncertainty and compare it with a practically meaningful threshold chosen for the problem.

For example, a well-measured improvement of 0.01 percentage points may still be too small to justify an expensive implementation. A p-value does not contain the implementation cost, downside risk, or value of waiting for better evidence.

“Not significant” is not the same as “no effect”

A large p-value means the data are not especially incompatible with the specified null model according to that test. It does not establish equality or prove that any remaining difference is small enough to ignore.

To argue that two approaches are practically equivalent, define an acceptable difference and use a suitable equivalence or noninferiority framework. Do not infer equivalence merely because a conventional difference test did not cross a threshold.

Look at the interval. If it includes both a substantial benefit and a substantial harm, the experiment may be inconclusive rather than reassuring.

Multiple tests and repeated peeking change the question

Testing many outcomes or segments creates more opportunities to find an apparently unusual result. Repeatedly checking an ordinary fixed-sample test and stopping when it becomes significant can also alter its error behavior.

Plan the primary outcome, analysis, sample size or stopping rule, and treatment of multiple comparisons in advance when feasible. Use a method designed for sequential monitoring when decisions must be made continuously.

Exploratory findings can be valuable. Label them as exploratory and seek additional evidence instead of presenting a discovered pattern as if it had been the original single test.

Use a complete reporting sentence

A useful report names the comparison, sample and design, estimated effect, uncertainty interval, test, p-value, and important limitations. Avoid reducing the conclusion to a green or red badge.

For the illustrative example: “The estimated difference was 4 units, with an approximate 95% interval from 0.08 to 7.92 under the stated normal model; the two-sided p-value was 0.0455.” The decision still requires context about costs, data quality, and the consequences of being wrong.

Use this guide thoughtfully. Examples illustrate a calculation method, not a guaranteed outcome. The usefulness of any result depends on the definitions, measurements, and assumptions used.