Applying Hypothesis Testing in the Real World

Hypothesis testing allows data scientists to objectively assess whether an observed pattern is likely due to chance or reflects a genuine effect.

Null vs. Alternative Hypotheses

Every test starts with a null hypothesis, representing the status quo, and an alternative hypothesis, representing a potential effect. By choosing a significance level and calculating a p-value, we can decide whether to reject the null hypothesis. The logic is indirect, and worth stating carefully. We assume $H_0$ is true, work out how surprising the observed data would be under that assumption, and reject $H_0$ only if the data is surprising enough. Formally, the p-value is

where $T$ is the test statistic. Read that conditioning bar closely: the p-value is the probability of the data given the hypothesis, never the probability of the hypothesis given the data. Those two quantities can differ by orders of magnitude, and conflating them is the single most common error in applied statistics.

Failing to reject $H_0$ is not evidence that $H_0$ is true. It means the data was insufficient to rule it out, which is a statement about your sample size as much as about the world. If you want to argue for the absence of an effect, you need an equivalence test or a Bayesian analysis, not a non-significant p-value.

The Two Ways to Be Wrong

Every decision rule trades off two errors:

Setting a nominal level $\alpha=0.05$ means designing the procedure so that its Type I error is controlled at, or approximately at, 5% under the null assumptions. In discrete or composite-null problems, the actual rejection probability need not equal 5% at every null parameter value. For a fixed design and test statistic, lowering $\alpha$ reduces Type I error and generally reduces power. More information—through larger samples, better measurement, stronger design, or more efficient modeling—can improve power without increasing the nominal Type I error level.

Power deserves more attention than it usually receives. An underpowered study is not merely inconclusive; it is actively misleading, because the effects that do reach significance in a small sample must be large, and are therefore systematically overestimated. This inflation is sometimes called the winner's curse, and it is a major driver of results that fail to replicate.

Choosing a Test

The test follows from the structure of the question and the data:

• Comparing two independent group means: use a mean-based procedure such as Welch's t-test or a regression model. Mann-Whitney answers a rank/distribution question and is not simply the same mean test without normality.
• Comparing paired measurements: use a paired mean model if the estimand is a mean difference; Wilcoxon signed-rank targets the distribution of paired differences under stronger symmetry assumptions.
• Comparing three or more group means: use ANOVA/Welch-type or regression approaches for means. Kruskal-Wallis is rank-based and does not preserve the same estimand automatically.
• Testing association between categorical variables: chi-square test of independence, or Fisher's exact test when expected counts are small.
• Testing a regression coefficient: $t$-test on the coefficient, with an $F$-test for groups of coefficients.

Each carries assumptions. The $t$-test assumes independent observations and approximate normality of the sampling distribution of the mean, and Welch's version should be the default because it does not additionally assume equal variances. The chi-square reference distribution is asymptotic and can be inaccurate in sparse tables. The familiar “expected count at least 5” rule is only a heuristic; exact or simulation-based calibration may be preferable for small or sparse tables.

A Concrete Example

Suppose a checkout redesign is tested against the existing flow:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
import numpy as np
from scipy import stats

control   = np.array([  # seconds to complete checkout
    42, 51, 38, 47, 55, 44, 49, 41, 53, 46,
    50, 39, 45, 52, 48, 43, 56, 40, 47, 51])
treatment = np.array([
    38, 44, 35, 41, 47, 39, 43, 36, 45, 40,
    42, 34, 40, 46, 41, 37, 48, 35, 42, 44])

t_stat, p_value = stats.ttest_ind(control, treatment, equal_var=False)

diff = control.mean() - treatment.mean()
pooled_sd = np.sqrt((control.var(ddof=1) + treatment.var(ddof=1)) / 2)

print(f"difference in means: {diff:.2f} seconds")
print(f"t = {t_stat:.3f}, p = {p_value:.5f}")
print(f"Cohen's d          : {diff / pooled_sd:.3f}")

The p-value tells you the difference is unlikely under the null. It says nothing about whether a few seconds of checkout time matters to the business. That judgement requires the effect size and its confidence interval, which is why reporting a p-value alone is never sufficient.

Common Pitfalls

Misinterpreting p-values or failing to consider effect sizes can lead to misguided conclusions. Always pair statistical significance with domain context to ensure results are meaningful. Several specific traps recur often enough to name.

Multiple comparisons. Testing 20 hypotheses at $\alpha = 0.05$ gives roughly a 64% chance of at least one false positive, since $1 - 0.95^{20} \approx 0.64$. Bonferroni correction divides $\alpha$ by the number of tests and is simple but conservative; controlling the false discovery rate via Benjamini-Hochberg is usually the better choice when many tests are genuinely exploratory. Optional stopping. Checking results as data accumulates and stopping when $p < 0.05$ inflates the false-positive rate dramatically, because you are effectively running many tests. Fixing the sample size in advance, or using a sequential design built for repeated looks, is the fix.

Dichotomising a continuum. There is no meaningful difference between $p = 0.049$ and $p = 0.051$, yet the 0.05 threshold treats them as opposites. Report the actual value alongside an interval estimate. Confusing statistical and practical significance. With a large enough sample, a trivially small effect becomes statistically detectable. Significance answers "is it distinguishable from zero", not "is it big enough to act on".

Reporting Results Honestly

A useful report contains the effect size in the units people care about, a confidence interval showing the range of values compatible with the data, the sample size, the test used and why, and the p-value as one input among several. It also discloses how many analyses were run and whether the hypothesis was specified before or after seeing the data. The American Statistical Association's 2016 statement on p-values makes the underlying point directly: scientific conclusions should not be based only on whether a p-value passes a threshold. Hypothesis testing is a tool for quantifying one specific kind of surprise, and it works well when used for exactly that and nothing more.

References

• Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129-133.
• Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum.
• Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate. Journal of the Royal Statistical Society: Series B, 57(1), 289-300.
• Gelman, A., & Carlin, J. (2014). Beyond power calculations: Assessing Type S and Type M errors. Perspectives on Psychological Science, 9(6), 641-651.