Chi-Square Tests: Expected Counts, Association, and Effect Size

Pearson's chi-square statistic is one formula used in several related categorical-data tests. The common structure is

where $O$ is an observed count and $E$ is the count expected under a specified null model. The important part is not the formula. It is how the expected counts were obtained.

Goodness-of-fit

Suppose one categorical variable has $k$ categories. Let

be the observed counts and let the null model specify category probabilities

For total sample size $n$, the expected counts are

Pearson's statistic is

If the null probabilities are fully specified and the usual asymptotic conditions hold,

If parameters used to compute $p_i$ are estimated from the same data, the degrees of freedom must be reduced accordingly and the ordinary reference distribution may require more care.

Independence in a contingency table

Now suppose two categorical variables form an $r\times c$ table. Let

be the observed count in row $i$, column $j$. Under independence,

The fitted expected count is

The statistic is

Under the usual asymptotic conditions,

This tests association. It does not estimate a causal effect.

Why the same formula appears twice

The goodness-of-fit and independence tests are both comparisons between observed counts and counts implied by a null model. In the goodness-of-fit problem, the expected probabilities are specified externally or by a fitted model. In the independence problem, expected counts are fitted under the factorization constraint

The chi-square statistic measures discrepancy from the corresponding constrained model.

The sampling design matters

The same numerical contingency table can arise from different designs. For example:

• multinomial sampling with fixed total $n$;
• independent multinomial samples with fixed row totals;
• product-binomial sampling in a case-control or cohort design;
• Poisson sampling of cell counts.

The large-sample Pearson statistic can look similar across these formulations. But the meaning of parameters and appropriate effect measures can differ. A two-by-two table from a cohort naturally supports risk ratios and risk differences. A case-control sample generally does not estimate population risks directly from the sampled row totals.

The table alone does not encode the design.

Expected counts and the chi-square approximation

The chi-square reference distribution is asymptotic. Small expected counts can make it inaccurate. The common rule

every expected cell count must be at least 5

is a heuristic, not a theorem. What matters is the entire table structure, sparsity, dimension, and how extreme the expected counts are. When counts are sparse, options include:

• exact conditional tests;
• Monte Carlo calibration;
• likelihood-based models;
• category aggregation when scientifically defensible.

Combining categories solely to satisfy a rule can change the estimand and discard meaningful information.

Fisher's exact test

For a two-by-two table with fixed margins, Fisher's exact test conditions on the row and column totals. Under the null, the cell count follows a hypergeometric distribution. This gives exact finite-sample calibration under that conditional sampling model. “Exact” does not mean universally superior. The test conditions on margins and can be conservative depending on the inferential target.

It is one tool for sparse two-by-two data, not a generic replacement for every chi-square test.

A significant chi-square statistic says only that the model does not fit

If an independence test rejects, we know that the observed table is incompatible with independence at the chosen level. We do not yet know:

• which cells drive the discrepancy;
• the direction of association;
• the magnitude of association;
• whether the difference is practically important.

The omnibus p-value should therefore be followed by residuals and effect sizes.

Pearson residuals

A simple residual is

Large absolute residuals identify cells contributing strongly to the Pearson statistic. Because cell residuals are not independent and their variance is affected by fitted margins, adjusted standardized residuals are often more useful for diagnostic interpretation. The point is to move from

to the structure of the departure.

Effect size

For an $r\times c$ table, Cramér's $V$ is

It ranges from 0 to 1. It summarizes association strength but does not reveal direction. For a two-by-two table, odds ratios, risk ratios, and risk differences can be more interpretable depending on the sampling design and scientific question. A p-value and an effect size answer different questions.

The chi-square test is not “non-parametric” in the sense of assumption-free

The statistic does not require normally distributed observations. That does not make it assumption-free. The analysis still depends on:

• independent sampling units or an appropriate dependence model;
• a correctly specified null structure;
• adequate asymptotic approximation or an exact alternative;
• correct classification of observations into categories.

Clustered survey data, repeated measures, or matched pairs violate the ordinary independence formulation. For those designs, a standard Pearson chi-square test can have the wrong variance.

Paired binary data need McNemar's test

Suppose the same subjects are measured before and after an intervention. The two responses are paired. A standard two-by-two independence test treats the counts as if the observations came from independent groups. That is incorrect. McNemar's test focuses on discordant pairs and is designed for paired binary data. Study design comes before table format.

Reproducible Python example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
from __future__ import annotations

import numpy as np
from scipy.stats import chi2_contingency

table: np.ndarray = np.array(
    [
        [42, 18, 10],
        [26, 24, 20],
    ],
    dtype=int,
)

result = chi2_contingency(
    table,
    correction=False,
)

chi2: float = float(result.statistic)
p_value: float = float(result.pvalue)
expected: np.ndarray = result.expected_freq

n: int = int(table.sum())
rows, cols = table.shape

cramers_v: float = float(
    np.sqrt(
        chi2
        / (
            n
            * min(rows - 1, cols - 1)
        )
    )
)

pearson_residuals: np.ndarray = (
    table - expected
) / np.sqrt(expected)

print(f"X^2 = {chi2:.3f}")
print(f"p = {p_value:.4f}")
print(f"Cramer's V = {cramers_v:.3f}")
print(pearson_residuals)

The omnibus statistic, effect size, and cell diagnostics should be interpreted together.

Conclusion

A chi-square test is a model-discrepancy test for counts. The core logic is

The formula is easy. The statistical work lies in defining the correct expected counts and the correct sampling model.

References

• Pearson, K. (1900). On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. Philosophical Magazine, 50, 157–175.
• Agresti, A. (2013). Categorical Data Analysis (3rd ed.). Wiley.
• Bishop, Y. M. M., Fienberg, S. E., & Holland, P. W. (1975). Discrete Multivariate Analysis. MIT Press.