Probability Distributions as Statistical Models

A probability distribution is not a decorative description of a histogram. It is a mathematical model for a random quantity. That distinction matters because the model determines which events are possible, how probability is allocated among them, how uncertainty is summarized, and which likelihood or inferential procedure follows. For a random variable $X$, a distribution can be represented through a probability mass function, density, cumulative distribution function, survival function, characteristic function, or another equivalent object.

The right representation depends on the problem.

A distribution is more than its mean and variance

Two distributions can have the same mean and variance and still differ substantially in skewness, tail behavior, discreteness, or support. For example, a normal random variable has support on the entire real line,

while an exponential random variable is restricted to

A binomial random variable can take only finitely many integer values,

These structural constraints are often more important than whether two distributions have similar first and second moments. A model should therefore begin with the mechanism and support of the random quantity, not with the visual resemblance of a histogram to a named curve.

Parameters index a family of distributions

A parametric family is a collection

indexed by a finite-dimensional parameter $\theta$. For a normal distribution,

For a Bernoulli model,

The parameter does not need to correspond directly to a simple descriptive statistic such as the median or skewness. A parameter is whatever quantity indexes the model family. This is why phrases such as “location parameter,” “scale parameter,” and “shape parameter” are useful, but they are descriptions of parameter roles rather than universal categories into which every parameter must fit.

The normal distribution

A normal random variable has density

with

Its mean and variance are

The normal family is symmetric and light-tailed. The familiar 68-95-99.7 rule follows directly from its quantiles, not from a generic property of symmetric data.

Why the normal distribution appears so often

One reason is the Central Limit Theorem. For independent and identically distributed random variables with finite mean $\mu$ and finite, positive variance $\sigma^2$,

The theorem concerns the standardized sum or mean. It does not say that the original observations become normal. It also does not guarantee that the approximation is good for every finite $n$. Heavy skewness, large kurtosis, dependence, or infinite variance can make convergence slow or invalidate the standard theorem's conditions. So “large sample” is not a universal number such as 30.

Approximation quality depends on the underlying distribution and statistic.

The binomial distribution

Suppose

where the $B_i$ are independent Bernoulli trials with common success probability $p$. Then

and

The mean and variance are

and

The assumptions are substantive. A count is not binomial merely because it lies between 0 and $n$. The trials must have a defensible common success probability and independence structure, or the model must be interpreted as an approximation to a more complicated mechanism. Overdispersion relative to the binomial variance can indicate heterogeneity, dependence, or misspecification.

The Poisson distribution

A Poisson random variable has probability mass function

Its defining mean-variance relationship is

In a homogeneous Poisson process, the number of events in an interval of length $t$ satisfies

with independent increments. The Poisson distribution itself is not memoryless. The memoryless property belongs to the exponential waiting-time distribution associated with a homogeneous Poisson process. Confusing those two objects is common because they are mathematically linked, but the distinction is exact.

The exponential distribution

If events arrive according to a homogeneous Poisson process with rate $\lambda$, the waiting time $T$ to the next event has density

Its survival function is

The memoryless property is

Equivalently, its hazard is constant:

That constant-hazard assumption is strong. Many physical components age, fatigue, or undergo changing environmental stress, making Weibull, lognormal, Gamma, or more flexible survival models preferable.

Distribution choice is a modeling decision

A distribution is justified by a combination of:

• support;
• data-generating mechanism;
• dependence structure;
• mean-variance relationship;
• tail behavior;
• scientific constraints;
• and the inferential target.

A count outcome might suggest Poisson, negative binomial, zero-inflated, hurdle, or binomial models depending on the mechanism. A positive continuous duration might suggest exponential, Weibull, Gamma, lognormal, or a semiparametric survival model. The histogram alone cannot decide among them.

Parametric methods do not require the raw data to be normal

A frequent mistake is to describe methods such as regression, ANOVA, or t-tests as requiring “normally distributed data.” The assumptions are more specific. In a classical linear model,

normal-theory inference concerns the conditional error distribution

not the pooled marginal distribution of every observed response. For a paired t-test, the relevant object is the distribution of paired differences. For a one-sample t-test, exact finite-sample theory uses normality of the observations, but large-sample inference for the mean can remain useful under much broader conditions. The assumption belongs to a model and estimand, not to the spreadsheet as a whole.

Parametric does not automatically mean more powerful

A correctly specified parametric model can exploit structure efficiently. That can yield greater power than a rank-based or distribution-free alternative. But there is no universal ordering. A misspecified parametric test can lose validity or efficiency. A rank test may have greater power against some alternatives. And two tests may target different estimands entirely.

For example, a two-sample t-test targets a mean contrast. The Mann-Whitney procedure is based on ranks and does not generally become “the same test without normality.” Method choice must preserve the scientific question.

Likelihood makes the model operational

For independent observations

from density or mass function $p(x\mid\theta)$, the likelihood is

Once a distributional model is specified, it determines the likelihood. That connects the probability model to estimation, likelihood-ratio tests, information criteria, and Bayesian inference. This is one reason distribution choice matters so much: it is not merely descriptive language. It changes the mathematics of inference.

Diagnostics ask how the model fails

No fitted distribution should be treated as true because a goodness-of-fit test failed to reject it. Useful diagnostics include:

• Q-Q plots for quantile structure;
• residual plots for conditional models;
• empirical versus fitted CDFs;
• tail plots for extreme-value behavior;
• checks of mean-variance relationships;
• simulation from the fitted model;
• posterior predictive checks in Bayesian models.

The question should be

Is the model adequate for the claim I intend to make?

not

Did a p-value certify the distribution?

Finance is a warning against convenient distributions

Financial returns are often approximated by normal models for analytical convenience. Empirical returns commonly exhibit heavier tails and volatility clustering. A lognormal model for prices arises when log prices or log returns follow a specified Gaussian process. It is not justified merely because asset prices cannot be negative. That positivity is a support constraint.

Many positive-valued distributions share it. Risk calculations are especially sensitive to tail assumptions, so distributional convenience should not be confused with empirical adequacy.

Survival models encode hazard assumptions

An exponential survival model assumes a constant hazard. A Weibull model permits monotone increasing or decreasing hazard depending on its shape parameter. A Cox model leaves the baseline hazard unspecified while imposing proportional covariate effects. These are different scientific assumptions. Choosing among them is not simply a matter of which curve visually fits the observed survival times.

Conclusion

Probability distributions are models for random quantities. Their support, parameters, dependence assumptions, tail behavior, and mean-variance structure determine what they say about the data-generating process. The practical sequence is

That is more useful than treating a distribution as a descriptive label attached after looking at a histogram.

References

• Casella, G., & Berger, R. L. (2002). Statistical Inference (2nd ed.). Duxbury.
• Ross, S. M. (2019). A First Course in Probability (10th ed.). Pearson.
• Lehmann, E. L., & Romano, J. P. (2005). Testing Statistical Hypotheses (3rd ed.). Springer.
• McCullagh, P., & Nelder, J. A. (1989). Generalized Linear Models (2nd ed.). Chapman & Hall.