Probability Distributions in Machine Learning

Probability distributions form the foundation of statistical modeling and machine learning, enabling the representation and analysis of uncertainty in data. These distributions provide a mathematical framework to describe how data points are spread, which is crucial for making inferences, predictions, and decisions in various applications. This document will explore key probability distributions used in machine learning, their properties, and their applications.

We will cover the following distributions:

1. Bernoulli Distribution
2. Multinoulli Distribution
3. Gaussian Distribution
4. Various Continuous Distributions (Exponential, Beta, etc.)

Binary Outcomes - The Bernoulli Distribution

The Bernoulli distribution models binary outcomes, representing scenarios where there are only two possible outcomes: success (1) or failure (0). This distribution is essential in fields like medical diagnostics (e.g., presence or absence of a disease) and quality control (e.g., defective or non-defective products).

Mathematical Formulation

The Bernoulli distribution is parameterized by a single parameter $p$, which represents the probability of success: $P(X=1) = p$ $P(X=0) = 1 - p$ The probability mass function (PMF) is given by: $P(X=x) = p^x (1-p)^{1-x} \quad \text{for } x \in {0, 1}$

Properties

• Mean (Expected Value): $E[X] = p$
• Variance: $\text{Var}(X) = p(1 - p)$

Real-World Example

Consider a coin toss where the probability of landing heads (success) is $p$. If $p = 0.5$, the coin is fair; otherwise, it is biased.

Categorical Data - The Multinoulli Distribution

The Multinoulli (or Categorical) distribution generalizes the Bernoulli distribution to scenarios with more than two possible outcomes. It is useful in natural language processing (e.g., predicting the next word in a sentence) and recommendation systems (e.g., suggesting one of many products).

Mathematical Formulation

The Multinoulli distribution is parameterized by a vector $\mathbf{p} = (p_1, p_2, \ldots, p_k)$ where $p_i$ represents the probability of the $i$-th category and $\sum_{i=1}^k p_i = 1$. The probability mass function is: $P(X=i) = p_i \quad \text{for } i = 1, 2, \ldots, k$

Properties

• Mean (Expected Value): $E[X_i] = p_i$
• Variance: $\text{Var}(X_i) = p_i (1 - p_i)$
• Covariance: $\text{Cov}(X_i, X_j) = -p_i p_j \quad \text{for } i \ne j$

Real-World Example

In a survey with multiple choices, each choice corresponds to a category, and the probabilities $p_i$ represent the likelihood of each choice being selected.

The Ubiquitous Gaussian Distribution

The Gaussian distribution appears frequently because of analytical convenience, measurement models, latent-variable assumptions, and central-limit behavior. A standard CLT requires conditions such as independence or weak dependence and finite variance; it concerns normalized sums or averages, not the original observations themselves.

Mathematical Formulation

The Gaussian distribution is characterized by two parameters: the mean $\mu$ and the variance $\sigma^2$. Its probability density function (PDF) is: $f(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left( -\frac{(x - \mu)^2}{2\sigma^2} \right)$

Properties

• Mean (Expected Value): $E[X] = \mu$
• Variance: $\text{Var}(X) = \sigma^2$
• Symmetry: The Gaussian distribution is symmetric around its mean $\mu$.

Real-World Example

Gaussian return models are analytically convenient in finance, but empirical returns commonly show heavy tails, skewness, volatility clustering, and dependence. Normal models should be treated as approximations whose tail adequacy must be checked.

Continuous Distributions and Their Applications

Beyond the Gaussian distribution, other continuous distributions are vital in various applications. We will discuss the Exponential, Beta, and other distributions.

Exponential Distribution

The Exponential distribution models the time between events in a Poisson process. It is characterized by a single parameter $\lambda$ (rate parameter): $f(x; \lambda) = \lambda e^{-\lambda x} \quad \text{for } x \ge 0$

Properties

• Mean (Expected Value): $E[X] = \frac{1}{\lambda}$
• Variance: $\text{Var}(X) = \frac{1}{\lambda^2}$

Real-World Example

The exponential distribution is appropriate for waiting times under a constant-hazard model. Equipment that ages or wears out often violates that assumption, making Weibull, lognormal, or richer survival models more plausible.

Beta Distribution

The Beta distribution is useful for modeling proportions and probabilities. It is characterized by two shape parameters, $\alpha$ and $\beta$: $f(x; \alpha, \beta) = \frac{x^{\alpha-1} (1-x)^{\beta-1}}{B(\alpha, \beta)} \quad \text{for } 0 \le x \le 1$ where $B(\alpha, \beta)$ is the Beta function.

Properties

• Mean (Expected Value): $E[X] = \frac{\alpha}{\alpha + \beta}$
• Variance: $\text{Var}(X) = \frac{\alpha \beta}{(\alpha + \beta)^2 (\alpha + \beta + 1)}$

Real-World Example

The Beta distribution can model an unknown defect probability on $[0,1]$. The observed number of defective items in a batch is more naturally binomial conditional on that probability.

Understanding probability distributions is crucial for effectively modeling, analyzing, and making predictions based on data in machine learning. The Bernoulli, Multinoulli, Gaussian, Exponential, and Beta distributions each play unique roles in handling different types of data and scenarios. By mastering these distributions, one can harness the power of statistical modeling to address a wide range of real-world problems.

References

Casella, G., & Berger, R. L. (2002). Statistical Inference. Duxbury.

Murphy, K. P. (2012). Machine Learning: A Probabilistic Perspective. MIT Press.

Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.

Wasserman, L. (2004). All of Statistics: A Concise Course in Statistical Inference. Springer.

Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.

Distribution versus likelihood versus prior

The same named distribution can play different roles. For example:

• Bernoulli can be the likelihood for binary outcomes;
• Beta can be a prior for an unknown Bernoulli probability;
• Gaussian can be an observation model, latent-variable prior, or approximation;
• categorical distributions can define multiclass likelihoods.

The scientific interpretation therefore depends on what quantity is modeled, not only on the distribution name.

Probability models should match support

Support is often the first useful diagnostic. A Gaussian model assigns positive density to every real number. That is inappropriate for quantities that are physically restricted to be positive when negative values are not merely negligible but impossible. Likewise, a Beta distribution is bounded to

Support mismatch is a model error, not something a large dataset automatically fixes.

Dependencies matter

Writing

assumes independence. In time series, grouped observations, spatial data, and repeated measures, that factorization may be false. A correct marginal distribution for each observation does not imply a correct joint model.

Calibration

For probabilistic prediction, the quality of a distributional model should be evaluated through proper scoring rules and calibration, not only point accuracy. A classifier can have good accuracy while producing badly calibrated probabilities. A regression model can have a good RMSE while prediction intervals undercover. Probability modeling is useful precisely because it lets us evaluate the whole predictive distribution.