Maximum Likelihood Estimation: What It Guarantees and What It Does Not

Maximum likelihood estimation is one of the central ideas in statistical modeling. Its definition is simple. Given observed data $x$ and a model indexed by parameter $\theta$, choose the parameter value that makes the observed data most likely under that model. The difficulty is everything hidden inside the phrase under that model. MLE does not tell us whether the model is scientifically appropriate.

It tells us which parameter value fits best within the family we chose.

Likelihood is a function of the parameter

Suppose

are modeled as independent observations with density or mass function

After observing

the likelihood is

The observations are fixed inside this function. The parameter varies. This is why a likelihood is not a probability distribution over $\theta$. Without a prior, it does not integrate to one over parameter space and need not be interpreted probabilistically as

Log-likelihood

Products of many small probabilities or densities are numerically inconvenient. Because the logarithm is monotone,

where

For independent observations,

This turns products into sums and usually simplifies differentiation.

Bernoulli example

Let

independently. If there are $k$ successes among $n$ observations,

The log-likelihood is

Differentiating,

Setting the score to zero gives

So the sample proportion is the Bernoulli MLE. The result is familiar, but the derivation shows the estimation principle explicitly.

Normal example

Suppose

The log-likelihood is

Maximizing over $\mu$ gives

Maximizing over $\sigma^2$ gives

This is not the unbiased sample-variance estimator, whose denominator is $n-1$. MLE and unbiasedness are different criteria.

The score and information

The score is

Under regularity conditions and the correctly specified model,

The Fisher information can be written as

or, under additional regularity,

Information measures local curvature and parameter sensitivity. Flat likelihood directions correspond to weak identification and large uncertainty.

Consistency is not automatic

Textbook summaries often say that MLE is consistent. The correct statement is conditional. Consistency requires assumptions such as:

• the data-generating distribution belongs to, or is appropriately represented by, the model;
• the parameter is identifiable;
• the likelihood obeys suitable continuity and compactness or coercivity conditions;
• observations satisfy the dependence assumptions required by the theorem;
• the criterion converges uniformly enough to its population target.

When those conditions fail, MLE can be inconsistent, non-unique, or undefined.

Identifiability

A model is identifiable if different parameter values imply different observable distributions. Formally,

If two different parameter vectors generate exactly the same distribution, the data cannot distinguish them. No optimizer can solve an identification problem. This is a property of the model, not of the numerical algorithm.

Asymptotic normality

Under standard regularity conditions,

where $I_1$ denotes information per observation. Equivalently,

This approximation motivates Wald standard errors and confidence intervals. It can fail near parameter boundaries, under weak identification, with mixture models, under nonregular likelihoods, or in small samples.

Efficiency also needs qualification

MLE is often described as “efficient.” Under the regular correctly specified parametric model, the MLE is asymptotically efficient in the usual Cramér-Rao sense. That is not the statement that the MLE has minimum variance among all unbiased estimators in every finite sample. Nor does it imply that an MLE from a wrong model is optimal for the scientific target.

The word asymptotic matters.

Misspecification

Suppose the true distribution is $g(x)$ but we fit a family

The MLE can still converge. But it generally converges to the parameter value

Equivalently, this is the member of the model family minimizing Kullback-Leibler divergence from the truth, when the relevant quantities exist. The parameter $\theta^\ast$ is a pseudo-true parameter. That can be useful. It is not evidence that the fitted model is literally true. Under misspecification, the usual inverse-Fisher covariance formula also needs replacement by a sandwich form.

Logistic regression

For binary outcomes,

with

The log-likelihood is

There is no general closed-form solution for $\hat\beta$. Numerical optimization is used. This is a genuine example of a common machine-learning loss arising directly as a negative log-likelihood.

Not every machine-learning algorithm is MLE

The previous version of this article incorrectly grouped support vector machines, decision trees, and random forests as if they were likelihood-based MLE procedures. They are not, in their standard forms. A soft-margin SVM minimizes hinge loss plus a regularization term. A decision tree recursively optimizes split criteria. A random forest averages randomized trees.

These can be studied statistically, but they are not automatically maximum-likelihood estimators. Neural networks are more nuanced. A network trained with cross-entropy for a categorical outcome can be interpreted as maximizing a conditional likelihood. A network trained under another objective may not have that interpretation. The loss function determines the statistical connection.

Optimization is not inference

Finding

is an optimization problem. Inference requires additional work. Questions include:

• Is the optimum unique?
• Is it global or local?
• Is the parameter identifiable?
• What is the sampling distribution?
• Are standard errors valid?
• Is the model misspecified?
• Does the parameter answer the scientific question?

An optimizer returning “success” does not answer any of those.

Local maxima and numerical issues

For a simple concave likelihood such as ordinary logistic regression without separation, optimization is well behaved. Other likelihoods can contain:

• multiple local maxima;
• flat ridges;
• singularities;
• boundary optima;
• unbounded likelihoods.

Mixture models are a classic example. Numerical diagnostics are part of statistical modeling. Convergence flags, gradients, Hessians, starting values, and repeated initializations can matter.

Reproducible Python examples

The closed-form Bernoulli and normal MLEs can be implemented without pretending an optimizer is necessary.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
from __future__ import annotations

import numpy as np
from numpy.typing import NDArray

FloatArray = NDArray[np.float64]

def bernoulli_mle(
    observations: NDArray[np.int64],
) -> float:
    if observations.ndim != 1:
        raise ValueError(
            "observations must be one-dimensional"
        )

if not np.all(
        (observations == 0)
        | (observations == 1)
    ):
        raise ValueError(
            "Bernoulli observations must be 0 or 1"
        )

return float(observations.mean())

def normal_mle(
    observations: FloatArray,
) -> tuple[float, float]:
    if observations.ndim != 1:
        raise ValueError(
            "observations must be one-dimensional"
        )

if observations.size == 0:
        raise ValueError(
            "observations cannot be empty"
        )

mean_mle: float = float(
        observations.mean()
    )

variance_mle: float = float(
        np.mean(
            (observations - mean_mle) ** 2
        )
    )

return mean_mle, variance_mle

rng = np.random.default_rng(2026)

binary = rng.binomial(
    n=1,
    p=0.7,
    size=1_000,
)

normal = rng.normal(
    loc=5.0,
    scale=2.0,
    size=1_000,
)

print(bernoulli_mle(binary))
print(normal_mle(normal))

The normal variance uses denominator $n$ because it is the MLE. That is intentional.

Likelihood ratios

Likelihood also supports model comparison. For nested models with maximized log-likelihoods

and

the likelihood-ratio statistic is

Under regular conditions, Wilks' theorem gives an asymptotic chi-square distribution with degrees of freedom equal to the difference in parameter dimension. Again, “under regular conditions” matters. Boundary parameters and non-identifiable models can invalidate the ordinary chi-square reference distribution.

Bayesian inference uses the same likelihood differently

Bayesian inference combines the likelihood with a prior:

The maximum a posteriori estimator solves

This often resembles penalized likelihood. But MLE and Bayesian inference answer different probability questions. The likelihood is common to both. The inferential framework is not.

Conclusion

MLE is a disciplined way to estimate parameters inside a probabilistic model. Its strongest properties are conditional:

The maximized likelihood cannot validate the model that generated it. A good likelihood analysis therefore combines estimation with diagnostics, uncertainty, model comparison, and explicit discussion of misspecification.

References

• Fisher, R. A. (1922). On the mathematical foundations of theoretical statistics. Philosophical Transactions of the Royal Society A, 222, 309–368.
• Myung, I. J. (2003). Tutorial on maximum likelihood estimation. Journal of Mathematical Psychology, 47(1), 90–100.
• Casella, G., & Berger, R. L. (2002). Statistical Inference (2nd ed.). Duxbury.
• White, H. (1982). Maximum likelihood estimation of misspecified models. Econometrica, 50(1), 1–25.
• van der Vaart, A. W. (1998). Asymptotic Statistics. Cambridge University Press.