Wasserstein Distance Is Geometry, Not Just Another Divergence

Two probability distributions can be nearly identical in shape and still have disjoint support. Shift a narrow distribution by a tiny distance and every point moves only slightly, yet a density-ratio divergence may become undefined or infinite because one distribution assigns positive probability where the other assigns zero. Whether that should count as an enormous difference depends on the question.

Optimal transport answers a different question from information-theoretic divergence. Instead of comparing the amount of probability assigned to the same location, it asks how probability mass can be rearranged from one distribution into another and how much that rearrangement costs. A unit of mass moved one millimetre is cheaper than a unit moved one kilometre if the underlying metric says those distances are different.

That simple change introduces geometry into probability.

Wasserstein distance is therefore not merely another score to place beside Kullback-Leibler divergence, Jensen-Shannon divergence or total variation. It lives on a different idea of proximity. A probability distribution is treated as a configuration of mass over a metric space, and two distributions are close when there exists a low-cost transport plan moving one mass configuration into the other.

The resulting geometry is rich enough to define shortest paths between probability distributions, averages of distributions, gradient flows on spaces of measures and robust optimization neighbourhoods. It is also computationally demanding and statistically delicate in high dimension.

The right way to understand Wasserstein distance is to begin with the transport problem itself.

From moving piles of mass to a linear programme

Consider two discrete probability distributions,

and

where

and

Suppose moving one unit of probability mass from location $x_i$ to location $y_j$ costs

A transport plan is a nonnegative matrix

where

is the amount of mass moved from $x_i$ to $y_j$.

The plan must exhaust the source mass,

and produce the destination mass,

The optimal transport problem is

subject to those marginal constraints.

This is a linear programme.

The important object is not only the minimum cost. The optimizer

says how much mass should move between every source and destination.

Take

and

With one-dimensional cost

the obvious optimal plan moves the mass at zero to one and the mass at two to three. The total cost is

Crossing the assignments would move zero to three and two to one, costing

The optimal plan is therefore shaped by the geometry of the support.

If the labels $0,1,2,3$ had no meaningful metric interpretation, the transport distance would be meaningless. Optimal transport is only as scientifically sensible as the ground cost.

This dependence on the underlying geometry is a feature, not a bug.

Monge's map and Kantorovich's relaxation

The original transport formulation associated with Monge asks for a deterministic map

that pushes one distribution into another,

while minimizing

The pushforward notation

means that if

then

A deterministic map can fail to exist. One source location containing positive mass may need to split that mass across several destinations.

Kantorovich's formulation allows exactly this.

Instead of a map, choose a joint probability distribution

on source-destination pairs

whose first marginal is

and second marginal is

The collection of such couplings is written

The Kantorovich problem is

The coupling can split mass.

This relaxation is one of the decisive ideas in modern optimal transport. It converts a difficult nonlinear map problem into a convex optimization problem over probability measures.

For suitable costs and regularity conditions, the optimal Kantorovich coupling can still be concentrated on a deterministic transport map. But the theory no longer requires such a map to exist in advance.

Wasserstein distance is transport cost with a metric power

Let the sample space carry a metric

For

define the transport cost

The $p$-Wasserstein distance is

When both distributions have finite $p$th moments, this defines a metric on the corresponding Wasserstein space.

The word metric matters. We have

symmetry,

identity of indiscernibles,

and the triangle inequality.

The geometry of the sample space is inherited by the space of probability measures.

If the ground metric is Euclidean,

then $W_2$ penalizes squared Euclidean displacement before taking the square root.

If the ground metric encodes graph distance, geodesic distance, transportation cost or another scientifically meaningful geometry, the Wasserstein metric changes accordingly.

There is no universal Wasserstein distance detached from a choice of ground geometry.

Support mismatch reveals the difference from KL divergence immediately

Take the simplest possible distributions,

and

Every unit of mass must move from zero to $a$, so

for every

If

is small, the distributions are close in Wasserstein distance.

Now consider Kullback-Leibler divergence.

For

the point masses have disjoint support. Neither distribution is absolutely continuous with respect to the other. Therefore,

and

KL sees an impossible density ratio.

Wasserstein sees a finite displacement.

Neither answer is "more correct" without context.

If the question is information loss under one probability model used in place of another, support mismatch is catastrophic and KL's infinity is meaningful.

If the question is how far a probability distribution moved in physical space, the magnitude of $a$ matters and Wasserstein geometry is natural.

This is why transport distance became attractive in generative modelling. Two empirical distributions supported on nearby low-dimensional sets can have almost no overlap. Density-ratio divergences can saturate or become undefined, while transport cost changes continuously as the supports approach one another.

The same property is useful in distribution shift. If a sensor distribution translates by one degree, one metre or one unit of concentration, a geometric metric can express the size of that change directly.

In one dimension, optimal transport is monotone matching

One-dimensional optimal transport has an exceptionally clean form.

Let

and

be the CDFs of

and

Define generalized quantile functions

and similarly for

Then

The optimal coupling matches equal quantile levels.

The smallest source observation moves to the smallest destination observation, the next smallest to the next smallest, and so on.

This is a form of monotone rearrangement.

The result gives both intuition and computation. For empirical distributions with equal sample sizes, sort both samples and average pairwise powered distances between corresponding order statistics.

Suppose

and

Their quantile functions are

Therefore,

and

The entire distribution has translated rigidly by $a$, and Wasserstein distance returns exactly the translation magnitude.

This is what geometric faithfulness looks like.

Gaussian distributions have a closed form under W2

For multivariate Gaussian distributions,

and

the squared 2-Wasserstein distance has the closed form

The first term compares means geometrically.

The second compares covariance structure.

In one dimension,

and the formula reduces to

Consider

and

Then

so

The mean shift and standard-deviation shift contribute through Euclidean geometry in the

plane.

KL divergence behaves differently.

For univariate normals,

For the same pair,

while reversing the order gives approximately

The numerical values should not be compared directly to

because the objects have different units and meanings.

The useful contrast is structural.

Wasserstein is symmetric and geometric.

KL is asymmetric and density-relative.

W1 has a dual form that turns transport into a function optimization problem

The primal transport problem optimizes over couplings.

For

Kantorovich-Rubinstein duality gives a remarkably different representation:

The supremum is over all 1-Lipschitz functions,

This dual form says that

is the largest difference in expectation detectable by a function that cannot change faster than the underlying geometry permits.

This connects optimal transport to integral probability metrics.

The coupling formulation asks how mass moves.

The dual formulation asks which smooth-enough critic best distinguishes the distributions.

The two answers are equal.

This duality is one reason Wasserstein ideas became influential in generative modelling. Instead of solving a full transport plan in high dimension, one can try to optimize a parameterized critic subject to an approximate Lipschitz constraint.

The computational implementation can be imperfect while the mathematical dual remains exact.

The distinction should not be blurred.

Mixture interpolation and transport interpolation are fundamentally different

Suppose two point masses are

and

A conventional convex mixture is

At

the distribution is

Half the mass remains at the starting point and half has appeared at the destination.

Nothing occupies the space between them.

Wasserstein geometry produces a different path.

The optimal transport map sends

Displacement interpolation moves the point continuously,

Therefore,

At

the distribution is

The mass has moved halfway.

This is one of the most important geometric differences between Wasserstein space and ordinary linear mixture space.

A mixture interpolates probability weights.

A Wasserstein geodesic transports locations.

For distributions representing shapes, images, spatial densities or physical populations, displacement interpolation can preserve structure in a way that linear mixing does not.

For other problems, mixture interpolation may be exactly the correct operation.

Again, geometry should follow meaning.

Wasserstein barycentres average distributions through transport

A scalar average minimizes squared Euclidean distance:

A Wasserstein barycentre generalizes this idea to probability distributions:

with

This defines an average in Wasserstein geometry.

For translated point masses,

the barycentre reduces to the ordinary weighted mean point mass,

For richer distributions, the barycentre can align and transport structures rather than simply overlaying them.

Consider two narrow unimodal distributions centred at different locations.

A density mixture can become bimodal.

A Wasserstein barycentre can remain unimodal at an intermediate location.

Whether that is desirable depends on what "average distribution" is intended to mean.

If two populations genuinely coexist, mixture is natural.

If one object has been translated or deformed relative to another, transport barycentres can be much more faithful.

The word average is therefore geometry-dependent.

Wasserstein space supports gradient-flow interpretations

The space of probability measures equipped with

has enough geometric structure to interpret certain evolution equations as gradient flows.

One celebrated example is the heat equation,

In the Jordan-Kinderlehrer-Otto formulation, diffusion can be interpreted as steepest descent of entropy in Wasserstein space.

A time-discretized step can be written schematically as

where

is an energy functional.

The first term penalizes moving too far from the previous distribution.

The second lowers the energy.

This variational structure connects optimal transport to PDEs, diffusion, Fokker-Planck equations and statistical mechanics.

It also reveals why Wasserstein geometry is more than a distance formula.

Probability distributions form a nonlinear metric space in which dynamics can be described geometrically.

One should not overextend the analogy with finite-dimensional Riemannian geometry, but it is mathematically productive and has generated a large theory.

Exact optimal transport can be computationally expensive

For discrete distributions with

source points and

destination points, the transport plan contains

variables.

Large dense problems can therefore become expensive.

Entropic regularization changes the optimization problem to

The entropy term encourages diffuse transport plans.

The regularized optimum can be computed efficiently through Sinkhorn iterations because the solution has a multiplicative scaling structure.

If

the regularized coupling can be written in the form

with scaling vectors

and

adjusted until the marginal constraints are satisfied.

This makes large transport problems dramatically more tractable.

It also changes the problem.

As

increases, the solution becomes smoother and more diffuse. The regularized transport cost is biased relative to exact optimal transport.

The entropy term is therefore both a computational device and a modelling perturbation.

Sinkhorn divergences correct part of the entropic self-bias by combining regularized cross-costs and self-costs, but they still define a family of geometries depending on the regularization scale.

A numerical approximation parameter has become part of the statistical object.

That should be acknowledged when results are sensitive to

Statistical estimation of Wasserstein distance can be hard in high dimension

Optimal transport has attractive geometric behaviour and an important statistical limitation.

Suppose

is the empirical distribution from

iid samples of a distribution

The empirical Wasserstein distance

can converge slowly as dimension increases.

Under common regularity conditions, prototypical rates in sufficiently high dimension scale like

rather than the dimension-free

rate associated with many scalar averages.

The exact rate depends on

dimension, moment assumptions and regularity, so

should be read as the characteristic curse-of-dimensionality regime rather than a universal formula.

The practical point is clear.

Wasserstein distance can be geometrically meaningful and statistically expensive.

This becomes severe when estimating high-dimensional distributions from finite samples. Two empirical point clouds can have substantial Wasserstein distance partly because finite samples do not cover high-dimensional space densely.

Regularized transport, sliced Wasserstein distances, projection-based methods and structural assumptions can reduce computation or statistical difficulty.

A sliced Wasserstein distance projects distributions onto one-dimensional directions,

computes one-dimensional Wasserstein distances there, and averages over directions.

The one-dimensional quantile formula makes those projected distances cheap.

Projection simplifies the geometry and can discard multivariate structure.

Again there is a trade-off.

The ground metric is part of the scientific model

Suppose two categorical outcomes are encoded as integers

Applying Euclidean Wasserstein distance implies that moving probability from category one to category two costs one unit, while moving from one to four costs three.

That assumption may be meaningful for ordered severity levels.

It is nonsensical for arbitrary labels such as blood type or country code.

The same issue appears in high-dimensional continuous data.

Euclidean pixel distance between images may treat a one-pixel translation as a large change despite strong perceptual similarity.

Euclidean distance between raw sensor vectors can overemphasize high-variance dimensions.

A graph metric may be more appropriate for network locations.

Geodesic distance may be required on a manifold.

A learned metric can be useful when domain similarity is not captured by raw coordinates, but then the transport result inherits the assumptions and errors of the learned representation.

Optimal transport never removes the need to define what distance means.

It amplifies that modelling choice by building the entire probability metric from it.

Unbalanced transport relaxes conservation when total mass differs

Classical optimal transport assumes source and target contain the same total mass.

Probability distributions satisfy this automatically because both integrate to one.

Many physical applications involve measures whose total mass differs. Cell populations grow, image intensity changes, demand volumes differ, or material can be created and destroyed.

Unbalanced optimal transport relaxes exact marginal conservation and penalizes mass creation or destruction.

Schematically, one solves an objective combining transport cost with divergence penalties measuring how far the transported marginals depart from the original measures.

This distinction is scientifically important.

If total mass difference is meaningful signal, normalizing both objects to probability distributions can erase it.

If only shape matters, normalization may be appropriate.

The choice between balanced and unbalanced transport should therefore follow the process being represented.

Wasserstein neighbourhoods also define uncertainty sets

Optimal transport connects naturally to optimization under uncertainty.

Suppose

is the empirical distribution of observed uncertain quantities.

A Wasserstein ambiguity set is

Distributionally robust optimization can then solve

The radius

controls how far the adversarial distribution may move from the empirical law in transport geometry.

This differs from moment-based ambiguity, which allows any distribution sharing specified means or covariances, and from divergence balls based on density ratios.

A Wasserstein ball says that probability mass may be relocated, but large relocations are expensive according to the ground metric.

This can be attractive when nearby perturbations of observed scenarios are more plausible than arbitrary reweighting.

It also inherits the same modelling choice: the ground metric defines what counts as a nearby distributional perturbation.

Robustness is therefore geometric.

Optimal transport is useful when location matters

The simplest support-mismatch example contains the central lesson.

For

and

KL divergence is infinite for every nonzero

Wasserstein distance is exactly

The two answers differ because they ask different questions.

KL asks whether one distribution assigns probability where the other does and how costly the density-ratio mismatch is in information terms.

Wasserstein asks how much probability mass must move through the sample space.

When the geometry of locations matters, that movement can be the scientifically relevant difference.

The one-dimensional quantile representation makes the idea exact.

The Gaussian formula shows how location and covariance geometry combine.

Displacement interpolation turns probability distributions into points connected by transport geodesics.

Barycentres define geometric averages.

Kantorovich-Rubinstein duality connects transport to Lipschitz test functions.

Entropic regularization makes large problems computationally manageable at the cost of changing the objective.

High-dimensional statistics remind us that elegant geometry does not eliminate finite-sample difficulty.

Optimal transport is therefore powerful precisely because it commits to a notion of geometry.

It should be used when that geometry means something.

References

Ambrosio, L., Gigli, N., & Savaré, G. (2008). Gradient Flows in Metric Spaces and in the Space of Probability Measures (2nd ed.). Birkhäuser.

Cuturi, M. (2013). Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems 26.

Kantorovich, L. V. (1942). On the translocation of masses. Doklady Akademii Nauk SSSR, 37, 227–229.

Peyré, G., & Cuturi, M. (2019). Computational optimal transport. Foundations and Trends in Machine Learning, 11(5-6), 355–607. https://doi.org/10.1561/2200000073

Santambrogio, F. (2015). Optimal Transport for Applied Mathematicians. Birkhäuser.

Sinkhorn, R. (1964). A relationship between arbitrary positive matrices and doubly stochastic matrices. The Annals of Mathematical Statistics, 35(2), 876–879.

Villani, C. (2003). Topics in Optimal Transportation. American Mathematical Society.

Villani, C. (2009). Optimal Transport: Old and New. Springer.