Tail-Risk Metrics

risk

Tail-risk metrics: value at risk, expected shortfall and Monte Carlo estimation.

These are the quantities practitioners ask for, rather than the distributions underneath them. Two points recur and are easy to get wrong:

Expected shortfall is infinite when the mean does not exist. For a Pareto tail with alpha <= 1 there is no finite answer, and returning a large number instead of inf would be worse than useless: it would look like a result. Every function here checks and reports inf.

A Monte Carlo estimate without a standard error is not usable. The whole point of these metrics is the far tail, where few samples land, so :func:monte_carlo_tail_risk always reports the uncertainty alongside the estimate.

expected_shortfall

expected_shortfall(dist, level, method='auto', nodes=20000)

Expected shortfall: the mean loss given that value at risk is exceeded.

Also called conditional value at risk. Where value at risk reports a threshold, this reports the average of what lies beyond it, which is the question that matters when the tail is heavy.

Parameters:

Name	Type	Description	Default
dist	Any	A distribution with a ppf method.	required
level	float	Confidence level in the open interval (0, 1).	required
method	str	auto uses the closed form where this module has one and falls back to quadrature; analytic raises if there is no closed form; numeric always integrates.	'auto'
nodes	int	Quadrature nodes for the numeric path.	20000

Returns:

Type	Description
float	The expected shortfall, or inf when the distribution has no finite
float	mean.

Raises:

Type	Description
ValueError	If level is outside (0, 1), method is unknown, or analytic was requested for a family without a closed form here.

Examples:

>>> from heavytails import Cauchy, Pareto
>>> round(expected_shortfall(Pareto(alpha=2.0, xm=1.0), 0.99), 4)
20.0
>>> expected_shortfall(Pareto(alpha=0.5, xm=1.0), 0.99)
inf
>>> expected_shortfall(Cauchy(), 0.99)
inf

Source code in heavytails/risk.py

def expected_shortfall(
    dist: Any, level: float, method: str = "auto", nodes: int = 20000
) -> float:
    """
    Expected shortfall: the mean loss given that value at risk is exceeded.

    Also called conditional value at risk. Where value at risk reports a
    threshold, this reports the average of what lies beyond it, which is the
    question that matters when the tail is heavy.

    Args:
        dist: A distribution with a ``ppf`` method.
        level: Confidence level in the open interval (0, 1).
        method: ``auto`` uses the closed form where this module has one and
            falls back to quadrature; ``analytic`` raises if there is no closed
            form; ``numeric`` always integrates.
        nodes: Quadrature nodes for the numeric path.

    Returns:
        The expected shortfall, or ``inf`` when the distribution has no finite
        mean.

    Raises:
        ValueError: If ``level`` is outside (0, 1), ``method`` is unknown, or
            ``analytic`` was requested for a family without a closed form here.

    Examples:
        >>> from heavytails import Cauchy, Pareto
        >>> round(expected_shortfall(Pareto(alpha=2.0, xm=1.0), 0.99), 4)
        20.0
        >>> expected_shortfall(Pareto(alpha=0.5, xm=1.0), 0.99)
        inf
        >>> expected_shortfall(Cauchy(), 0.99)
        inf
    """
    if not (0.0 < level < 1.0):
        raise ValueError("level must be in (0,1).")
    if method not in {"auto", "analytic", "numeric"}:
        raise ValueError(
            f"Unknown method {method!r}. Available: auto, analytic, numeric"
        )

    # A conditional mean cannot be finite when the mean is not.
    if not mean_exists(dist):
        return math.inf

    if method in {"auto", "analytic"}:
        closed_form = _analytic_expected_shortfall(dist, level)
        if closed_form is not None:
            return float(closed_form)
        if method == "analytic":
            raise ValueError(
                f"No closed-form expected shortfall for {type(dist).__name__} "
                "in this module; use method='auto' or 'numeric'."
            )

    return _numeric_expected_shortfall(dist, level, nodes)

mean_exists

mean_exists(dist)

Report whether a distribution has a finite mean.

Expected shortfall is a conditional mean, so it is infinite exactly when this is False. The check is by family, since it depends on the parameters: a Pareto tail has a finite mean only for alpha > 1.

Parameters:

Name	Type	Description	Default
dist	Any	A distribution instance from this package.	required

Returns:

Type	Description
bool	True if the mean is finite. Unknown families are assumed to have one,
bool	which is the less surprising default; the numeric path will produce a
bool	very large value rather than a wrong finite one if that is wrong.

Examples:

>>> from heavytails import Cauchy, Pareto
>>> mean_exists(Pareto(alpha=2.0, xm=1.0))
True
>>> mean_exists(Pareto(alpha=0.5, xm=1.0))
False
>>> mean_exists(Cauchy())
False

Source code in heavytails/risk.py

def mean_exists(dist: Any) -> bool:
    """Report whether a distribution has a finite mean.

    Expected shortfall is a conditional mean, so it is infinite exactly when
    this is False. The check is by family, since it depends on the parameters:
    a Pareto tail has a finite mean only for ``alpha > 1``.

    Args:
        dist: A distribution instance from this package.

    Returns:
        True if the mean is finite. Unknown families are assumed to have one,
        which is the less surprising default; the numeric path will produce a
        very large value rather than a wrong finite one if that is wrong.

    Examples:
        >>> from heavytails import Cauchy, Pareto
        >>> mean_exists(Pareto(alpha=2.0, xm=1.0))
        True
        >>> mean_exists(Pareto(alpha=0.5, xm=1.0))
        False
        >>> mean_exists(Cauchy())
        False
    """
    name = type(dist).__name__
    if name == "Cauchy":
        return False
    if name == "Pareto":
        return bool(dist.alpha > 1.0)
    if name == "StudentT":
        return bool(dist.nu > 1.0)
    if name == "Frechet":
        return bool(dist.alpha > 1.0)
    if name in {"GeneralizedPareto", "GEV_Frechet"}:
        return bool(dist.xi < 1.0)
    if name == "BurrXII":
        return bool(dist.c * dist.k > 1.0)
    if name == "LogLogistic":
        return bool(dist.kappa > 1.0)
    if name == "InverseGamma":
        return bool(dist.alpha > 1.0)
    if name == "BetaPrime":
        return bool(dist.b > 1.0)
    # LogNormal and Weibull always have a finite mean; anything unrecognised is
    # assumed to as well.
    return True

monte_carlo_tail_risk

monte_carlo_tail_risk(
    dist, level, n_samples=100000, seed=None
)

Estimate value at risk and expected shortfall by simulation, with errors.

An estimate of a tail quantity without a standard error is not usable: the whole point is the region where few samples land, so the uncertainty is large and varies with the level. This always reports it.

The standard error of the expected shortfall is the standard error of the mean of the exceedances. The standard error of the value at risk uses the asymptotic result for a sample quantile, sqrt(p(1-p)/n) / f(VaR), evaluated with the density where the distribution provides one.

Parameters:

Name	Type	Description	Default
dist	Any	A distribution with ppf, rvs and ideally pdf.	required
level	float	Confidence level in the open interval (0, 1).	required
n_samples	int	Number of variates to draw.	100000
seed	int | None	Seed, for a reproducible estimate.	None

Returns:

Type	Description
dict[str, Any]	Dictionary with value_at_risk, expected_shortfall, their
dict[str, Any]	standard errors, n_exceedances, n_samples and level. The
dict[str, Any]	expected shortfall is inf when the distribution has no finite mean.

Raises:

Type	Description
ValueError	If level is outside (0, 1), n_samples is too small, or the level leaves fewer than two exceedances to average.

Examples:

>>> from heavytails import Pareto
>>> result = monte_carlo_tail_risk(
...     Pareto(alpha=2.0, xm=1.0), 0.99, n_samples=20000, seed=1
... )
>>> result["n_exceedances"]
200

Source code in heavytails/risk.py

def monte_carlo_tail_risk(
    dist: Any,
    level: float,
    n_samples: int = 100000,
    seed: int | None = None,
) -> dict[str, Any]:
    """
    Estimate value at risk and expected shortfall by simulation, with errors.

    An estimate of a tail quantity without a standard error is not usable: the
    whole point is the region where few samples land, so the uncertainty is
    large and varies with the level. This always reports it.

    The standard error of the expected shortfall is the standard error of the
    mean of the exceedances. The standard error of the value at risk uses the
    asymptotic result for a sample quantile,
    ``sqrt(p(1-p)/n) / f(VaR)``, evaluated with the density where the
    distribution provides one.

    Args:
        dist: A distribution with ``ppf``, ``rvs`` and ideally ``pdf``.
        level: Confidence level in the open interval (0, 1).
        n_samples: Number of variates to draw.
        seed: Seed, for a reproducible estimate.

    Returns:
        Dictionary with ``value_at_risk``, ``expected_shortfall``, their
        standard errors, ``n_exceedances``, ``n_samples`` and ``level``. The
        expected shortfall is ``inf`` when the distribution has no finite mean.

    Raises:
        ValueError: If ``level`` is outside (0, 1), ``n_samples`` is too small,
            or the level leaves fewer than two exceedances to average.

    Examples:
        >>> from heavytails import Pareto
        >>> result = monte_carlo_tail_risk(
        ...     Pareto(alpha=2.0, xm=1.0), 0.99, n_samples=20000, seed=1
        ... )
        >>> result["n_exceedances"]
        200
    """
    if not (0.0 < level < 1.0):
        raise ValueError("level must be in (0,1).")
    if n_samples < 2:
        raise ValueError("n_samples must be at least 2")

    sample = sorted(dist.rvs(n_samples, seed=seed))
    cut = int(level * n_samples)
    exceedances = sample[cut:]
    if len(exceedances) < 2:
        raise ValueError(
            f"level={level} leaves only {len(exceedances)} exceedances in "
            f"{n_samples} samples; raise n_samples or lower the level"
        )

    var = sample[cut - 1] if cut > 0 else sample[0]

    # Standard error of a sample quantile, where the density is available.
    var_error: float | None = None
    density = getattr(dist, "pdf", None)
    if callable(density):
        try:
            f = float(density(var))
            if f > 0.0:
                var_error = math.sqrt(level * (1.0 - level) / n_samples) / f
        except (ValueError, OverflowError, ZeroDivisionError):
            var_error = None

    if not mean_exists(dist):
        # The sample mean of the exceedances does not converge, so reporting it
        # with a standard error would dress up a meaningless number.
        return {
            "level": level,
            "n_samples": n_samples,
            "n_exceedances": len(exceedances),
            "value_at_risk": float(var),
            "value_at_risk_std_error": var_error,
            "expected_shortfall": math.inf,
            "expected_shortfall_std_error": None,
        }

    count = len(exceedances)
    shortfall = sum(exceedances) / count
    variance = sum((x - shortfall) ** 2 for x in exceedances) / max(count - 1, 1)

    return {
        "level": level,
        "n_samples": n_samples,
        "n_exceedances": count,
        "value_at_risk": float(var),
        "value_at_risk_std_error": var_error,
        "expected_shortfall": float(shortfall),
        "expected_shortfall_std_error": math.sqrt(variance / count),
    }

tail_conditional_expectation

tail_conditional_expectation(dist, level, **kwargs)

Tail conditional expectation, E[X | X > VaR].

For a continuous distribution this is the same quantity as :func:expected_shortfall, and this function delegates to it. The two names come from different literatures, actuarial and financial, and they diverge only for distributions with an atom at the quantile, which none of the continuous families here have.

Parameters:

Name	Type	Description	Default
dist	Any	A distribution with a ppf method.	required
level	float	Confidence level in the open interval (0, 1).	required
**kwargs	Any	Passed to :func:expected_shortfall.	{}

Returns:

Type	Description
float	The tail conditional expectation.

Source code in heavytails/risk.py

def tail_conditional_expectation(dist: Any, level: float, **kwargs: Any) -> float:
    """
    Tail conditional expectation, ``E[X | X > VaR]``.

    For a continuous distribution this is the same quantity as
    :func:`expected_shortfall`, and this function delegates to it. The two
    names come from different literatures, actuarial and financial, and they
    diverge only for distributions with an atom at the quantile, which none of
    the continuous families here have.

    Args:
        dist: A distribution with a ``ppf`` method.
        level: Confidence level in the open interval (0, 1).
        **kwargs: Passed to :func:`expected_shortfall`.

    Returns:
        The tail conditional expectation.
    """
    return expected_shortfall(dist, level, **kwargs)

value_at_risk

value_at_risk(dist, level)

Value at risk: the quantile of the loss distribution at level.

This is exactly dist.ppf(level), and exists as a named function because the terminology is what practitioners search for, and because pairing it with :func:expected_shortfall makes the distinction between the two explicit.

Value at risk says how large a loss is exceeded with probability 1 - level. It says nothing about how large the exceedances are, which is what expected shortfall answers and why the two are usually reported together.

Parameters:

Name	Type	Description	Default
dist	Any	A distribution with a ppf method.	required
level	float	Confidence level in the open interval (0, 1), for example 0.99.	required

Returns:

Type	Description
float	The quantile. inf when the quantile exceeds the float range.

Raises:

Type	Description
ValueError	If level is outside (0, 1).

Examples:

>>> from heavytails import Pareto
>>> round(value_at_risk(Pareto(alpha=2.0, xm=1.0), 0.99), 4)
10.0

Source code in heavytails/risk.py

def value_at_risk(dist: Any, level: float) -> float:
    """
    Value at risk: the quantile of the loss distribution at ``level``.

    This is exactly ``dist.ppf(level)``, and exists as a named function because
    the terminology is what practitioners search for, and because pairing it
    with :func:`expected_shortfall` makes the distinction between the two
    explicit.

    Value at risk says how large a loss is exceeded with probability
    ``1 - level``. It says nothing about how large the exceedances are, which
    is what expected shortfall answers and why the two are usually reported
    together.

    Args:
        dist: A distribution with a ``ppf`` method.
        level: Confidence level in the open interval (0, 1), for example 0.99.

    Returns:
        The quantile. ``inf`` when the quantile exceeds the float range.

    Raises:
        ValueError: If ``level`` is outside (0, 1).

    Examples:
        >>> from heavytails import Pareto
        >>> round(value_at_risk(Pareto(alpha=2.0, xm=1.0), 0.99), 4)
        10.0
    """
    if not (0.0 < level < 1.0):
        raise ValueError("level must be in (0,1).")
    return float(dist.ppf(level))