oversampleqa.validator

oversampleqa.validator

Oversampling validation utilities.

ValidationSplit

Bases: NamedTuple

Training data and reference sets for one validation run.

Shared by :func:validate_oversampling, MemoryEfficientValidator and TypedValidator so the three cannot drift apart on what the error rate measures.

Source code in src/oversampleqa/validator.py

class ValidationSplit(NamedTuple):
    """Training data and reference sets for one validation run.

    Shared by :func:`validate_oversampling`, ``MemoryEfficientValidator`` and
    ``TypedValidator`` so the three cannot drift apart on what the error rate
    measures.
    """

    X_train: NDArray[np.floating]
    y_train: NDArray[np.integer]
    hid_majority: NDArray[np.floating]
    fit_minority: NDArray[np.floating]
    reference_minority: NDArray[np.floating]
    hidden_majority_index: NDArray[np.integer]

infer_minority_label(y)

Return the least frequent label in y.

Benchmark catalogs used to hardcode minority_label=1, which is right for a make_classification dataset built with descending weights and wrong for real data: load_breast_cancer has 212 malignant against 357 benign, so its minority is class 0 and the catalog declared 1 -- the whole benchmark was oversampling the majority.

Truncation makes a fixed answer impossible rather than merely wrong. The first 200 rows of load_breast_cancer are 104 class-0 against 96 class-1, so max_samples inverts which class is rarer. Deriving the label from the data actually returned is the only answer correct at every size.

Parameters:

Name	Type	Description	Default
y	NDArray[integer]	Label array.	required

Returns:

Type	Description
int	The least frequent label. Ties resolve to the smaller label, which
int	keeps the result deterministic; a perfectly balanced dataset has no
int	minority and the caller should not be asking.

Raises:

Type	Description
ValueError	If y is empty.

Source code in src/oversampleqa/validator.py

def infer_minority_label(y: NDArray[np.integer]) -> int:
    """Return the least frequent label in ``y``.

    Benchmark catalogs used to hardcode ``minority_label=1``, which is right for
    a ``make_classification`` dataset built with descending weights and wrong
    for real data: ``load_breast_cancer`` has 212 malignant against 357 benign,
    so its minority is class 0 and the catalog declared 1 -- the whole benchmark
    was oversampling the majority.

    Truncation makes a fixed answer impossible rather than merely wrong. The
    first 200 rows of ``load_breast_cancer`` are 104 class-0 against 96 class-1,
    so ``max_samples`` inverts which class is rarer. Deriving the label from the
    data actually returned is the only answer correct at every size.

    Args:
        y: Label array.

    Returns:
        The least frequent label. Ties resolve to the smaller label, which
        keeps the result deterministic; a perfectly balanced dataset has no
        minority and the caller should not be asking.

    Raises:
        ValueError: If ``y`` is empty.
    """
    labels, counts = np.unique(np.asarray(y), return_counts=True)
    if labels.size == 0:
        raise ValueError("cannot infer a minority label from an empty array")
    return int(labels[int(np.argmin(counts))])

prepare_validation_split(X, y, minority_label, majority_label, hidden_ratio, *, reference='hidden_minority', minority_hidden_ratio=None, min_hidden=5, random_state=42, stratify_by=None)

Build the train/hidden split defining the validation estimand.

See :func:validate_oversampling for the meaning of reference, random_state and stratify_by.

Raises

ValueError If the held-out minority would contain fewer than min_hidden points, or if stratify_by is not aligned with the majority class.

Source code in src/oversampleqa/validator.py

def prepare_validation_split(
    X: NDArray[np.floating],
    y: NDArray[np.integer],
    minority_label: int,
    majority_label: int,
    hidden_ratio: float,
    *,
    reference: ReferenceSet = "hidden_minority",
    minority_hidden_ratio: float | None = None,
    min_hidden: int = 5,
    random_state: RandomStateLike = 42,
    stratify_by: NDArray[Any] | None = None,
) -> ValidationSplit:
    """Build the train/hidden split defining the validation estimand.

    See :func:`validate_oversampling` for the meaning of ``reference``,
    ``random_state`` and ``stratify_by``.

    Raises
    ------
    ValueError
        If the held-out minority would contain fewer than ``min_hidden``
        points, or if ``stratify_by`` is not aligned with the majority class.
    """
    minority, majority = _split_classes(X, y, minority_label)
    rng = as_generator(random_state)

    majority_strata = None
    if stratify_by is not None:
        stratify_by = np.asarray(stratify_by)
        if len(stratify_by) != len(y):
            raise ValueError(
                f"stratify_by has length {len(stratify_by)} but y has {len(y)}; "
                "it must be aligned with the full dataset."
            )
        majority_strata = stratify_by[y != minority_label]

    vis_idx, hid_idx = _holdout_indices(
        len(majority), hidden_ratio, rng, majority_strata
    )
    vis_majority = majority[vis_idx]
    hid_majority = majority[hid_idx]

    if reference == "hidden_minority":
        ratio = hidden_ratio if minority_hidden_ratio is None else minority_hidden_ratio
        _validate_hidden_ratio(ratio)
        n_hidden_minority = int(len(minority) * ratio)
        if n_hidden_minority < min_hidden:
            raise ValueError(
                f"Holding out {ratio:.3g} of a minority class of "
                f"{len(minority)} leaves {n_hidden_minority} held-out points, "
                f"below min_hidden={min_hidden}. A nearest-neighbour comparison "
                "against so few points is not meaningful. Either supply more "
                "minority data, raise minority_hidden_ratio, or pass "
                "reference='train_minority' -- noting that it compares against "
                "the oversampler's own training data and is biased toward zero."
            )
        fit_idx, ref_idx = _holdout_indices(len(minority), ratio, rng)
        fit_minority = minority[fit_idx]
        reference_minority = minority[ref_idx]
    else:
        fit_minority = minority
        reference_minority = minority

    X_train = np.vstack([vis_majority, fit_minority])
    y_train = np.hstack(
        [
            np.full(len(vis_majority), majority_label, dtype=y.dtype),
            np.full(len(fit_minority), minority_label, dtype=y.dtype),
        ]
    )
    return ValidationSplit(
        X_train, y_train, hid_majority, fit_minority, reference_minority, hid_idx
    )

score_nearest_distances(nearest_hidden, nearest_min)

Count errors and ties from nearest-neighbour distances.

Returns (n_errors, n_ties). The comparison is strict: a point exactly equidistant from both reference sets is not evidence of a majority-like artefact, and counting ties as errors is a one-directional bias on discrete or quantised features.

Source code in src/oversampleqa/validator.py

def score_nearest_distances(
    nearest_hidden: NDArray[np.floating],
    nearest_min: NDArray[np.floating],
) -> tuple[int, int]:
    """Count errors and ties from nearest-neighbour distances.

    Returns ``(n_errors, n_ties)``. The comparison is strict: a point exactly
    equidistant from both reference sets is not evidence of a majority-like
    artefact, and counting ties as errors is a one-directional bias on
    discrete or quantised features.
    """
    errors = int(np.sum(nearest_hidden < nearest_min))
    ties = int(np.sum(nearest_hidden == nearest_min))
    return errors, ties

warn_reference_bias(reference, stacklevel=3)

Emit the train_minority bias warning.

Source code in src/oversampleqa/validator.py

def warn_reference_bias(reference: ReferenceSet, stacklevel: int = 3) -> None:
    """Emit the ``train_minority`` bias warning."""
    if reference == "train_minority":
        warnings.warn(
            "reference='train_minority' compares synthetic points against the "
            "minority data the oversampler interpolated from. Held-out data on "
            "one side and training data on the other biases the error rate "
            "toward zero by an amount that depends on minority density, not on "
            "oversampler quality. Use reference='hidden_minority' for a "
            "comparison where both sides are unseen.",
            FutureWarning,
            stacklevel=stacklevel,
        )

extract_synthetic_samples(X_original, X_resampled, y_resampled, minority_label)

Return synthetic minority samples from a resampled dataset.

Parameters

X_original : ndarray Original feature matrix used for fitting the oversampler. X_resampled : ndarray Feature matrix returned by oversampler.fit_resample. y_resampled : ndarray Corresponding labels for X_resampled. minority_label : int Label of the minority class that was oversampled.

Returns

ndarray Array containing only the synthetic minority samples.

Raises

ValueError If the oversampler did not preserve the original samples as a prefix of its output, so synthetic rows cannot be identified positionally.

Notes

Synthetic samples are identified positionally: everything after the original rows is assumed to be new. That holds for the SMOTE family and RandomOverSampler, which append. It does not hold for combined over/under-samplers such as SMOTEENN and SMOTETomek, which delete original rows. A length check alone does not catch this -- SMOTEENN can still return more rows than it was given while having removed some of the originals -- so the prefix is compared element-wise.

Source code in src/oversampleqa/validator.py

def extract_synthetic_samples(
    X_original: NDArray[np.floating],
    X_resampled: NDArray[np.floating],
    y_resampled: NDArray[np.integer],
    minority_label: int,
) -> NDArray[np.floating]:
    """Return synthetic minority samples from a resampled dataset.

    Parameters
    ----------
    X_original : ndarray
        Original feature matrix used for fitting the oversampler.
    X_resampled : ndarray
        Feature matrix returned by ``oversampler.fit_resample``.
    y_resampled : ndarray
        Corresponding labels for ``X_resampled``.
    minority_label : int
        Label of the minority class that was oversampled.

    Returns
    -------
    ndarray
        Array containing only the synthetic minority samples.

    Raises
    ------
    ValueError
        If the oversampler did not preserve the original samples as a prefix
        of its output, so synthetic rows cannot be identified positionally.

    Notes
    -----
    Synthetic samples are identified positionally: everything after the
    original rows is assumed to be new. That holds for the SMOTE family and
    ``RandomOverSampler``, which append. It does **not** hold for combined
    over/under-samplers such as ``SMOTEENN`` and ``SMOTETomek``, which delete
    original rows. A length check alone does not catch this -- ``SMOTEENN``
    can still return more rows than it was given while having removed some of
    the originals -- so the prefix is compared element-wise.
    """
    require_prefix_preserved(X_original, X_resampled, "validate_oversampling")
    n = len(X_original)
    synthetic: NDArray[np.floating] = X_resampled[n:][y_resampled[n:] == minority_label]
    return synthetic

require_prefix_preserved(X_original, X_resampled, caller)

Raise unless the resampler left the original rows as an unchanged prefix.

Synthetic rows are identified positionally, so a resampler that deletes or reorders originals makes that slice meaningless. A length check alone does not catch it -- SMOTEENN can return more rows than it was given while having removed some originals -- so the prefix is compared element-wise.

Shared by the binary and multiclass paths. The multiclass path had no such check and would return plausible-looking numbers from a misaligned slice.

Parameters:

Name	Type	Description	Default
X_original	NDArray[floating]	Matrix passed to fit_resample.	required
X_resampled	NDArray[floating]	Matrix returned by it.	required
caller	str	Function name, for the message.	required

Raises:

Type	Description
ValueError	If the originals are not an unchanged prefix.

Source code in src/oversampleqa/validator.py

def require_prefix_preserved(
    X_original: NDArray[np.floating],
    X_resampled: NDArray[np.floating],
    caller: str,
) -> None:
    """Raise unless the resampler left the original rows as an unchanged prefix.

    Synthetic rows are identified positionally, so a resampler that deletes or
    reorders originals makes that slice meaningless. A length check alone does
    not catch it -- ``SMOTEENN`` can return more rows than it was given while
    having removed some originals -- so the prefix is compared element-wise.

    Shared by the binary and multiclass paths. The multiclass path had no such
    check and would return plausible-looking numbers from a misaligned slice.

    Args:
        X_original: Matrix passed to ``fit_resample``.
        X_resampled: Matrix returned by it.
        caller: Function name, for the message.

    Raises:
        ValueError: If the originals are not an unchanged prefix.
    """
    n = len(X_original)
    if len(X_resampled) < n or not np.array_equal(X_resampled[:n], X_original):
        raise ValueError(
            "The oversampler did not preserve the original samples as a prefix of "
            "its output, so synthetic samples cannot be identified positionally. "
            "This is expected for combined over/under-samplers such as SMOTEENN "
            f"and SMOTETomek, which are not supported by {caller}."
        )

validate_oversampling(X, y, minority_label, oversampler, hidden_ratio=0.1, metric='hassanat', metric_kwargs=None, return_details=False, *, reference='hidden_minority', minority_hidden_ratio=None, min_hidden=5, duplication_warn_threshold=0.5, random_state=42, stratify_by=None, n_repeats=1, reseed_oversampler=False)

Validate oversampling using the hidden majority approach.

A fraction of the majority class is held out. The oversampler is fitted on what remains, and each synthetic minority point is scored by comparing its nearest-neighbour distance to the hidden majority against its nearest-neighbour distance to a minority reference set. A synthetic point strictly closer to the hidden majority counts as an error.

Parameters

X, y : ndarray Input data and labels. minority_label : int Label of the minority class. oversampler : BaseOverSampler Instance of an imbalanced-learn oversampler. hidden_ratio : float, default=0.1 Fraction of majority samples to hide during validation. metric : str, default="hassanat" Distance metric to use. metric_kwargs : dict, optional Additional keyword arguments passed to :func:distance_matrix. return_details : bool, default=False If True return a :class:~oversampleqa.types.ValidationDetails instead of the bare error rate. reference : {"hidden_minority", "train_minority"}, default="hidden_minority" Which minority set the synthetic points are compared against.

``"hidden_minority"`` also holds out part of the minority class and
compares against those held-out points. Both sides of the comparison
are then unseen, which is the same estimand
:func:`validate_multiclass_oversampling` uses.

``"train_minority"`` compares against the full minority class -- the
very data the oversampler interpolated from. This is the historical
behaviour, retained so old numbers can be reproduced. It biases the
result toward "no error" by an amount that depends on minority
density rather than on oversampler quality, and emits a
``FutureWarning``.

minority_hidden_ratio : float, optional Fraction of the minority class to hide when reference="hidden_minority". Defaults to hidden_ratio. min_hidden : int, default=5 Minimum number of held-out minority points required. A nearest-neighbour comparison against fewer than a handful of points is not meaningful, so this raises rather than warns. duplication_warn_threshold : float, default=0.5 Emit a UserWarning when this fraction of synthetic points coincide with a real point. random_state : int, Generator, SeedSequence or None, default=42 Seeds the hold-out split. Which points get hidden is the single largest driver of the error rate, so varying this varies the result. The default of 42 reproduces previously documented numbers; None draws fresh entropy and is not reproducible. stratify_by : ndarray, optional Group labels aligned with y. When given, the majority hold-out takes hidden_ratio within each group, so a hold-out cannot miss a cluster entirely. Strata are never inferred automatically. n_repeats : int, default=1 Number of independent hold-out splits. Above 1, the returned details carry the per-repeat vector and its dispersion. Repeat streams are spawned from a SeedSequence rather than derived as seed + i, which would correlate them. reseed_oversampler : bool, default=False Give the oversampler a fresh seed on each repeat. This changes what the dispersion covers -- see Notes.

Returns

float or ValidationDetails Error rate by default, nan if no synthetic samples were produced. With n_repeats > 1 the bare return is the mean across repeats.

Raises

ValueError If the labels are not binary, if minority_label is absent, if the held-out minority would be smaller than min_hidden, or if the oversampler does not preserve the original samples as a prefix.

Notes

The error rate is a relative quantity. Its scale depends on hidden_ratio, on the density of the data, and on dimensionality, so values are not comparable across datasets. See :doc:/concepts.

What the repeat interval covers. With n_repeats > 1 the reported interval is a percentile bootstrap over the per-repeat error rates. It describes the variability of the hold-out split, conditional on this dataset and on the oversampler's own seed. It is not a confidence interval for a population quantity, and with reseed_oversampler=False it does not include the oversampler's own randomness at all. Setting reseed_oversampler=True clones the sampler with a fresh seed per repeat, so the dispersion then covers both sources together -- a different, wider decomposition.

Synthetic points generated from shared parent points are not independent, so a binomial interval on the error rate would be too narrow. Nothing here claims more than the repeat-level bootstrap supports.

Source code in src/oversampleqa/validator.py

def validate_oversampling(
    X: NDArray[np.floating],
    y: NDArray[np.integer],
    minority_label: int,
    oversampler: BaseOverSampler,
    hidden_ratio: float = 0.1,
    metric: str = "hassanat",
    metric_kwargs: dict[str, Any] | None = None,
    return_details: bool = False,
    *,
    reference: ReferenceSet = "hidden_minority",
    minority_hidden_ratio: float | None = None,
    min_hidden: int = 5,
    duplication_warn_threshold: float = 0.5,
    random_state: RandomStateLike = 42,
    stratify_by: NDArray[Any] | None = None,
    n_repeats: int = 1,
    reseed_oversampler: bool = False,
) -> float | ValidationDetails:
    """Validate oversampling using the hidden majority approach.

    A fraction of the majority class is held out. The oversampler is fitted on
    what remains, and each synthetic minority point is scored by comparing its
    nearest-neighbour distance to the hidden majority against its
    nearest-neighbour distance to a minority reference set. A synthetic point
    strictly closer to the hidden majority counts as an error.

    Parameters
    ----------
    X, y : ndarray
        Input data and labels.
    minority_label : int
        Label of the minority class.
    oversampler : BaseOverSampler
        Instance of an `imbalanced-learn` oversampler.
    hidden_ratio : float, default=0.1
        Fraction of majority samples to hide during validation.
    metric : str, default="hassanat"
        Distance metric to use.
    metric_kwargs : dict, optional
        Additional keyword arguments passed to :func:`distance_matrix`.
    return_details : bool, default=False
        If ``True`` return a :class:`~oversampleqa.types.ValidationDetails`
        instead of the bare error rate.
    reference : {"hidden_minority", "train_minority"}, default="hidden_minority"
        Which minority set the synthetic points are compared against.

        ``"hidden_minority"`` also holds out part of the minority class and
        compares against those held-out points. Both sides of the comparison
        are then unseen, which is the same estimand
        :func:`validate_multiclass_oversampling` uses.

        ``"train_minority"`` compares against the full minority class -- the
        very data the oversampler interpolated from. This is the historical
        behaviour, retained so old numbers can be reproduced. It biases the
        result toward "no error" by an amount that depends on minority
        density rather than on oversampler quality, and emits a
        ``FutureWarning``.
    minority_hidden_ratio : float, optional
        Fraction of the minority class to hide when
        ``reference="hidden_minority"``. Defaults to ``hidden_ratio``.
    min_hidden : int, default=5
        Minimum number of held-out minority points required. A
        nearest-neighbour comparison against fewer than a handful of points is
        not meaningful, so this raises rather than warns.
    duplication_warn_threshold : float, default=0.5
        Emit a ``UserWarning`` when this fraction of synthetic points coincide
        with a real point.
    random_state : int, Generator, SeedSequence or None, default=42
        Seeds the hold-out split. **Which** points get hidden is the single
        largest driver of the error rate, so varying this varies the result.
        The default of 42 reproduces previously documented numbers; ``None``
        draws fresh entropy and is not reproducible.
    stratify_by : ndarray, optional
        Group labels aligned with ``y``. When given, the majority hold-out
        takes ``hidden_ratio`` within each group, so a hold-out cannot miss a
        cluster entirely. Strata are never inferred automatically.
    n_repeats : int, default=1
        Number of independent hold-out splits. Above 1, the returned details
        carry the per-repeat vector and its dispersion. Repeat streams are
        spawned from a ``SeedSequence`` rather than derived as ``seed + i``,
        which would correlate them.
    reseed_oversampler : bool, default=False
        Give the oversampler a fresh seed on each repeat. This changes what the
        dispersion covers -- see Notes.

    Returns
    -------
    float or ValidationDetails
        Error rate by default, ``nan`` if no synthetic samples were produced.
        With ``n_repeats > 1`` the bare return is the mean across repeats.

    Raises
    ------
    ValueError
        If the labels are not binary, if ``minority_label`` is absent, if the
        held-out minority would be smaller than ``min_hidden``, or if the
        oversampler does not preserve the original samples as a prefix.

    Notes
    -----
    The error rate is a **relative** quantity. Its scale depends on
    ``hidden_ratio``, on the density of the data, and on dimensionality, so
    values are not comparable across datasets. See :doc:`/concepts`.

    **What the repeat interval covers.** With ``n_repeats > 1`` the reported
    interval is a percentile bootstrap over the per-repeat error rates. It
    describes the variability of the **hold-out split**, conditional on this
    dataset and on the oversampler's own seed. It is *not* a confidence
    interval for a population quantity, and with ``reseed_oversampler=False``
    it does not include the oversampler's own randomness at all. Setting
    ``reseed_oversampler=True`` clones the sampler with a fresh seed per
    repeat, so the dispersion then covers both sources together -- a different,
    wider decomposition.

    Synthetic points generated from shared parent points are **not
    independent**, so a binomial interval on the error rate would be too
    narrow. Nothing here claims more than the repeat-level bootstrap supports.
    """
    _validate_hidden_ratio(hidden_ratio)
    require_pointwise_metric(metric)
    labels = np.unique(y)
    if minority_label not in labels:
        raise ValueError(f"minority_label {minority_label} not found in y")
    if len(labels) != 2:
        raise ValueError(
            "validate_oversampling expects binary labels; use validate_multiclass_oversampling for multi-class data"
        )
    majority_label = int(labels[labels != minority_label][0])

    if reference not in ("hidden_minority", "train_minority"):
        raise ValueError(
            f"reference must be 'hidden_minority' or 'train_minority'; got {reference!r}"
        )
    warn_reference_bias(reference, stacklevel=3)

    if n_repeats < 1:
        raise ValueError(f"n_repeats must be at least 1; got {n_repeats}")

    if n_repeats > 1:
        return _validate_repeated(
            X,
            y,
            minority_label,
            oversampler,
            hidden_ratio=hidden_ratio,
            metric=metric,
            metric_kwargs=metric_kwargs,
            return_details=return_details,
            reference=reference,
            minority_hidden_ratio=minority_hidden_ratio,
            min_hidden=min_hidden,
            duplication_warn_threshold=duplication_warn_threshold,
            random_state=random_state,
            stratify_by=stratify_by,
            n_repeats=n_repeats,
            reseed_oversampler=reseed_oversampler,
        )

    minority, _ = _split_classes(X, y, minority_label)
    split = prepare_validation_split(
        X,
        y,
        minority_label,
        majority_label,
        hidden_ratio,
        reference=reference,
        minority_hidden_ratio=minority_hidden_ratio,
        min_hidden=min_hidden,
        random_state=random_state,
        stratify_by=stratify_by,
    )
    X_train = split.X_train
    y_train = split.y_train
    hid_majority = split.hid_majority
    fit_minority = split.fit_minority
    reference_minority = split.reference_minority

    try:
        X_res, y_res = oversampler.fit_resample(X_train, y_train)
    except Exception as exc:
        logger.exception("Oversampler failed during fit_resample")
        raise ValueError(
            f"{type(oversampler).__name__} failed to fit on the reduced training "
            f"set ({len(fit_minority)} minority points after holding out "
            f"{len(minority) - len(fit_minority)}). Neighbour-based samplers "
            "such as SMOTE require more minority points than their k_neighbors "
            "setting. Lower k_neighbors, lower minority_hidden_ratio, or supply "
            f"more minority data. Original error: {exc}"
        ) from exc

    synthetic = extract_synthetic_samples(X_train, X_res, y_res, minority_label)

    kwargs = metric_kwargs or {}
    empty = np.empty((0, 0))

    if len(synthetic) == 0:
        warnings.warn(
            f"{type(oversampler).__name__} produced no synthetic minority "
            "samples, so there is nothing to validate. Returning nan rather "
            "than 0.0, which would be indistinguishable from a perfect score.",
            UserWarning,
            stacklevel=2,
        )
        rate = float("nan")
        if return_details:
            return ValidationDetails(
                error_rate=rate,
                n_errors=0,
                n_synthetic=0,
                n_ties=0,
                duplication_rate=float("nan"),
                reference=reference,
                dist_hidden=empty,
                dist_min=empty,
            )
        return rate

    dup_rate = duplication_rate(synthetic, fit_minority)
    if dup_rate >= duplication_warn_threshold:
        warnings.warn(
            f"{dup_rate:.0%} of the synthetic samples produced by "
            f"{type(oversampler).__name__} are exact copies of real minority "
            "points. The validation error rate is not informative for a sampler "
            "that mostly duplicates: copied points sit at distance zero from the "
            "minority set and so can never be scored as errors.",
            UserWarning,
            stacklevel=2,
        )

    dist_hidden = distance_matrix(synthetic, hid_majority, metric, **kwargs)
    dist_min = distance_matrix(synthetic, reference_minority, metric, **kwargs)

    nearest_hidden = dist_hidden.min(axis=1)
    nearest_min = dist_min.min(axis=1)

    errors, n_ties = score_nearest_distances(nearest_hidden, nearest_min)
    rate = calculate_error_rate(errors, len(synthetic))

    if n_ties > 0.01 * len(synthetic):
        warnings.warn(
            f"{n_ties} of {len(synthetic)} synthetic points are exactly "
            "equidistant from the hidden majority and the minority reference "
            "set. Ties are excluded from the error count, but this many "
            "suggests duplicated or heavily quantised features.",
            UserWarning,
            stacklevel=2,
        )

    if return_details:
        return ValidationDetails(
            error_rate=rate,
            n_errors=errors,
            n_synthetic=len(synthetic),
            n_ties=n_ties,
            duplication_rate=dup_rate,
            reference=reference,
            dist_hidden=dist_hidden,
            dist_min=dist_min,
        )

    return rate

validate_multiclass_oversampling(X, y, oversampler, hidden_ratio=0.1, metric='hassanat', metric_kwargs=None, return_matrix=False, *, min_hidden=5, random_state=42)

Validate oversampling for multi-class datasets.

For each class label, a portion of samples is hidden and the remainder is used for training the oversampler along with all other visible classes. The nearest hidden class to each synthetic sample determines the error attribution. The function returns the per-class error rates and optionally the full error matrix where matrix[i, j] counts how many synthetic samples generated for class i are closest to hidden samples from class j.

Parameters

X, y : ndarray Input data and labels. oversampler : BaseOverSampler Instance of an imbalanced-learn oversampler supporting multi-class data. hidden_ratio : float, default=0.1 Fraction of each class to hide during validation. metric : str, default="hassanat" Distance metric to use. metric_kwargs : dict, optional Additional keyword arguments passed to :func:distance_matrix. return_matrix : bool, default=False If True also return the error matrix. min_hidden : int, default=5 Minimum held-out points per class. Every class is a reference for every other, so a class that cannot supply a usable hidden set makes attribution unreliable for all of them. random_state : int, Generator, SeedSequence or None, default=42 Seeds the per-class hold-out. Defaults to 42, which reproduces previously documented numbers.

Returns

dict or tuple Mapping of class_label -> error_rate. If return_matrix is True the second element is the error matrix.

A class for which the sampler generated no synthetic points maps to
``nan``, not ``0.0``: nothing was measured, and ``0.0`` is the score of
a perfect result. Use :func:`macro_error_rate` to summarise, or
``np.nanmean`` over the values -- a plain mean propagates the ``nan``.

Raises

ValueError If hidden_ratio is out of range, if any class would hide fewer than min_hidden points, or if the oversampler did not preserve the original rows as a prefix of its output.

Notes

A synthetic point equidistant from its own class and another is attributed to its own class, matching :func:score_nearest_distances. Ties previously went to whichever class appeared first in label order, which biased results toward low-numbered classes for reasons unrelated to the data.

Source code in src/oversampleqa/validator.py

def validate_multiclass_oversampling(
    X: NDArray[np.floating],
    y: NDArray[np.integer],
    oversampler: BaseOverSampler,
    hidden_ratio: float = 0.1,
    metric: str = "hassanat",
    metric_kwargs: dict[str, Any] | None = None,
    return_matrix: bool = False,
    *,
    min_hidden: int = 5,
    random_state: RandomStateLike = 42,
) -> dict[int, float] | tuple[dict[int, float], NDArray[np.floating]]:
    """Validate oversampling for multi-class datasets.

    For each class label, a portion of samples is hidden and the remainder
    is used for training the oversampler along with all other visible
    classes. The nearest hidden class to each synthetic sample determines
    the error attribution. The function returns the per-class error rates
    and optionally the full error matrix where ``matrix[i, j]`` counts how
    many synthetic samples generated for class ``i`` are closest to hidden
    samples from class ``j``.

    Parameters
    ----------
    X, y : ndarray
        Input data and labels.
    oversampler : BaseOverSampler
        Instance of an `imbalanced-learn` oversampler supporting
        multi-class data.
    hidden_ratio : float, default=0.1
        Fraction of each class to hide during validation.
    metric : str, default="hassanat"
        Distance metric to use.
    metric_kwargs : dict, optional
        Additional keyword arguments passed to :func:`distance_matrix`.
    return_matrix : bool, default=False
        If ``True`` also return the error matrix.
    min_hidden : int, default=5
        Minimum held-out points per class. Every class is a reference for
        every other, so a class that cannot supply a usable hidden set makes
        attribution unreliable for all of them.
    random_state : int, Generator, SeedSequence or None, default=42
        Seeds the per-class hold-out. Defaults to 42, which reproduces
        previously documented numbers.

    Returns
    -------
    dict or tuple
        Mapping of ``class_label -> error_rate``. If ``return_matrix`` is
        ``True`` the second element is the error matrix.

        A class for which the sampler generated no synthetic points maps to
        ``nan``, not ``0.0``: nothing was measured, and ``0.0`` is the score of
        a perfect result. Use :func:`macro_error_rate` to summarise, or
        ``np.nanmean`` over the values -- a plain mean propagates the ``nan``.

    Raises
    ------
    ValueError
        If ``hidden_ratio`` is out of range, if any class would hide fewer than
        ``min_hidden`` points, or if the oversampler did not preserve the
        original rows as a prefix of its output.

    Notes
    -----
    A synthetic point equidistant from its own class and another is attributed
    to its own class, matching :func:`score_nearest_distances`. Ties previously
    went to whichever class appeared first in label order, which biased results
    toward low-numbered classes for reasons unrelated to the data.
    """

    _validate_hidden_ratio(hidden_ratio)
    require_pointwise_metric(metric)
    labels = np.unique(y)
    rng = as_generator(random_state)

    # Every class supplies a hidden reference, so every class must be able to
    # support one. Previously a class whose hold-out rounded to zero got an
    # empty reference and was silently dropped from attribution: synthetic
    # points could never be scored against it, so it could never receive an
    # error, and its own rate was computed against a reduced set of rivals.
    too_small = {
        int(lbl): int(np.sum(y == lbl) * hidden_ratio)
        for lbl in labels
        if int(np.sum(y == lbl) * hidden_ratio) < min_hidden
    }
    if too_small:
        detail = ", ".join(
            f"class {lbl} would hide {n}" for lbl, n in sorted(too_small.items())
        )
        raise ValueError(
            f"Hiding {hidden_ratio:.3g} of each class leaves fewer than "
            f"min_hidden={min_hidden} points for: {detail}. A nearest-neighbour "
            "comparison against so few points is not meaningful, and every class "
            "is used as a reference for every other. Supply more data for those "
            "classes, raise hidden_ratio, or lower min_hidden."
        )

    visible = {}
    hidden = {}
    for label in labels:
        cls_samples = X[y == label]
        n_hidden = int(len(cls_samples) * hidden_ratio)
        idx = rng.permutation(len(cls_samples))
        hidden[label] = cls_samples[idx[:n_hidden]]
        visible[label] = cls_samples[idx[n_hidden:]]

    X_train = np.vstack([visible[lbl] for lbl in labels])
    y_train = np.hstack([[lbl] * len(visible[lbl]) for lbl in labels])

    try:
        X_res, y_res = oversampler.fit_resample(X_train, y_train)
    except Exception:  # pragma: no cover - defensive
        logger.exception("Oversampler failed during fit_resample")
        raise

    # The binary path has always checked this; the multiclass path did not, so
    # a combined sampler that deletes original rows produced a misaligned slice
    # and plausible-looking numbers from it.
    require_prefix_preserved(X_train, X_res, "validate_multiclass_oversampling")

    start = len(X_train)
    X_syn = X_res[start:]
    y_syn = y_res[start:]

    metric_kwargs = metric_kwargs or {}
    matrix = np.zeros((len(labels), len(labels)), dtype=int)

    # Distance from every synthetic point to each class's hidden reference.
    hidden_dists = {
        lbl: distance_matrix(X_syn, hidden[lbl], metric, **metric_kwargs)
        for lbl in labels
    }

    for i, lbl in enumerate(labels):
        rows = y_syn == lbl
        if not np.any(rows):
            continue
        # Nearest distance to each class, as (n_synthetic, n_classes).
        per_class = np.column_stack(
            [hidden_dists[other][rows].min(axis=1) for other in labels]
        )
        own = per_class[:, i]
        rivals = np.delete(per_class, i, axis=1)
        best_rival = rivals.min(axis=1)

        # A tie is attributed to the point's own class. Strict `<` on a running
        # minimum previously gave ties to whichever class came first in label
        # order, a systematic bias toward low-numbered classes that had nothing
        # to do with the data. This matches score_nearest_distances, where a tie
        # is not evidence of an error.
        own_wins = own <= best_rival
        rival_idx = np.argmin(rivals, axis=1)
        # argmin indexes the array with column i removed; shift back past it.
        rival_idx[rival_idx >= i] += 1

        attribution = np.where(own_wins, i, rival_idx)
        for j in range(len(labels)):
            matrix[i, j] = int(np.sum(attribution == j))

    error_rates = {}
    for i, lbl in enumerate(labels):
        n_syn = int(matrix[i].sum())
        if n_syn == 0:
            # nan, not 0.0. The sampler generated nothing for this class, so
            # nothing was measured -- and 0.0 is the score of a perfect result.
            error_rates[int(lbl)] = float("nan")
            continue
        errors = n_syn - int(matrix[i, i])
        error_rates[int(lbl)] = calculate_error_rate(errors, n_syn)

    if return_matrix:
        return error_rates, matrix

    return error_rates

macro_error_rate(error_rates)

Average per-class error rates over the classes actually measured.

:func:validate_multiclass_oversampling reports nan for a class the sampler generated nothing for. A plain mean propagates that to the summary, turning "one class was not measured" into "no result at all"; counting the nan as zero would be worse still, since it would read as a perfect score for the class that was never evaluated.

Parameters:

Name	Type	Description	Default
error_rates	Mapping[int, float]	Mapping of class label to error rate.	required

Returns:

Type	Description
float	Mean over the classes with a measurement, or nan when none has one.

Source code in src/oversampleqa/validator.py

def macro_error_rate(error_rates: Mapping[int, float]) -> float:
    """Average per-class error rates over the classes actually measured.

    :func:`validate_multiclass_oversampling` reports ``nan`` for a class the
    sampler generated nothing for. A plain mean propagates that to the summary,
    turning "one class was not measured" into "no result at all"; counting the
    ``nan`` as zero would be worse still, since it would read as a perfect score
    for the class that was never evaluated.

    Args:
        error_rates: Mapping of class label to error rate.

    Returns:
        Mean over the classes with a measurement, or ``nan`` when none has one.
    """
    values = np.asarray(list(error_rates.values()), dtype=float)
    measured = values[~np.isnan(values)]
    if measured.size == 0:
        return float("nan")
    return float(measured.mean())