API Reference

Public API

oversampleqa: A diagnostic toolkit for validating oversampling methods.

This package implements validation methods for synthetic data generated by oversampling techniques like SMOTE, ADASYN, and their variants.

SCHEMA_VERSION = '1.0' module-attribute

Version of the exported JSON structure.

Bump the minor part for additive changes and the major part when a field is removed or changes meaning. Consumers should refuse a major version they do not recognise rather than guess.

ReferenceSet = Literal['hidden_minority', 'train_minority'] module-attribute

Which minority set validation compares synthetic points against.

DatasetRepository

Repository for curated real-world and synthetic benchmarking datasets.

load_research_datasets(domains=None, max_samples=10000, include_openml=False)

Load curated datasets for benchmarking.

Parameters:

Name	Type	Description	Default
domains	Sequence[str] | None	Domain names to load.	None
max_samples	int	Maximum number of samples per dataset.	10000
include_openml	bool	Whether to attempt OpenML downloads.	False

Returns:

Type	Description
list[dict[str, Any]]	List of dataset descriptors.

create_synthetic_benchmark_suite(difficulty_levels=None)

Generate synthetic datasets for the requested difficulty levels.

Parameters:

Name	Type	Description	Default
difficulty_levels	Sequence[str] | None	Difficulty labels to generate.	None

Returns:

Type	Description
list[dict[str, Any]]	List of synthetic dataset descriptors.

StatisticalBenchmark

Advanced benchmarking engine with statistical analysis.

run_comprehensive_benchmark(datasets, oversamplers, metrics=None)

Run repeated stratified benchmarking across datasets.

Parameters

datasets: Sequence of dataset descriptors. Each entry should provide data, target and optionally name and minority_label. oversamplers: Sequence of initialised oversampler instances (will be cloned). metrics: Distance metrics to evaluate. Defaults to Hassanat, Euclidean, Mahalanobis.

fold_results()

Return one row per attempted fold from the most recent run.

The summary frame reports a mean and interval per (dataset, oversampler, metric). That is enough to read a ranking and not enough to check one: it cannot be re-aggregated, plotted as a distribution, or given a different interval, and it does not say how many folds actually contributed.

This frame answers those. Skipped folds are present with error_rate of nan, skipped true and a stated skip_reason, because a mean over three surviving folds of twenty-five is indistinguishable from a mean over twenty-five once the skips are dropped.

split_seed is the seed given to the fold splitter for that repeat, so a single repeat can be reproduced without rerunning the sweep.

Returns:

Type	Description
DataFrame	A long-format frame with :data:_FOLD_COLUMNS. Empty of rows but
DataFrame	not of columns when no run has happened yet, so column access works
DataFrame	either way.

ValidationCache

Caching layer for validation results and distance computations.

Caching is opt-in. Constructing this class is the caller's decision; nothing in the package builds one at import time, and no directory is created until the first write.

.. warning::

Not thread-safe across instances, and not process-safe. A single instance guards its own in-memory bookkeeping with a lock, so concurrent reads and writes through one instance will not corrupt its accounting. joblib on-disk writes are not atomic, so two processes (or two instances pointed at the same directory) writing the same key can interleave and leave a truncated file. Give each process its own cache_dir.

.. note::

Whether caching pays depends entirely on how expensive the metric is relative to hashing its inputs. Content hashing must read every input byte, so for a BLAS-backed metric such as euclidean the cache is a net loss; for hassanat it is worth tens of times the compute. See :doc:/reproducibility.

Parameters

cache_dir : str or Path, optional Where to store cached artefacts. Defaults to the per-user cache directory, never the working directory. max_entries : int, default=128 Upper bound on in-memory distance matrices. Least-recently-used entries are evicted first. memory_mb : int, default=1000 Upper bound on the in-memory tier, in megabytes. Enforced: entries are evicted oldest-first until the total fits.

memory property

Lazily-created joblib store; creates the directory on first use.

size_bytes property

Bytes currently held by the in-memory tier.

clear()

Drop everything held in memory. Does not touch the disk store.

get_data_hash(X, y)

Return stable SHA256 hash for dataset.

Parameters:

Name	Type	Description	Default
X	NDArray[Any]	Feature matrix.	required
y	NDArray[Any]	Target labels.	required

Returns:

Type	Description
str	SHA256 hex digest.

cache_validation_result(params_hash, result)

Persist validation result using joblib.

Parameters:

Name	Type	Description	Default
params_hash	str	Cache key for the run parameters.	required
result	float	Error rate to persist.	required

load_validation_result(params_hash)

Retrieve cached validation result if present.

Parameters:

Name	Type	Description	Default
params_hash	str	Cache key for the run parameters.	required

Returns:

Type	Description
float | None	Cached error rate if available.

cached_distance_matrix(optimizer, X1, X2, metric, batch_size='auto', **kwargs)

Return cached distance matrix or compute and cache it.

The returned array is read-only. Cache hits hand back the stored array rather than a copy, so an in-place operation downstream would otherwise corrupt every later hit silently; the write flag turns that into a loud ValueError instead. Call .copy() if you need to modify it.

batch_size is deliberately not part of the key: batching splits the same computation into chunks and concatenates them, so it cannot change the result. test_caching.py pins that invariant for every registered metric.

Parameters:

Name	Type	Description	Default
optimizer	OptimizedDistanceMatrix	OptimizedDistanceMatrix instance. Used only to compute a miss -- it is never part of the cache key.	required
X1	NDArray[floating]	First feature matrix.	required
X2	NDArray[floating]	Second feature matrix.	required
metric	str	Distance metric name.	required
batch_size	int | str	Batch size or mode.	'auto'
**kwargs	Any	Metric keyword arguments.	{}

Returns:

Type	Description
NDArray[floating]	Read-only distance matrix.

OversamplingValidator

Bases: BaseEstimator

Validate an oversampler, following the scikit-learn estimator contract.

Lower scores are better: the score is the hidden-majority error rate, so score returns its negation, matching scikit-learn's "greater is better" convention for scorers.

Parameters

oversampler : object An imbalanced-learn sampler. minority_label : int, optional Minority class. Inferred as the least frequent label when omitted. hidden_ratio : float, default=0.1 Fraction held out. reference : {"hidden_minority", "train_minority"}, default="hidden_minority" Which minority set to compare against. metric : str, default="hassanat" Distance metric. metric_params : dict, optional Extra keyword arguments for the metric. n_repeats : int, default=1 Independent hold-out splits. random_state : int, Generator, SeedSequence or None, default=42 Seeds the split.

Attributes

report_ : ValidationReport Set by :meth:fit. error_rate_ : float Set by :meth:fit.

Notes

The constructor stores its arguments unchanged and does no validation or computation, as scikit-learn requires -- get_params / set_params round-trip, and clone works. All checking happens in :meth:fit.

.. warning::

Cross-validation folds must be large enough to support the estimand. Scoring runs a full validation on each test fold, which holds out hidden_ratio of that fold's minority. With cv=3 on 136 minority points, a test fold has ~45 and a 10% hold-out leaves 4 — below min_hidden, so validation raises.

scikit-learn catches scorer exceptions and records nan, so this surfaces as an all-nan cv_results_ with no explanation. Pass error_score="raise" to see the real message. Either use fewer folds, supply more minority data, or lower min_hidden deliberately.

Examples

Tuning a sampler against synthetic-sample quality becomes two lines::

search = GridSearchCV(
    OversamplingValidator(SMOTE(random_state=0)),
    {"oversampler": [SMOTE(k_neighbors=k) for k in (3, 5, 9)]},
    scoring=validation_scorer,
)
search.fit(X, y)

fit(X, y)

Run validation and store the report.

Parameters:

Name	Type	Description	Default
X	NDArray[floating]	Feature matrix.	required
y	NDArray[integer]	Target labels.	required

Returns:

Type	Description
OversamplingValidator	self, so calls chain.

Raises:

Type	Description
ValidationError	If the inputs cannot support validation.

score(X=None, y=None)

Return the negated error rate, so greater is better.

Scikit-learn's convention is that a higher score is better, but a higher error rate is worse. Returning the raw rate would make GridSearchCV select the worst sampler, so it is negated here.

Parameters:

Name	Type	Description	Default
X	NDArray[floating] | None	Optional data to validate instead of the fitted run.	None
y	NDArray[integer] | None	Labels matching X.	None

Returns:

Type	Description
float	Negated error rate.

BoundaryReport dataclass

How often synthetic points land in majority territory.

to_dict()

Flat mapping for the reporting layer.

FidelityReport dataclass

Every fidelity signal for one oversampler on one dataset.

to_dict()

Flat mapping across every component.

to_frame()

Single-row frame, for concatenating across samplers.

interpret()

Readings of the patterns that matter, in plain language.

ManifoldMetrics dataclass

k-NN manifold estimates of fidelity and diversity.

Attributes

precision: Fraction of synthetic points inside the real manifold. Fidelity: are the generated points plausible? recall: Fraction of real points inside the synthetic manifold. Diversity: does the generator cover the real distribution? density: Like precision, but counts how many real k-NN spheres contain each synthetic point. Not saturated by a single real outlier whose sphere is enormous, which is precision's main failure mode. coverage: Fraction of real points with at least one synthetic point inside their own k-NN sphere. More robust than recall for the same reason.

Notes

Density and coverage are the more reliable pair (Naeem et al. 2020) and are what the report surfaces first. Precision and recall are reported too, because their disagreement with density/coverage is itself informative: it usually means an outlier is inflating one manifold.

to_dict()

Flat mapping for the reporting layer.

MemorisationReport dataclass

How much of the "synthetic" output is really copied training data.

Attributes

distance_ratio: The headline number. Median nearest-neighbour distance from synthetic points to their training set, divided by the median nearest-neighbour distance within the real minority. Below 1 means the generator sits closer to its training points than real points sit to each other -- it is copying. Near 0 means outright duplication. exact_duplicate_rate: Fraction of synthetic points exactly coinciding with a training point. near_duplicate_rates: Fraction within a threshold taken from the real minority's own nearest-neighbour distance distribution, keyed by quantile. Deriving the threshold from the data makes it scale-free: an absolute tolerance means something different on every dataset.

to_dict()

Flat mapping for the reporting layer.

interpret()

One-line reading of the headline ratio.

NullCalibration dataclass

Where an observed error rate sits against known reference points.

Attributes

observed: The error rate being interpreted. null_rates: Error rates from scoring real held-out minority points -- what an ideal generator, drawing from the true minority distribution, achieves. ceiling_rates: Error rates from deliberately bad points drawn from the majority region. The other end of the scale. z_score: (observed - null_mean) / null_sd. Positive means worse than ideal. nan when the null has no spread. percentile: Empirical percentile of observed within null_rates. scaled: Position on a 0-1 scale where 0 is the null mean and 1 the ceiling mean. Above 1 is worse than a deliberately bad generator.

null_mean property

Mean of the null distribution.

null_sd property

Standard deviation of the null distribution.

ceiling_mean property

Mean of the ceiling distribution.

null_interval(confidence=0.95)

Percentile interval of the null distribution.

interpret()

One-line reading of where the observed rate falls.

to_dict()

Flat mapping for the reporting layer.

TwoSampleTestResult dataclass

Outcome of a two-sample test between synthetic and real points.

A high p-value is weak evidence that the two samples are distributionally indistinguishable -- which is what good synthesis looks like. See the warning in :func:nn_two_sample_test about what failing to reject does not mean.

to_dict()

Flat mapping for the reporting layer.

MemoryEfficientValidator

Drop-in replacement for :func:validate_oversampling with memory safeguards.

validate_oversampling(X, y, minority_label, oversampler, hidden_ratio=0.1, metric='hassanat', metric_kwargs=None, return_details=False, *, reference='hidden_minority', minority_hidden_ratio=None, min_hidden=5, random_state=42, stratify_by=None)

Validate oversampling with streaming-aware distance calculations.

Uses the same estimand as :func:oversampleqa.validate_oversampling via the shared :func:~oversampleqa.validator.prepare_validation_split helper, so the two cannot drift apart.

Parameters:

Name	Type	Description	Default
X	NDArray[floating]	Feature matrix.	required
y	NDArray[integer]	Target labels.	required
minority_label	int	Minority class label.	required
oversampler	BaseOverSampler	Oversampler instance.	required
hidden_ratio	float	Fraction of majority to hide.	0.1
metric	str	Distance metric name.	'hassanat'
metric_kwargs	dict[str, Any] | None	Metric keyword arguments.	None
return_details	bool	Whether to return a ValidationDetails.	False
reference	ReferenceSet	Which minority set to compare against. See :func:oversampleqa.validate_oversampling.	'hidden_minority'
minority_hidden_ratio	float | None	Fraction of the minority to hide.	None
min_hidden	int	Minimum held-out minority points.	5

Returns:

Type	Description
float | ValidationDetails	Error rate, or ValidationDetails when return_details is True.

cleanup()

Remove temporary files created during streaming computations.

This cleans any memmap-backed temporary directories created during streaming validation.

OptimizedDistanceMatrix

Memory-aware distance matrix computation with vectorisation and batching.

.. note::

The effective memory limit is min(memory_limit_gb, available), where available comes from psutil. Without psutil installed it is assumed to be 1 GB, regardless of the machine, so batching is more conservative and throughput differs from an otherwise identical environment that has it. The fallback is logged once at INFO. Install the performance extra to get the real figure.

Parameters

cache_size : int, default=128 Retained for API compatibility. memory_limit_gb : float, default=4.0 Upper bound on the memory one computation may use. metric_registry : dict, optional Name-to-callable mapping of metrics. show_progress : bool, default=False Display a progress bar for large computations. progress_threshold : int, default=10000 Row count above which progress is shown. cache : ValidationCache, optional Opt-in cache. None means nothing is written to disk. safety_factor : float, default=0.8 Fraction of the limit a batched computation is allowed to plan against. The remainder is headroom for allocator overhead and transient copies, which the analytic estimate does not model.

compute_distance_matrix(X1, X2, metric='hassanat', batch_size='auto', **kwargs)

Compute pairwise distances with automatic optimisation.

Parameters

X1, X2: Input matrices of shape (n_samples, n_features). metric: Name of the distance metric registered in metric_registry. batch_size: "auto" selects the largest batch size fitting within memory_limit_gb. An integer enforces a specific chunk length. "stream" yields rows sequentially without storing the full matrix in memory. kwargs: Extra keyword arguments forwarded to the underlying metric.

estimate_memory_gb(n_rows, n_cols, dtype=None, n_features=1, metric='')

Public helper returning estimated peak footprint of a distance matrix.

Parameters:

Name	Type	Description	Default
n_rows	int	Number of rows.	required
n_cols	int	Number of columns.	required
dtype	dtype[Any] | None	Data type of the distance matrix.	None
n_features	int	Feature dimension.	1
metric	str	Metric name; selects the intermediate multiplier.	''

Returns:

Type	Description
float	Estimated memory usage in gigabytes.

AxiomReport dataclass

Which metric axioms a callable satisfied, and how it failed.

ok property

Whether every checked axiom held.

__bool__()

Truthy when every axiom held.

MetricPlugin

Bases: Protocol

A distance metric: two vectors in, one float out.

__call__(x1, x2, **kwargs)

Return the distance between x1 and x2.

RunMetadata dataclass

Everything needed to reproduce and audit a run.

A number without its provenance is not a result. This records the package and dependency versions, the sampler and its parameters, the seed, and a hash of the data -- so a report exported today can be checked against a rerun in a year, and a mismatch localised to whichever of those changed.

capture(X, y, oversampler, *, minority_label=None, metric='hassanat', hidden_ratio=0.1, reference='hidden_minority', random_state=None, n_repeats=1) classmethod

Collect metadata for a run about to happen, or just completed.

to_dict()

JSON-safe mapping.

from_dict(payload) classmethod

Rebuild from :meth:to_dict output, ignoring unknown keys.

ValidationReport dataclass

Everything known about one oversampler on one dataset.

calibration, inference and fidelity are optional because each costs real time: the calibration fits nothing but resamples repeatedly, the two-sample tests permute, and the fidelity suite can fit models. A report with only error_rate and details is the cheap default.

to_dict()

JSON-serialisable mapping of the whole report.

Non-finite floats become null; see :func:_json_safe.

from_dict(payload) classmethod

Rebuild from :meth:to_dict output.

Components come back as plain dicts rather than their original dataclasses: the export is the interchange format, and rehydrating each component type would couple this module to every one of them. Round trips are therefore compared on to_dict(), which is what a consumer actually reads.

to_json(indent=2)

Serialise to JSON. allow_nan=False guarantees valid output.

to_frame()

Tidy one-row frame with every scalar flattened.

__rich__()

Compact CLI rendering.

with_components(**components)

Return a copy carrying additional components.

PydanticValidationConfig

Bases: BaseModel

Runtime validation for configuration parameters.

validate_metric(value)

Validate that the metric is supported.

Accepts registered plugin metrics as well as built-ins. Checking the built-in table alone rejected a plugin metric at config construction -- before any validation ran -- even though distance_matrix would compute it.

Parameters:

Name	Type	Description	Default
value	str	Metric name.	required

Returns:

Type	Description
str	The validated metric name.

Raises:

Type	Description
ValueError	If the metric is neither built in nor registered.

validate_random_state(value)

Validate random_state bounds when provided.

Parameters:

Name	Type	Description	Default
value	int | None	Optional random state.	required

Returns:

Type	Description
int | None	The validated random state.

TypedValidator

Bases: BaseValidator[ValidationResult]

Type-safe validator wrapper with runtime validation.

validate(X, y, minority_label, oversampler, config=None, **kwargs)

validate(
    X: FloatArray,
    y: IntArray,
    minority_label: int,
    oversampler: OversamplerProtocol,
    config: ValidationConfig,
) -> ValidationResult

validate(
    X: FloatArray,
    y: IntArray,
    minority_label: int,
    oversampler: OversamplerProtocol,
    *,
    hidden_ratio: float = 0.1,
    metric: str = "hassanat",
    return_details: bool = False,
    random_state: int | None = None,
) -> ValidationResult

Validate oversampling with typed configuration.

Parameters:

Name	Type	Description	Default
X	FloatArray	Feature matrix.	required
y	IntArray	Target labels.	required
minority_label	int	Minority class label.	required
oversampler	OversamplerProtocol	Oversampler instance.	required
config	ValidationConfig | None	ValidationConfig, or None to build from kwargs.	None
**kwargs	Any	ValidationConfig fields when config is None.	{}

Returns:

Type	Description
ValidationResult	ValidationResult with error rate and optional details.

validate_async(X, y, minority_label, oversampler, config) async

Async wrapper around validate using an executor.

Parameters:

Name	Type	Description	Default
X	FloatArray	Feature matrix.	required
y	IntArray	Target labels.	required
minority_label	int	Minority class label.	required
oversampler	OversamplerProtocol	Oversampler instance.	required
config	ValidationConfig	ValidationConfig.	required

Returns:

Type	Description
ValidationResult	ValidationResult.

BenchmarkConfig dataclass

Configuration for benchmarking experiments.

ConfigurationError

Bases: OversampleQAError

Configuration is invalid, missing, or internally inconsistent.

Raised for bad parameter combinations and for lookups of things that were never registered.

MetricError

Bases: OversampleQAError

A distance metric could not be resolved or computed.

OversampleQAError

Bases: Exception

Base class for every error raised by OversampleQA.

ValidationConfig dataclass

Immutable validation configuration.

ValidationDetails dataclass

Detailed outcome of a single validation run.

Replaces the former (error_rate, n_errors, dist_hidden, dist_min) 4-tuple returned by return_details=True.

Attributes

error_rate: Fraction of synthetic points strictly closer to the hidden majority than to the minority reference set. nan when no synthetic samples were produced -- that is an absent measurement, not a perfect score. n_errors: Count behind error_rate. n_synthetic: Number of synthetic points scored. n_ties: Points exactly equidistant from both reference sets. Counted separately rather than scored as errors; a large value indicates duplicated or heavily quantised features. duplication_rate: Fraction of synthetic points coinciding with a reference point. A sampler that only duplicates scores 1.0, and its error rate carries no information about synthesis quality. reference: Which minority set the comparison used. dist_hidden, dist_min: Distance matrices from synthetic points to the hidden majority and to the minority reference set.

has_dispersion property

Whether more than one hold-out split was drawn.

to_dict()

Flat, JSON-safe mapping.

dist_hidden and dist_min are deliberately excluded: they are working arrays of shape (n_synthetic, n_reference), often megabytes, and they are inputs to the summary rather than part of it. Callers that need them have the dataclass.

ValidationError

Bases: OversampleQAError

A validation run could not produce a meaningful result.

Covers malformed input as well as data that cannot support the estimand -- for example a minority class too small to hold anything out of.

ValidationMode

Bases: Enum

Validation execution modes.

ValidationResult

Bases: TypedDict

Typed structure for validation result.

create_benchmark_report(results_df, output_path='benchmark_report.html')

Create a lightweight HTML report summarising benchmark statistics.

Parameters:

Name	Type	Description	Default
results_df	DataFrame	Benchmark results dataframe.	required
output_path	str	Output HTML path.	'benchmark_report.html'

Returns:

Type	Description
Path	Path to the generated report.

format_statistical_summary(results_df, significance_level=0.05)

Render a Markdown summary of a statistical benchmark frame.

The frame is expected to come from :meth:StatisticalBenchmark.run_comprehensive_benchmark. The summary lists, per dataset, the mean error, standard deviation and confidence interval for each oversampler/metric, followed by the statistically significant pairwise comparisons (corrected p-value below significance_level).

Parameters:

Name	Type	Description	Default
results_df	DataFrame	Benchmark results dataframe.	required
significance_level	float	Threshold below which a pairwise p-value is reported.	0.05

Returns:

Type	Description
str	A Markdown-formatted string.

compute_ranking(results)

Rank oversamplers within each experiment, then aggregate the ranks.

Error rates are not comparable across datasets, hold-out ratios or metrics: an easy dataset scores near 0.1 and a hard one near 0.9, and hassanat scores roughly twice euclidean on the same data. Pooling them and taking a mean asks a question with no answer.

Ranking within each (dataset, hidden_ratio, metric) and averaging those ranks is the Demsar (2006) protocol, and the same logic underlying :func:~oversampleqa.inference.friedman_nemenyi -- so the ranking here and the significance test there answer the same question.

Parameters:

Name	Type	Description	Default
results	DataFrame	Long-format benchmark frame from :func:run_benchmark.	required

Returns:

Type	Description
DataFrame	Summary indexed by oversampler with mean_rank (lower is better),
DataFrame	rank, n_specifications, and the pooled mean, std and
DataFrame	n_missing retained for reference.

Warns:

Type	Description
UserWarning	If oversamplers were ranked over different numbers of experiments. Mean ranks computed over different sets are not comparable, and the imbalance is usually caused by skipped runs.

Notes

Averaging the raw error rate was not merely imprecise, it inverted results. Given a sampler that beats another on every dataset while having more of its runs skipped on the hard one, the pooled mean favours the loser -- Simpson's paradox, reachable here because the hold-out guards legitimately drop runs.

nan runs are excluded rather than counted as zero, and the count is reported in n_missing.

export_benchmark_results(results, output_path, fmt='csv')

Export benchmark summary to CSV, JSON or Markdown.

Parameters:

Name	Type	Description	Default
results	DataFrame	Benchmark results dataframe.	required
output_path	str	Destination path.	required
fmt	str	Output format: csv, json, markdown or html. All four render the same ranking frame.	'csv'

Raises:

Type	Description
ValueError	If fmt is not one of the four.

load_standard_datasets(include_openml=False)

Return a list of simple synthetic datasets for benchmarking.

Parameters

include_openml: Whether to attempt downloading additional datasets from OpenML. The default is False to avoid slow network calls during tests.

Returns

list of dict Each entry contains name, data, target, minority_label and provenance keys. The provenance value is a dict describing the dataset's source, generator, params, url, license and notes.

run_benchmark(datasets, oversamplers, hidden_ratios=None, n_runs=10, distance_metric='hassanat', random_state=None)

Run validation across datasets and oversampling methods.

Parameters:

Name	Type	Description	Default
datasets	list[dict]	Dataset descriptors containing data and target.	required
oversamplers	list	Oversampler instances.	required
hidden_ratios	list[float] | None	Hidden ratios to evaluate.	None
n_runs	int	Number of repetitions per configuration.	10
distance_metric	str	Distance metric name.	'hassanat'
random_state	RandomStateLike	RNG seed for reproducibility.	None

Returns:

Type	Description
DataFrame	DataFrame with per-run error rates.

cli_main()

Run the CLI validation workflow.

This entry point loads the dataset, configures the oversampler, runs the validation, and optionally writes a report or plot.

cluster_based_diagnostics(majority, synthetic, n_clusters=5, algorithm='kmeans', eps=0.5, min_samples=5, random_state=None)

Flag synthetic samples that fall in majority-dominated clusters.

Parameters

majority, synthetic : ndarray Arrays of majority and synthetic samples with shape (n_samples, n_features). n_clusters : int, default=5 Number of clusters for the k-means algorithm. algorithm : {"kmeans", "dbscan"}, default="kmeans" Clustering algorithm to use. eps : float, default=0.5 Neighborhood radius when using DBSCAN. min_samples : int, default=5 Minimum samples per cluster for DBSCAN. random_state : int, optional Random state for k-means.

Returns

flagged : ndarray of bool Boolean mask indicating which synthetic samples are located in clusters dominated by majority data. overlap_score : float Silhouette score of the clustering which acts as a crude overlap metric.

deprecated(*, removal_version, replacement=None, reason=None, category=DeprecationWarning)

Mark a function, method or class as deprecated.

The emitted warning names the replacement and the removal version, which is what :doc:/api_stability promises and what a caller needs in order to act. A note is appended to the docstring so the deprecation is visible in the rendered documentation as well as at runtime.

The warning is raised with stacklevel pointing at the caller, not at this wrapper. This matters more than it looks: Python's default filters hide DeprecationWarning unless it originates in __main__, and per-module filters key on the reported location. A warning that reports itself as coming from inside oversampleqa is invisible to exactly the people who need to see it.

Parameters:

Name	Type	Description	Default
removal_version	str	Release in which the name disappears, e.g. "0.6.0". Required -- a deprecation without a deadline is a permanent warning.	required
replacement	str | None	What to use instead, if there is a direct successor.	None
reason	str | None	Extra context appended to the message, for cases where the replacement is not a simple substitution.	None
category	type[Warning]	Warning class. Defaults to DeprecationWarning. Use FutureWarning when the change alters results rather than spelling, since that one is shown to end users by default.	DeprecationWarning

Returns:

Type	Description
Callable[[F], F]	A decorator that wraps the target, preserving its metadata.

Example

@deprecated(removal_version="0.6.0", replacement="new_name") ... def old_name() -> int: ... return 1 import warnings with warnings.catch_warnings(record=True) as caught: ... warnings.simplefilter("always") ... old_name() ... str(caught[0].message) 1 'old_name is deprecated and will be removed in 0.6.0. Use new_name instead.'

braycurtis_distance(x1, x2)

Compute Bray-Curtis distance between two vectors.

Often used in ecology and environmental science.

canberra_distance(x1, x2)

Compute Canberra distance between two vectors.

Canberra distance is a weighted version of Manhattan distance, useful when dealing with features of different scales.

chebyshev_distance(x1, x2)

Compute Chebyshev (L-infinity) distance between two vectors.

This is the maximum absolute difference across all dimensions.

correlation_distance(x1, x2)

Compute correlation distance between two vectors.

Correlation distance = 1 - Pearson correlation coefficient

distance_matrix(X1, X2, metric='hassanat', *, batch_size='auto', cache=None, **metric_kwargs)

Compute pairwise distance matrix using the given metric.

Parameters

X1, X2 : ndarray Input matrices containing observations. metric : str, default="hassanat" Identifier of the distance metric to use. batch_size : int or {"auto", "stream"}, default="auto" Controls batching strategy. "auto" selects a batch size that fits memory_limit_gb of :class:OptimizedDistanceMatrix. "stream" forces row-wise streaming when memory is constrained. cache : ValidationCache, optional Opt-in cache. Caching is off by default: nothing is written to disk and no directory is created unless you supply one. Worth it for expensive metrics such as hassanat; a net loss for euclidean, where hashing the inputs costs more than recomputing the result. **metric_kwargs : Additional keyword arguments are forwarded to the metric function. This enables configuration of metrics that require extra parameters, such as the inverse covariance matrix for Mahalanobis distance.

Returns

ndarray Distance matrix. When cache is supplied the array is read-only; call .copy() before modifying it.

energy_distance(x1, x2)

Compute energy distance between two 1D or 2D vectors.

The implementation follows the definition from energy statistics.

.. warning::

This is a sample-based metric, not a point metric. A 1-D input is reshaped to (len(x), 1) and treated as a set of scalar observations, not as one point in len(x)-dimensional feature space. It therefore does not measure the same kind of quantity as euclidean or hassanat, even though it is reachable through the same registry. Use it to compare two samples, not two points.

hamming_distance(x1, x2)

Compute Hamming distance between two vectors.

Counts the number of positions where elements differ. Useful for categorical or binary features.

hassanat_distance(x1, x2)

Compute the Hassanat distance between two vectors.

For each dimension :math:i, with :math:m = \min(a_i, b_i) and :math:M = \max(a_i, b_i):

.. math::

D(a_i, b_i) = \begin{cases} 1 - \dfrac{1 + m}{1 + M} & m \ge 0 \[2ex] 1 - \dfrac{1 + m + |m|}{1 + M + |m|} & m < 0 \end{cases}

and :math:HD(a, b) = \sum_i D(a_i, b_i).

Every per-dimension term lies in :math:[0, 1), which is what makes the metric invariant to feature scale and robust to outliers: no single dimension can contribute more than 1 regardless of its magnitude.

Parameters

x1, x2 : NDArray[np.floating] Input vectors of identical shape.

Returns

float Hassanat distance, in [0, n_features).

Raises

ValueError If the two vectors do not have the same shape.

References

Hassanat, A. B. (2014). Dimensionality invariant similarity measure. Journal of American Science, 10(8).

hellinger_distance(x1, x2)

Compute the Hellinger distance between two probability vectors.

The input vectors are normalized to sum to 1 and must contain non-negative values. The distance is bounded between 0 and 1.

jaccard_distance(x1, x2)

Compute Jaccard distance between two binary vectors.

Jaccard distance = 1 - Jaccard similarity where Jaccard similarity = :math:|intersection| / |union|

jensen_shannon_distance(x1, x2)

Compute the Jensen-Shannon distance between two probability vectors.

The Jensen-Shannon distance is the square root of the Jensen-Shannon divergence and is symmetric and bounded between 0 and sqrt(log(2)) when using natural logarithms.

mahalanobis_distance(x1, x2, cov_inv=None)

Compute Mahalanobis distance between two vectors.

Parameters

x1, x2 : np.ndarray Input vectors cov_inv : np.ndarray Inverse covariance matrix. Required, and must be symmetric positive semi-definite -- that is what makes the result a distance. It is not validated as such on every call, because an eigenvalue check per pair would cost more than the distance itself; a negative squared distance is caught instead, which is how a non-PSD matrix usually shows up.

Note the residual case: a matrix that is not PSD can still return 0
for two distinct points, and no per-pair check can detect that. If you
build ``cov_inv`` by any route other than inverting a sample
covariance, check it once with ``np.linalg.eigvalsh``.

Returns

float Mahalanobis distance

Raises

ValueError If cov_inv is omitted, or if it yields a negative squared distance.

minkowski_distance(x1, x2, p=3.0)

Compute Minkowski distance between two vectors.

Parameters

x1, x2 : np.ndarray Input vectors of same shape p : float, default=3.0 Order of the norm (p >= 1). np.inf is accepted and gives the Chebyshev distance, which is the limit as p grows.

Returns

float Minkowski distance

Raises

ValueError If the shapes differ, or p < 1.

wasserstein_1d_distance(x1, x2)

Compute the 1D Wasserstein distance between two empirical distributions.

.. warning::

This is a sample-based metric, not a point metric. The input vector is flattened and treated as a set of scalar observations drawn from a distribution, not as one point in feature space. It therefore does not measure the same kind of quantity as euclidean or hassanat, even though it is reachable through the same registry. Use it to compare two samples, not two points.

Parameters:

Name	Type	Description	Default
x1	NDArray[floating]	Samples from distribution 1.	required
x2	NDArray[floating]	Samples from distribution 2.	required

Returns:

Type	Description
float	Wasserstein distance.

validation_scorer(estimator, X, y)

Scorer callable for cross_validate and GridSearchCV.

Follows the scorer(estimator, X, y) signature and the greater-is-better convention, so it can be passed directly as scoring=.

boundary_violation_rate(synthetic, X_real, y_real, minority_label, *, k=5, metric='hassanat', metric_kwargs=None)

Fraction of synthetic points sitting in majority territory.

Measures the failure this package exists to detect, per point and without a hold-out -- so it can still be reported when the minority is too small for :func:~oversampleqa.validate_oversampling's hold-out guard.

Two versions are returned because they answer different questions:

strict_rate Fraction whose all k nearest real neighbours are majority. Unambiguous violations. graded_rate Mean majority fraction among the k neighbours. Sensitive to points drifting toward the boundary before they cross it.

This is unrelated to :func:~oversampleqa.noise_sensitivity_diagnostic, which measures how the error rate responds to injected label noise -- a different question, so the two do not overlap.

Returns

BoundaryReport

fidelity_report(X, y, minority_label, oversampler, *, metric='hassanat', k=5, hidden_ratio=0.1, random_state=42, include_utility=False)

Run the full fidelity suite for one oversampler.

Parameters

X, y : ndarray Full dataset. minority_label : int Minority class label. oversampler : object An imbalanced-learn sampler. metric : str, default="hassanat" Distance metric for every geometric measure. k : int, default=5 Neighbours for the manifold and boundary estimates. hidden_ratio : float, default=0.1 Fraction held out, matching validate_oversampling. include_utility : bool, default=False Fit models to measure downstream gain. Off by default because it is far slower than the geometric measures.

Returns

FidelityReport

memorisation_report(synthetic, train_minority, *, metric='hassanat', metric_kwargs=None, quantiles=(0.01, 0.05))

Assess how much of the output is copied from the training minority.

The headline is distance_ratio: the median distance from a synthetic point to its nearest training point, over the median nearest-neighbour distance within the real minority. That denominator is what makes the number legible -- it is the natural spacing of real data, so a ratio well below 1 says the generator sits closer to its training points than real points sit to each other.

Near-duplicate thresholds come from the same distribution rather than an absolute tolerance, so they mean the same thing on any dataset.

Parameters

synthetic, train_minority : ndarray Synthetic points and the minority data the sampler was fitted on. quantiles : tuple of float, default=(0.01, 0.05) Quantiles of the real nearest-neighbour distance distribution to use as near-duplicate thresholds.

Returns

MemorisationReport

precision_recall_density_coverage(synthetic, real, *, k=5, metric='hassanat', metric_kwargs=None)

Estimate fidelity and diversity from k-NN manifolds.

The real manifold is the union of hyperspheres centred on each real point with radius its k-th nearest neighbour distance; the synthetic manifold is the same construction on synthetic points.

Parameters

synthetic, real : ndarray Synthetic points and real held-out minority points. k : int, default=5 Neighbours defining each sphere. These metrics are sensitive to k; use :func:sweep_k rather than trusting one value. metric : str, default="hassanat" Any metric from the package registry.

Returns

ManifoldMetrics

Raises

ValidationError If synthetic is empty or real has fewer than k + 1 points.

cross_match_test(synthetic, real, *, metric='hassanat', metric_kwargs=None, n_permutations=999, parents=None, n_subsamples=9, random_state=42)

Rosenbaum cross-match test, with a greedy matching.

Pair up the pooled sample and count how many pairs join the two samples. Well-mixed samples yield many cross pairs, so the p-value is left-tailed.

.. note::

Rosenbaum's test uses optimal non-bipartite matching, which minimises total matched distance and admits an exact null distribution. This implementation uses a greedy nearest-available matching instead, so the exact distribution does not apply and the p-value comes from permutation. The greedy statistic is generally close but not identical; treat it as an approximation to the published test rather than the test itself.

Returns

TwoSampleTestResult

mst_two_sample_test(synthetic, real, *, metric='hassanat', metric_kwargs=None, n_permutations=999, parents=None, n_subsamples=9, random_state=42)

Friedman-Rafsky minimum-spanning-tree two-sample test.

Build the MST on the pooled sample and count edges joining the two samples. Well-mixed samples produce many cross edges; separated ones produce few, so small counts are evidence against equality and the p-value is left-tailed.

The same power caveat as :func:nn_two_sample_test applies.

Returns

TwoSampleTestResult

nn_two_sample_test(synthetic, real, *, k=3, metric='hassanat', metric_kwargs=None, n_permutations=999, parents=None, n_subsamples=9, random_state=42)

Schilling-Henze nearest-neighbour two-sample test.

Of the k nearest neighbours of each point in the pooled sample, count how many share its sample label. If the two samples come from the same distribution, neighbours are labelled roughly at the base rate; if they are separated, points cluster with their own kind and the count rises.

Applied to synthetic points against held-out real minority points, this tests the question a user actually has: are these synthetic points distributionally indistinguishable from real ones? A high p-value is evidence of good synthesis.

.. warning::

Failing to reject is not proof of equality. The power of every nearest-neighbour test collapses as dimension grows, so on high-dimensional data a large p-value may reflect a lack of power rather than genuine similarity. Always read it next to n_synthetic and n_real, which are returned for exactly this reason.

Parameters

synthetic, real : ndarray The two samples. k : int, default=3 Neighbours considered per point. metric : str, default="hassanat" Any metric from the package registry, so hassanat composes with the inferential layer. n_permutations : int, default=999 Permutations behind the p-value. The pooled distance matrix is computed once and reused; permutations only relabel. random_state : int, Generator, SeedSequence or None, default=42 Seeds the permutations.

Returns

TwoSampleTestResult Carries both the permutation p-value and the asymptotic normal approximation, so the user can see where they disagree.

null_error_rate(X, y, minority_label, observed, *, hidden_ratio=0.1, metric='hassanat', metric_kwargs=None, n_draws=200, min_hidden=5, random_state=42)

Calibrate an observed error rate against ideal and worst-case references.

The null is built by scoring real held-out minority points through the identical pipeline. Those points are, by construction, drawn from the true minority distribution, so their error rate is what a perfect generator would score. Anything an actual oversampler achieves can then be read as a position relative to that.

The ceiling uses points drawn from the majority region -- what a deliberately bad generator produces -- bounding the other end of the scale.

Parameters

X, y : ndarray Input data and labels. minority_label : int Label of the minority class. observed : float The error rate to interpret, e.g. from :func:~oversampleqa.validate_oversampling. hidden_ratio : float, default=0.1 Fraction held out. Must match the run that produced observed, or the comparison is meaningless. metric : str, default="hassanat" Distance metric. Must also match. n_draws : int, default=200 Independent splits behind the null distribution. min_hidden : int, default=5 Minimum held-out minority points per draw. random_state : int, Generator, SeedSequence or None, default=42 Seeds the draws.

Returns

NullCalibration

Raises

ValidationError If the labels are not binary or the minority is too small.

Notes

hidden_ratio and metric must match the run that produced observed. The error rate's scale depends on both, so calibrating against a null computed with different settings compares two different quantities.

calculate_error_rate(errors, total)

Return error rate given the number of errors and total samples.

Parameters:

Name	Type	Description	Default
errors	int	Number of error samples.	required
total	int	Total number of samples.	required

Returns:

Type	Description
float	Error rate in the range [0, 1], or nan when total is zero.

Notes

A zero denominator means nothing was measured. Returning 0.0 in that case would be indistinguishable from a perfect score, so nan is returned instead. Callers that aggregate error rates must use nan-aware reductions (np.nanmean) deliberately.

check_model_fairness(y_true, y_pred, protected_attr, minority_label)

Return absolute difference in minority recall across protected groups.

Parameters:

Name	Type	Description	Default
y_true	NDArray[Any]	True labels.	required
y_pred	NDArray[Any]	Predicted labels.	required
protected_attr	NDArray[Any]	Protected group labels.	required
minority_label	int	Minority class label.	required

Returns:

Type	Description
float	Absolute recall gap between the two groups.

confidence_ratio(dist_min, dist_maj)

Return ratio between distances to minority and majority classes.

Parameters:

Name	Type	Description	Default
dist_min	float	Distance to minority class.	required
dist_maj	float	Distance to majority class.	required

Returns:

Type	Description
float	Ratio dist_min / dist_maj (inf if dist_maj is zero).

duplication_rate(synthetic, reference, *, atol=0.0)

Fraction of synthetic points that coincide with a reference point.

Parameters

synthetic : ndarray Synthetic samples of shape (n_synthetic, n_features). reference : ndarray Real samples the synthetic points may have been copied from. atol : float, default=0.0 Absolute tolerance for treating a synthetic point as a duplicate. The default of 0.0 requires exact equality.

Returns

float Value in [0, 1]; nan when there are no synthetic samples.

Notes

An oversampler that duplicates rather than synthesises -- such as RandomOverSampler -- scores 1.0. Its validation error rate is then uninformative about synthesis quality, because every "synthetic" point sits exactly on top of a real one.

local_density_divergence(synthetic_samples, reference_samples, k=5)

Compute divergence of local densities between synthetic and reference data.

This metric compares the average distance to the k nearest neighbours for synthetic samples against the same statistic computed on the reference samples themselves. A higher value indicates that synthetic samples reside in sparser regions of the space compared to the reference distribution.

Parameters

synthetic_samples, reference_samples : ndarray Arrays of shape (n_samples, n_features) representing synthetic and reference data respectively. k : int, default=5 Number of nearest neighbours to consider when estimating local density.

Returns

float Relative difference in mean neighbourhood radii. 0.0 indicates that both sets have similar local density.

minority_recall_loss(y_true, y_pred, minority_label)

Return recall loss for the minority class.

Parameters

y_true, y_pred : ndarray True and predicted class labels. minority_label : int Label of the minority class.

Returns

float 1 - recall for the minority class.

noise_sensitivity_diagnostic(X, y, minority_label, oversampler, noise_levels=None, hidden_ratio=0.1, metric='hassanat', random_state=None)

Evaluate error rate under different label noise levels.

Parameters:

Name	Type	Description	Default
X	NDArray[floating]	Feature matrix.	required
y	NDArray[Any]	Target labels.	required
minority_label	int	Minority class label.	required
oversampler	Any	Oversampler instance.	required
noise_levels	list[float] | None	Noise levels to evaluate.	None
hidden_ratio	float	Fraction of majority to hide.	0.1
metric	str	Distance metric name.	'hassanat'
random_state	int | None	Optional random seed.	None

Returns:

Type	Description
DataFrame	DataFrame with noise, error_rate and n_flipped -- the number
DataFrame	of labels actually changed, so the applied noise can be checked against
DataFrame	the requested level rather than assumed.

Raises:

Type	Description
ValueError	If y contains fewer than two classes, leaving no label to flip to.

Notes

Replacement labels are drawn from the other classes. Drawing from all classes, as this used to, lets a selected point keep its own label, so the realised noise was noise * (k - 1) / k: on binary data -- this package's main case -- half the requested level. A run labelled noise=0.3 applied about 0.15, and the x-axis of every noise-sensitivity plot was overstated by that factor.

umap_manifold_distance(real, synthetic, n_neighbors=15, random_state=None)

Return Wasserstein distance between real and synthetic data in UMAP space.

Parameters:

Name	Type	Description	Default
real	NDArray[floating]	Real samples.	required
synthetic	NDArray[floating]	Synthetic samples.	required
n_neighbors	int	UMAP neighborhood size.	15
random_state	int | None	Optional random seed.	None

Returns:

Type	Description
float	Mean Wasserstein distance across UMAP dimensions.

plot_class_balance(labels_before, labels_after, save_path=None)

Bar chart comparing class counts before and after oversampling.

Parameters

labels_before, labels_after : ndarray Class labels prior to oversampling and after applying an oversampler. save_path : str, optional If given, path to save the resulting plot. Otherwise the figure is closed and not displayed.

plot_distance_histogram(dist_hidden, dist_minority, save_path=None)

Histogram of nearest distances to hidden majority and real minority samples.

Parameters

dist_hidden, dist_minority : ndarray Distance matrices where rows correspond to synthetic samples and columns to hidden majority or real minority samples respectively. save_path : str, optional If given, path to save the resulting plot. Otherwise the figure is closed and not displayed.

plot_error_boxplot(benchmark_results, save_path=None)

Boxplot of error rates for each oversampler.

Parameters:

Name	Type	Description	Default
benchmark_results	DataFrame	Benchmark results dataframe.	required
save_path	str | None	Optional output image path.	None

plot_error_comparison(benchmark_results, save_path=None)

Bar plot showing mean error rates for each oversampler.

Parameters:

Name	Type	Description	Default
benchmark_results	DataFrame	Benchmark results dataframe.	required
save_path	str | None	Optional output image path.	None

plot_error_heatmap(error_matrix, class_labels=None, save_path=None)

Plot heatmap of a multi-class error attribution matrix.

Parameters

error_matrix : ndarray Matrix where matrix[i, j] counts synthetic samples generated for class i that are closest to hidden samples from class j. class_labels : list of int, optional Labels for the classes corresponding to the rows/columns of the matrix. If not provided, integer indices are used. save_path : str, optional If given, path to save the resulting plot. Otherwise the figure is closed and not displayed.

plot_error_ranking(benchmark_results, save_path=None)

Line chart of mean error rate ranked by oversampler.

Parameters:

Name	Type	Description	Default
benchmark_results	DataFrame	Benchmark results dataframe.	required
save_path	str | None	Optional output image path.	None

plot_noise_sensitivity(results, save_path=None)

Line plot showing error rate as label noise increases.

Parameters

results : DataFrame Output of :func:oversampleqa.metrics.noise_sensitivity_diagnostic, expected to contain noise and error_rate columns. save_path : str, optional If given, path to save the resulting plot. Otherwise the figure is closed and not displayed.

plot_sample_distribution(majority, minority, synthetic, hidden_majority=None, method='pca', save_path=None)

Visualize sample distribution using PCA or UMAP.

Parameters

majority, minority, synthetic : ndarray Arrays of majority, minority and synthetic samples. hidden_majority : ndarray, optional Hidden majority samples for reference. method : {{"pca", "umap"}}, default="pca" Dimensionality reduction method to use. save_path : str, optional If given, path to save the resulting plot. Otherwise the figure is closed and not displayed.

check_metric_axioms(func, name='metric', *, domain='real', n_trials=50, n_features=4, tolerance=1e-09, random_state=0, **metric_kwargs)

Check that a callable behaves like a distance metric.

Checks, on random vectors:

identity d(x, x) == 0. identity_of_indiscernibles d(x, y) > 0 whenever x != y. This is the check the built-in Hassanat implementation failed -- it scored [-5] against [5] as zero, because it compared absolute values. symmetry d(x, y) == d(y, x). non_negativity d(x, y) >= 0. finiteness No nan or inf on ordinary input.

The triangle inequality is deliberately not checked: several useful registry entries are genuine semi-metrics, so requiring it would reject metrics the package intends to support. Identity of indiscernibles is the one whose violation makes a metric silently meaningless.

Parameters:

Name	Type	Description	Default
func	Any	Candidate metric.	required
name	str	Name used in failure messages.	'metric'
domain	MetricDomain	Input the metric is defined on. "sample" metrics compare distributions rather than points, so the point-metric axioms are skipped for them. See :data:METRIC_DOMAINS.	'real'
n_trials	int	Random vector pairs to test.	50
n_features	int	Dimension of the test vectors.	4
tolerance	float	Numerical slack.	1e-09
random_state	int	Seed, so failures reproduce.	0
**metric_kwargs	Any	Extra arguments forwarded to the metric.	{}

Returns:

Type	Description
AxiomReport	AxiomReport, falsy when any axiom failed.

register_metric(name)

Decorator to register a metric plugin by name.

Parameters:

Name	Type	Description	Default
name	str	Metric identifier.	required

register_validator(name)

Decorator to register a validator plugin by name.

Parameters:

Name	Type	Description	Default
name	str	Validator identifier.	required

generate_report(benchmark_results, output_format='markdown', output_path=None, include_plots=True, fidelity_reports=None)

Generate a report from benchmark results.

Parameters:

Name	Type	Description	Default
benchmark_results	DataFrame	Benchmark results dataframe.	required
output_format	str	Output format (markdown or html).	'markdown'
output_path	str | None	Optional output file path.	None
include_plots	bool	Whether to include plot artifacts.	True
fidelity_reports	dict[str, Any] | None	Optional mapping of oversampler name to :class:~oversampleqa.fidelity.FidelityReport. When given, a fidelity section is appended covering the axis the error rate cannot express.	None

Returns:

Type	Description
str	Rendered report content as a string.

Raises:

Type	Description
ValueError	If output_format is not recognised.

evaluate_surrogate_models(X, y, minority_label, oversampler, model, test_size=0.3, random_state=None)

Evaluate model performance with and without synthetic data.

The function trains the provided model under three scenarios:

1. real_only – using the original training data without oversampling.
2. real_plus_synth – using the oversampled training data.
3. synth_only – replacing the real minority samples with the synthetic samples generated by the oversampler.

Parameters

X, y : ndarray Input data and labels. minority_label : int Label of the minority class. oversampler : imblearn BaseOverSampler Oversampler instance used to generate synthetic samples. model : sklearn estimator Classifier implementing fit/predict. test_size : float, default=0.3 Fraction of the dataset reserved for testing. random_state : int, optional Random seed for the split.

Returns

dict Mapping of scenario names to dictionaries with f1, recall and precision scores.

validation_session(config) async

Async context manager that yields a TypedValidator.

Parameters:

Name	Type	Description	Default
config	ValidationConfig	ValidationConfig (reserved for future use).	required

Yields:

Type	Description
AsyncIterator[TypedValidator]	TypedValidator instance.

extract_synthetic_samples(X_original, X_resampled, y_resampled, minority_label)

Return synthetic minority samples from a resampled dataset.

Parameters

X_original : ndarray Original feature matrix used for fitting the oversampler. X_resampled : ndarray Feature matrix returned by oversampler.fit_resample. y_resampled : ndarray Corresponding labels for X_resampled. minority_label : int Label of the minority class that was oversampled.

Returns

ndarray Array containing only the synthetic minority samples.

Raises

ValueError If the oversampler did not preserve the original samples as a prefix of its output, so synthetic rows cannot be identified positionally.

Notes

Synthetic samples are identified positionally: everything after the original rows is assumed to be new. That holds for the SMOTE family and RandomOverSampler, which append. It does not hold for combined over/under-samplers such as SMOTEENN and SMOTETomek, which delete original rows. A length check alone does not catch this -- SMOTEENN can still return more rows than it was given while having removed some of the originals -- so the prefix is compared element-wise.

validate_multiclass_oversampling(X, y, oversampler, hidden_ratio=0.1, metric='hassanat', metric_kwargs=None, return_matrix=False, *, min_hidden=5, random_state=42)

Validate oversampling for multi-class datasets.

For each class label, a portion of samples is hidden and the remainder is used for training the oversampler along with all other visible classes. The nearest hidden class to each synthetic sample determines the error attribution. The function returns the per-class error rates and optionally the full error matrix where matrix[i, j] counts how many synthetic samples generated for class i are closest to hidden samples from class j.

Parameters

X, y : ndarray Input data and labels. oversampler : BaseOverSampler Instance of an imbalanced-learn oversampler supporting multi-class data. hidden_ratio : float, default=0.1 Fraction of each class to hide during validation. metric : str, default="hassanat" Distance metric to use. metric_kwargs : dict, optional Additional keyword arguments passed to :func:distance_matrix. return_matrix : bool, default=False If True also return the error matrix. min_hidden : int, default=5 Minimum held-out points per class. Every class is a reference for every other, so a class that cannot supply a usable hidden set makes attribution unreliable for all of them. random_state : int, Generator, SeedSequence or None, default=42 Seeds the per-class hold-out. Defaults to 42, which reproduces previously documented numbers.

Returns

dict or tuple Mapping of class_label -> error_rate. If return_matrix is True the second element is the error matrix.

A class for which the sampler generated no synthetic points maps to
``nan``, not ``0.0``: nothing was measured, and ``0.0`` is the score of
a perfect result. Use :func:`macro_error_rate` to summarise, or
``np.nanmean`` over the values -- a plain mean propagates the ``nan``.

Raises

ValueError If hidden_ratio is out of range, if any class would hide fewer than min_hidden points, or if the oversampler did not preserve the original rows as a prefix of its output.

Notes

A synthetic point equidistant from its own class and another is attributed to its own class, matching :func:score_nearest_distances. Ties previously went to whichever class appeared first in label order, which biased results toward low-numbered classes for reasons unrelated to the data.

validate_oversampling(X, y, minority_label, oversampler, hidden_ratio=0.1, metric='hassanat', metric_kwargs=None, return_details=False, *, reference='hidden_minority', minority_hidden_ratio=None, min_hidden=5, duplication_warn_threshold=0.5, random_state=42, stratify_by=None, n_repeats=1, reseed_oversampler=False)

Validate oversampling using the hidden majority approach.

A fraction of the majority class is held out. The oversampler is fitted on what remains, and each synthetic minority point is scored by comparing its nearest-neighbour distance to the hidden majority against its nearest-neighbour distance to a minority reference set. A synthetic point strictly closer to the hidden majority counts as an error.

Parameters

X, y : ndarray Input data and labels. minority_label : int Label of the minority class. oversampler : BaseOverSampler Instance of an imbalanced-learn oversampler. hidden_ratio : float, default=0.1 Fraction of majority samples to hide during validation. metric : str, default="hassanat" Distance metric to use. metric_kwargs : dict, optional Additional keyword arguments passed to :func:distance_matrix. return_details : bool, default=False If True return a :class:~oversampleqa.types.ValidationDetails instead of the bare error rate. reference : {"hidden_minority", "train_minority"}, default="hidden_minority" Which minority set the synthetic points are compared against.

``"hidden_minority"`` also holds out part of the minority class and
compares against those held-out points. Both sides of the comparison
are then unseen, which is the same estimand
:func:`validate_multiclass_oversampling` uses.

``"train_minority"`` compares against the full minority class -- the
very data the oversampler interpolated from. This is the historical
behaviour, retained so old numbers can be reproduced. It biases the
result toward "no error" by an amount that depends on minority
density rather than on oversampler quality, and emits a
``FutureWarning``.

minority_hidden_ratio : float, optional Fraction of the minority class to hide when reference="hidden_minority". Defaults to hidden_ratio. min_hidden : int, default=5 Minimum number of held-out minority points required. A nearest-neighbour comparison against fewer than a handful of points is not meaningful, so this raises rather than warns. duplication_warn_threshold : float, default=0.5 Emit a UserWarning when this fraction of synthetic points coincide with a real point. random_state : int, Generator, SeedSequence or None, default=42 Seeds the hold-out split. Which points get hidden is the single largest driver of the error rate, so varying this varies the result. The default of 42 reproduces previously documented numbers; None draws fresh entropy and is not reproducible. stratify_by : ndarray, optional Group labels aligned with y. When given, the majority hold-out takes hidden_ratio within each group, so a hold-out cannot miss a cluster entirely. Strata are never inferred automatically. n_repeats : int, default=1 Number of independent hold-out splits. Above 1, the returned details carry the per-repeat vector and its dispersion. Repeat streams are spawned from a SeedSequence rather than derived as seed + i, which would correlate them. reseed_oversampler : bool, default=False Give the oversampler a fresh seed on each repeat. This changes what the dispersion covers -- see Notes.

Returns

float or ValidationDetails Error rate by default, nan if no synthetic samples were produced. With n_repeats > 1 the bare return is the mean across repeats.

Raises

ValueError If the labels are not binary, if minority_label is absent, if the held-out minority would be smaller than min_hidden, or if the oversampler does not preserve the original samples as a prefix.

Notes

The error rate is a relative quantity. Its scale depends on hidden_ratio, on the density of the data, and on dimensionality, so values are not comparable across datasets. See :doc:/concepts.

What the repeat interval covers. With n_repeats > 1 the reported interval is a percentile bootstrap over the per-repeat error rates. It describes the variability of the hold-out split, conditional on this dataset and on the oversampler's own seed. It is not a confidence interval for a population quantity, and with reseed_oversampler=False it does not include the oversampler's own randomness at all. Setting reseed_oversampler=True clones the sampler with a fresh seed per repeat, so the dispersion then covers both sources together -- a different, wider decomposition.

Synthetic points generated from shared parent points are not independent, so a binomial interval on the error rate would be too narrow. Nothing here claims more than the repeat-level bootstrap supports.

Module Reference

• Package
• Advanced benchmark
• Benchmark
• Caching
• CLI
• Enhanced CLI
• Clustering
• Config templates
• Distance
• Extended distances
• Memory efficient validator
• Metrics
• Optimized distance
• Plotting
• Plugin system
• Report
• Surrogate
• Typed validator
• Types
• Validator