Benchmarks API¶

Dataset catalog¶

anomalybench.benchmarks.catalog ¶

Dataset catalog metadata and selection utilities.

DatasetSpec ¶

Bases: TypedDict

Structured dataset entry returned by load_all_datasets.

get_dataset_functions `cached` ¶

get_dataset_functions() -> DatasetRegistry

Return mapping of canonical dataset names to loader callables.

load_catalog `cached` ¶

load_catalog() -> dict[str, dict[str, object]]

Load the dataset metadata catalog from disk.

get_dataset_metadata ¶

get_dataset_metadata(name: str) -> dict[str, object]

Return metadata for name from the catalog.

list_available_datasets ¶

list_available_datasets() -> list[str]

Return the canonical dataset names discovered in the loaders.

resolve_dataset_names ¶

resolve_dataset_names(selectors: Any) -> list[str] | None

Resolve selectors into canonical dataset names.

The selectors argument accepts strings (dataset names or tag:<name>), dictionaries with include/exclude/limit keys, metadata filters, or iterables of those forms. None returns None to signal "all datasets".

Loading¶

anomalybench.benchmarks.load_all_datasets ¶

Dataset loading helpers used throughout the benchmarking utilities.

load_all_datasets ¶

load_all_datasets(names: Sequence[str] | None = None) -> list[DatasetSpec]

Load benchmark datasets by canonical name.

Raw loaders may use historical label conventions. Dataset metadata declares source_anomaly_label and this function converts every loaded benchmark to the public evaluation contract 1 = anomaly before returning it.

Parameters¶

names: Optional sequence of canonical dataset identifiers (for example, "iris"). None loads every available dataset.

Metrics¶

anomalybench.benchmarks.metrics ¶

Metric utilities for benchmark evaluation.

DEFAULT_SCORE_ORIENTATION `module-attribute` ¶

DEFAULT_SCORE_ORIENTATION = 'higher_is_more_anomalous'

Orientation applied to scores that carry no orientation metadata.

MetricConfig `dataclass` ¶

Normalized benchmark metric configuration.

resolve_metric_config ¶

resolve_metric_config(config: Any = None) -> MetricConfig

Normalize CLI/YAML metric configuration.

canonicalize_anomaly_scores ¶

canonicalize_anomaly_scores(scores: Any) -> NDArray[np.floating[Any]]

Return scores where larger values consistently mean more anomalous.

Raw detector score semantics are preserved by detector score methods. Benchmark evaluation uses orientation metadata carried by BaseDetector.detect_anomalies to put heterogeneous detectors onto a common ranking direction before computing ranking- or threshold-based metrics.

evaluate_metrics ¶

evaluate_metrics(y_true: Any, scores: Any, *, runtime_seconds: float, config: MetricConfig | None = None) -> dict[str, float | None]

Evaluate configured benchmark metrics using canonical anomaly scores.

precision_at_k ¶

precision_at_k(y_true_positive: Any, scores: Any, k: int | None = None) -> float | None

Return precision among the top-k highest scores.

recall_at_k ¶

recall_at_k(y_true_positive: Any, scores: Any, k: int | None = None) -> float | None

Return recall among the top-k highest scores.

best_f1_score ¶

best_f1_score(y_true_positive: Any, scores: Any) -> float | None

Return the best F1 score across precision-recall thresholds.

Reproducibility¶

anomalybench.benchmarks.reproducibility ¶

Reproducibility helpers for benchmark runs.

utc_timestamp ¶

utc_timestamp() -> str

Return an ISO-8601 UTC timestamp.

canonical_json ¶

canonical_json(value: Any) -> str

Serialize value as stable JSON.

stable_hash ¶

stable_hash(value: Any, length: int = 16) -> str

Return a short SHA-256 hash for a JSON-serializable value.

package_version ¶

package_version() -> str

Return the project version without requiring an installed package.

normalize_run_id ¶

normalize_run_id(run_id: str | None, timestamp: str, config_hash: str) -> str

Return a user-supplied or deterministic benchmark run identifier.

seed_runtime ¶

seed_runtime(seed: int | None) -> None

Seed common Python and NumPy RNGs when a benchmark seed is provided.

apply_seed_to_detector_entries ¶

apply_seed_to_detector_entries(detector_entries: list[dict[str, Any]], seed: int | None) -> list[dict[str, Any]]

Add supported seed parameters to detector entries when absent.

benchmark_config_hash ¶

benchmark_config_hash(dataset_keys: list[str], detector_entries: list[dict[str, Any]], random_seed: int | None, n_jobs: int | None, metric_config: dict[str, Any] | None = None) -> str

Return the stable hash for the effective benchmark configuration.

build_manifest ¶

build_manifest(*, run_id: str, timestamp: str, config_hash: str, dataset_keys: list[str], detector_entries: list[dict[str, Any]], random_seed: int | None, n_jobs: int | None, output_directory: str | None, dataset_integrity: list[dict[str, Any]], metric_config: dict[str, Any] | None = None, dataset_metadata: list[dict[str, Any]] | None = None) -> dict[str, Any]

Build the reproducibility manifest for a benchmark run.

build_report ¶

build_report(manifest: dict[str, Any], rows: list[dict[str, Any]]) -> dict[str, Any]

Build a versioned benchmark report payload.

write_json ¶

write_json(path: str | Path, payload: dict[str, Any]) -> None

Write a reproducibility payload as pretty JSON.

collect_dataset_integrity ¶

collect_dataset_integrity(names: list[str] | None) -> list[dict[str, Any]]

Collect integrity metadata for bundled files used by selected datasets.