Skip to content

oversampleqa.surrogate

oversampleqa.surrogate

Surrogate model evaluation utilities.

evaluate_surrogate_models(X, y, minority_label, oversampler, model, test_size=0.3, random_state=None)

Evaluate model performance with and without synthetic data.

The function trains the provided model under three scenarios:

  1. real_only – using the original training data without oversampling.
  2. real_plus_synth – using the oversampled training data.
  3. synth_only – replacing the real minority samples with the synthetic samples generated by the oversampler.
Parameters

X, y : ndarray Input data and labels. minority_label : int Label of the minority class. oversampler : imblearn BaseOverSampler Oversampler instance used to generate synthetic samples. model : sklearn estimator Classifier implementing fit/predict. test_size : float, default=0.3 Fraction of the dataset reserved for testing. random_state : int, optional Random seed for the split.

Returns

dict Mapping of scenario names to dictionaries with f1, recall and precision scores.

Source code in src/oversampleqa/surrogate.py 
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
def evaluate_surrogate_models(
    X: NDArray[np.floating],
    y: NDArray[Any],
    minority_label: int,
    oversampler: Any,
    model: Any,
    test_size: float = 0.3,
    random_state: int | None = None,
) -> dict[str, dict[str, float]]:
    """Evaluate model performance with and without synthetic data.

    The function trains the provided ``model`` under three scenarios:

    1. ``real_only`` – using the original training data without oversampling.
    2. ``real_plus_synth`` – using the oversampled training data.
    3. ``synth_only`` – replacing the real minority samples with the synthetic
       samples generated by the oversampler.

    Parameters
    ----------
    X, y : ndarray
        Input data and labels.
    minority_label : int
        Label of the minority class.
    oversampler : imblearn BaseOverSampler
        Oversampler instance used to generate synthetic samples.
    model : sklearn estimator
        Classifier implementing ``fit``/``predict``.
    test_size : float, default=0.3
        Fraction of the dataset reserved for testing.
    random_state : int, optional
        Random seed for the split.

    Returns
    -------
    dict
        Mapping of scenario names to dictionaries with ``f1``, ``recall`` and
        ``precision`` scores.
    """

    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=test_size, stratify=y, random_state=random_state
    )

    # Scenario 1: real-only
    model_real = clone(model)
    model_real.fit(X_train, y_train)
    pred_real = model_real.predict(X_test)

    # Scenario 2: real + synthetic
    oversampler_plus = clone(oversampler)
    try:
        X_res, y_res = oversampler_plus.fit_resample(X_train, y_train)
    except Exception:  # pragma: no cover - defensive
        logger.exception("Oversampler failed during fit_resample")
        raise
    model_plus = clone(model)
    model_plus.fit(X_res, y_res)
    pred_plus = model_plus.predict(X_test)

    # Scenario 3: synthetic-only
    synthetic = extract_synthetic_samples(X_train, X_res, y_res, minority_label)
    majority_mask = y_train != minority_label
    X_majority = X_train[majority_mask]
    y_majority = y_train[majority_mask]
    if len(synthetic) == 0:
        X_syn = X_train
        y_syn = y_train
    else:
        X_syn = np.vstack([X_majority, synthetic])
        y_syn = np.hstack([y_majority, np.full(len(synthetic), minority_label)])
    model_syn = clone(model)
    model_syn.fit(X_syn, y_syn)
    pred_syn = model_syn.predict(X_test)

    def _scores(y_true: NDArray[Any], y_pred: NDArray[Any]) -> dict[str, float]:
        mask_true = y_true == minority_label
        mask_pred = y_pred == minority_label
        return {
            "f1": f1_score(mask_true, mask_pred),
            "recall": recall_score(mask_true, mask_pred),
            "precision": precision_score(mask_true, mask_pred),
        }

    return {
        "real_only": _scores(y_test, pred_real),
        "real_plus_synth": _scores(y_test, pred_plus),
        "synth_only": _scores(y_test, pred_syn),
    }