When Unlabelled Data Makes Semi-Supervised Learning Worse

Suppose you have 400 training observations but labels for only 40 of them.

The obvious temptation is to use the other 360 observations somehow. They contain feature information, they reveal the geometry of the training distribution, and throwing them away feels wasteful.

That intuition is the entry point to semi-supervised learning.

It is also where a dangerous sentence often appears:

Unlabelled data cannot hurt because it contains additional information.

That sentence is false.

Unlabelled observations contain information about a feature distribution. Whether that information helps classification depends on assumptions connecting that distribution to the class boundary we actually care about.

When those assumptions fail, additional unlabelled data can reduce accuracy.

The important word is not unlabelled.

It is relevant.

The Paired Comparison

Let

be the test accuracy of a supervised classifier trained only on the exposed labels, and let

be the accuracy of a semi-supervised method when the unlabelled covariates have been shifted by a vector (\delta).

Define the semi-supervised gain

Positive gain means the semi-supervised method beats the supervised comparator.

Negative gain means negative transfer:

Now define a second quantity,

This is the response to unlabelled covariate shift.

Negative response means the shift made the semi-supervised method worse relative to its own no-shift version:

These two events are not the same.

A method can be harmed by shift and still remain better than supervised learning:

Or it can cross all the way into negative transfer:

That distinction turns out to be useful.

Hold the Supervised Problem Fixed

If we want to isolate the effect of unlabelled data, the comparison must be paired.

Within each simulated dataset I keep fixed:

• every exposed labelled observation,
• every training label,
• the held-out test sample,
• the supervised model,
• the number of exposed labels.

Only the covariates of observations whose labels are hidden are translated.

If (L) is the set of labelled training points and (U) the unlabelled set, then

The supervised logistic regression therefore sees exactly the same labelled data for every (\delta).

Its accuracy cannot change because of the intervention.

The semi-supervised learner does see the shifted observations, so any change in its result comes from the altered unlabelled sample.

That is the experiment.

A Small Reproducible Example

The data are deliberately simple.

There are two balanced classes. The first feature contains a separated class signal and the second is noise. Ten percent of the training labels are exposed.

The supervised comparator is logistic regression. The semi-supervised method is confidence-based self-training using the same logistic model as its base estimator.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.semi_supervised import SelfTrainingClassifier

N_TRAIN = 400
N_TEST = 1_000
LABELLED_FRACTION = 0.10
SEPARATION = 4.0
NOISE = 1.0


def make_dataset(seed: int):
    """Generate one balanced binary classification problem."""
    rng = np.random.default_rng(seed)

    def sample(n: int):
        y = np.arange(n, dtype=int) % 2
        rng.shuffle(y)

        x1 = (2 * y - 1) * (SEPARATION / 2)
        x1 = x1 + rng.normal(0.0, NOISE, n)
        x2 = rng.normal(0.0, NOISE, n)
        return np.column_stack((x1, x2)), y

    x_train, y_train = sample(N_TRAIN)
    x_test, y_test = sample(N_TEST)

    labelled = np.zeros(N_TRAIN, dtype=bool)
    for class_label in (0, 1):
        indices = np.flatnonzero(y_train == class_label)
        count = round(indices.size * LABELLED_FRACTION)
        chosen = rng.choice(indices, size=count, replace=False)
        labelled[chosen] = True

    return x_train, y_train, x_test, y_test, labelled

The evaluation deliberately uses the same logistic model family on both sides.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
def evaluate(x_train, y_train, x_test, y_test, labelled):
    """Return supervised accuracy, self-training accuracy and SSL gain."""
    supervised = LogisticRegression(max_iter=1_000)
    supervised.fit(x_train[labelled], y_train[labelled])
    supervised_accuracy = accuracy_score(
        y_test,
        supervised.predict(x_test),
    )

    partially_observed = np.full_like(y_train, -1)
    partially_observed[labelled] = y_train[labelled]

    self_training = SelfTrainingClassifier(
        LogisticRegression(max_iter=1_000),
        threshold=0.75,
        criterion="threshold",
        max_iter=10,
    )
    self_training.fit(x_train, partially_observed)
    ssl_accuracy = accuracy_score(
        y_test,
        self_training.predict(x_test),
    )

    return (
        supervised_accuracy,
        ssl_accuracy,
        ssl_accuracy - supervised_accuracy,
    )

For every seed I first evaluate the unshifted data. I then move only the unlabelled observations along the informative feature.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
rows = []

for seed in range(50):
    x_train, y_train, x_test, y_test, labelled = make_dataset(seed)

    sup0, ssl0, gain0 = evaluate(
        x_train,
        y_train,
        x_test,
        y_test,
        labelled,
    )

    for shift in (0.0, 0.5, 1.0, 2.0):
        shifted = x_train.copy()
        shifted[~labelled, 0] += shift

        supervised, ssl, gain = evaluate(
            shifted,
            y_train,
            x_test,
            y_test,
            labelled,
        )

        rows.append(
            {
                "seed": seed,
                "shift": shift,
                "supervised": supervised,
                "self_training": ssl,
                "gain": gain,
                "response": gain - gain0,
            }
        )

results = pd.DataFrame(rows)

Because the labelled points are untouched, the supervised result is identical across shifts within each seed.

That is not an incidental property. It is the control that makes the comparison interpretable.

What Happens

Across the 50 repeated datasets, the average results are:

The no-shift semi-supervised method is slightly better on average:

The advantage is small, but positive.

A shift of only 0.5 already changes the sign of the mean gain:

At shift one,

and the self-training classifier is harmed relative to its own no-shift behaviour in 76 percent of the repeated datasets.

It falls below the supervised comparator in 68 percent.

The supervised accuracy remains

throughout because the supervised task has not changed.

Only the unlabelled sample changed.

More Unlabelled Data Is Not the Same as More Labelled Data

A labelled observation contributes both

An unlabelled observation contributes only

The learning algorithm has to infer how that feature distribution is related to the class structure.

Self-training does this through the model's own predictions. High-confidence predictions are promoted to pseudo-labels, after which they become part of the training process.

If the unlabelled covariates come from the same relevant distribution, those pseudo-labels can reinforce a useful boundary.

If they are shifted, confidence can become a liability.

A wrong pseudo-label is no longer merely one incorrect prediction. It becomes training data.

That creates a feedback loop:

The unlabeled sample has not become useless. It has become informative about the wrong feature distribution.

Shift Harm and Negative Transfer Are Different

The table contains two probabilities because they answer different questions.

The first is

This asks how often the distribution shift made the semi-supervised method worse than its own no-shift version.

The second is

This asks how often the shifted semi-supervised method became worse than simply ignoring the unlabelled observations.

Those are not interchangeable.

Imagine that no-shift self-training gains five percentage points and the shifted version gains only two. Then

so the shift harmed the method, but

so there is no negative transfer.

Conversely, a method can already be poor before the shift. In that case negative transfer may be common even if (R(\delta)) is close to zero.

The probabilities should therefore be reported separately.

Why the Response Does Not Have to Be Monotone

There is another detail in the table that I would not smooth away.

The mean response at shift one is

while at shift two it is slightly less negative:

At the same time, the empirical probability of harm rises from 76 percent to 78 percent.

So the larger shift is harmful on slightly more datasets but has a less negative mean response.

That is perfectly possible.

Different summaries depend on different parts of the response distribution.

There is therefore no reason to assume

Covariate shift interacts with the geometry of the labelled and unlabelled observations, the current decision boundary, confidence thresholds and the pseudo-label sequence.

The response can be directional and non-monotone.

Direction Matters Too

Writing only the magnitude

can also be misleading.

A shift of one unit toward a decision boundary is not equivalent to a shift of one unit parallel to it.

In more than one dimension,

and

have the same norm but may interact with the classifier in completely different ways.

For that reason, I would keep the full vector (\delta) in the experimental record and only compare magnitudes within a fixed direction.

A Better Evaluation Contract

A comparison such as

on one train-test split is not enough.

At minimum I would keep four invariants: the same labelled subset, the same target distribution, paired seeds between no-shift and shifted conditions, and explicit retention of negative transfer rather than clipping it away.

For repeated experiments I would report at least

Under an explicit shift experiment I would add

Averages describe magnitude. Sign probabilities describe frequency. Both matter.

The Supervised Baseline Is More Than a Benchmark

The supervised model is also a control arm.

Because it never sees the hidden-label observations, it should remain invariant when only those observations are shifted.

If the supervised accuracy changes, then something else in the experiment changed too.

This gives a simple falsification check:

for every shift applied exclusively to the unlabelled sample.

What This Example Does Not Prove

This small experiment is not an argument against semi-supervised learning.

The no-shift result actually shows the opposite: self-training provides a small average improvement when its assumptions are reasonably aligned with the data.

Nor does the example imply that every covariate shift will hurt.

The effect depends on direction, magnitude, labelled fraction, class geometry, model family, pseudo-label confidence and many other factors.

What the experiment establishes is narrower:

That is enough to reject the idea that more unlabelled data is automatically benign.

The Practical Question

Before adding unlabelled observations to a production training pipeline, I would ask a question more basic than which SSL algorithm to use:

Why should these unlabelled observations help with the decision boundary represented by the labelled data?

Sometimes the answer is strong. They come from the same acquisition process, time period and population. The cluster or smoothness assumptions are plausible. Pseudo-label confidence is calibrated.

Sometimes the answer is weak. The unlabelled pool was scraped from somewhere else, collected later, selected through another process, or contains a different mixture of populations.

In that situation the dataset is not simply "more data."

It is a distributional assumption.

And assumptions should be stress-tested.

The useful comparison is therefore not only

It is

If a modest shift turns a small gain into negative transfer, that tells us something important before deployment does.