Pseudo-Label Confidence Is Not the Same as Correctness

A common semi-supervised learning pipeline looks reassuringly conservative.

Train a classifier on the labelled observations. Predict class probabilities for the unlabelled observations. Keep only predictions above a confidence threshold such as

Treat those predictions as labels. Retrain.

The intuition is obvious:

if the model is 95 percent confident, the pseudo-label is probably correct.

That sentence contains an assumption.

A model score of

is not, by itself, a statement that 95 percent of such predictions are correct.

It is a number produced by a fitted model.

To interpret it probabilistically, we need calibration.

To use it safely for pseudo-labelling, we need something stronger: calibration on the particular unlabelled population from which pseudo-labels are selected.

And because those pseudo-labels are fed back into training, even a small systematic error can become part of the next model rather than remaining an isolated prediction mistake.

The issue is not that pseudo-labelling is inherently unreliable.

The issue is that confidence and correctness are different statistical objects.

The Basic Self-Training Rule

Suppose we have labelled observations

and unlabelled observations

Let a classifier produce class probabilities

For each unlabelled observation, define the predicted class

and its reported confidence

Confidence-based pseudo-labelling accepts the observation when

for some threshold $\tau$, perhaps

The new pseudo-labelled set is

The next model is then trained using some combination of

and

Nothing in this construction guarantees

That equality is a calibration statement.

It has to be earned.

Confidence Is a Model Output

Consider binary classification.

Let

be the model's estimated probability that

If the model outputs

then the mathematical fact is simply that the fitted model assigned the value 0.95 to that observation.

The empirical interpretation

requires calibration.

A perfectly calibrated probabilistic classifier satisfies, informally,

for relevant values of $p$.

For predicted-class confidence, define

A corresponding calibration condition is

Only under such a relation can model confidence be interpreted directly as correctness frequency.

Neural networks, boosted trees, logistic models under misspecification, and many other classifiers can all be miscalibrated.

A model can rank observations correctly while being systematically overconfident.

That distinction matters greatly for pseudo-label selection.

A Threshold Is Not a Correctness Guarantee

Suppose the accepted pseudo-labels all have reported confidence between

and

One might expect their error rate to be at most about five percent.

But imagine that the classifier is overconfident on the unlabelled population and that accepted predictions are correct only 80 percent of the time.

Then

not 0.95.

The threshold has selected observations according to the model's own score.

It has not changed the model's score into a calibrated probability.

This difference is easy to miss because the number

already looks probabilistic.

The semantic interpretation comes from calibration, not from the presence of a decimal between zero and one.

Perfect Calibration Would Help

It is useful to state the favourable case clearly.

Suppose predicted-class confidence is perfectly calibrated on the relevant population:

Then for the selected set

we have

Because every accepted confidence is at least $\tau$,

Therefore,

under perfect calibration.

So if calibration genuinely holds on the accepted unlabelled population, a threshold such as 0.95 has a useful interpretation.

The problem is not confidence thresholding itself.

The problem is assuming the condition without checking it.

Calibration on the Labelled Set Is Not Enough

Suppose the model is well calibrated on a held-out labelled sample.

That is good evidence.

It still does not establish calibration on the unlabelled pool.

Let

denote the labelled-data distribution and

the population from which unlabelled observations were drawn.

Calibration estimated under

need not transfer to

If the feature distribution changes,

or if the conditional relationship changes,

the calibration relationship can change as well.

Even if the classifier remains accurate overall, its high-confidence region may behave differently.

That is exactly the region self-training selects.

Pseudo-Label Selection Creates a New Population

The accepted pseudo-labels are not a random sample from the unlabelled pool.

They satisfy

So the pseudo-labelled training distribution is conditional:

That distribution can differ sharply from the full unlabelled distribution.

For example, the model may be most confident on:

• majority-class observations,
• observations close to familiar training examples,
• regions with low epistemic uncertainty,
• regions where its inductive bias is strongest,
• or groups over-represented in the labelled data.

The pseudo-labelled set can therefore be heavily selected even when the original unlabelled pool is representative.

Self-training does not merely add data.

It adds data through a model-dependent sampling mechanism.

High Confidence Can Be Concentrated in the Wrong Places

Suppose a binary classifier is trained from a small labelled sample.

The true class boundary is nonlinear, but the fitted model is linear.

Far from its estimated linear boundary, the model may produce probabilities near zero or one.

Those predictions can be highly confident because the model extrapolates strongly.

Yet some of those regions may lie on the wrong side of the true nonlinear boundary.

The problem is systematic.

The model is not uncertain because, inside its own model class, the prediction is clear.

Confidence reflects certainty conditional on the fitted representation and model.

It does not measure whether the model class itself is correct.

This distinction becomes important whenever model misspecification is plausible.

Confidence and Epistemic Uncertainty Are Different

A softmax probability or logistic probability is often treated as though it summarises all relevant uncertainty.

It does not.

A model can output

in a region poorly represented by training data.

The score may reflect a large logit magnitude rather than strong empirical support.

In discriminative classification, the predictive score is generated by the fitted function.

It need not increase uncertainty simply because the observation lies far from the labelled support.

Some model families extrapolate with increasing confidence.

For pseudo-labelling, this means that thresholding can preferentially accept points that are confidently wrong because of extrapolation.

Distance from labelled support, ensemble disagreement, posterior uncertainty, density estimates or other diagnostics may therefore provide information that the class probability alone does not.

Accepted Pseudo-Labels Can Outnumber Real Labels Very Quickly

Suppose we begin with

human-labelled observations.

Now assume the unlabelled pool is large and the 0.95 threshold accepts

pseudo-labels.

If the accepted pseudo-label error rate is only

then the expected number of incorrect pseudo-labels is

So the training set contains roughly

incorrect synthetic labels versus only

human labels.

This does not imply that the pseudo-labelled model must fail.

Correct pseudo-labels also contribute information, and observations differ in leverage.

But the arithmetic exposes an important asymmetry.

A seemingly small error rate can correspond to a large absolute amount of wrong supervision when

The relevant quantity is not only pseudo-label accuracy.

It is pseudo-label error mass relative to the trusted labelled signal.

A Weighted Objective Makes the Trade-Off Explicit

A typical self-training objective can be written as

where

• $U_\tau$ is the accepted pseudo-labelled set,
• $w_j$ is an optional confidence or reliability weight,
• $\lambda$ controls the contribution of pseudo-labelled observations.

This expression makes the central balance visible.

The labelled term contains fewer observations but trusted targets.

The pseudo-labelled term may contain far more observations but noisy targets generated by the current model.

If the second term becomes numerically dominant, the next model can be trained more strongly to reproduce its own previous decisions than to respect the original labelled evidence.

That is the mechanism behind confirmation bias in self-training.

One Wrong Prediction Is Not the Main Problem

A standard supervised model makes prediction errors.

Those errors do not normally become new training labels automatically.

Self-training changes that.

At iteration $t$, suppose the model predicts

for an unlabelled observation.

If that pseudo-label is accepted, iteration $t+1$ is fitted using

as a target.

The process is

A prediction error has therefore crossed the boundary from output to input.

That is why self-training should be understood as a feedback system.

A Simple Feedback Model

Let

denote the error rate among pseudo-labels accepted at iteration $t$.

Suppose the next model's pseudo-label error behaves approximately like

where

• $\alpha$ represents amplification or persistence of existing pseudo-label errors,
• $\beta$ represents fresh error introduced by limited labelled information, noise or model misspecification.

If

the process has a stable fixed point

If $\alpha$ is small, errors are corrected quickly.

If $\alpha$ is close to one, errors persist.

If the effective feedback is stronger than this simple stable model permits, self-training can move away from the labelled solution rather than toward it.

The formula is only a schematic model.

Its purpose is to emphasise that repeated pseudo-labelling is dynamic.

Evaluating only the accuracy of the first pseudo-label batch can miss what happens after those labels influence later models.

Class Imbalance Makes Thresholding Asymmetric

Suppose class zero is common and class one is rare.

A classifier trained on few labels may be highly confident about the majority class and cautious about the minority class.

With a common threshold $\tau$, accepted pseudo-labels may therefore satisfy

The retrained model now sees an even more majority-heavy target distribution.

That can increase the next round of majority-class confidence, creating a feedback loop in class proportions.

The process can be written schematically as

A global confidence threshold does not control this mechanism.

Class-specific thresholds, distribution alignment or explicit prior constraints may be needed when imbalance matters.

Calibration Can Differ by Class

Even when global expected calibration error looks acceptable, class-conditional calibration can be poor.

For class $k$, one may examine

A classifier can be well calibrated on average while being overconfident for one class and underconfident for another.

That matters for pseudo-labelling because accepted samples are partitioned by predicted class.

If class-specific calibration differs, the pseudo-label error rate also differs.

Reporting only one global calibration statistic can therefore conceal the part of the model that creates most pseudo-label noise.

Calibration Can Differ by Subgroup

The same issue appears across populations.

Suppose

denotes a subgroup, device type, geography, acquisition source or time period.

A model may satisfy approximate global calibration,

while failing conditionally:

If pseudo-label acceptance rates also differ by group, the training data can become concentrated in the groups where the model is most confident rather than the groups where additional supervision is most needed.

The resulting feedback is not merely a calibration problem.

It can become a representation problem because under-represented regions receive fewer pseudo-labels and therefore less influence during retraining.

Distribution Shift Makes Confidence Thresholds Fragile

Suppose a threshold was selected using labelled validation data from distribution

Later, unlabelled observations come from

Even if the model continues producing scores in the same numerical range, the mapping

may change.

A threshold calibrated under $P_0$ can therefore have a different selective risk under $P_1$.

Define the pseudo-label error rate at threshold $\tau$ as

The relevant quantity for self-training is not merely $\tau$.

It is

Under shift, that risk must be re-estimated or at least stress-tested.

Precision-Coverage Trade-Off

Raising the confidence threshold usually reduces the number of accepted pseudo-labels.

Define coverage

and pseudo-label error

A useful pseudo-label selection curve is therefore

High thresholds generally reduce coverage.

They may reduce error as well, but that relationship should be measured rather than assumed.

The relevant engineering question is not

Which threshold sounds conservative?

It is

How much pseudo-labelled coverage do we obtain for a tolerable error rate?

That is a selective-classification problem.

Why 0.95 Is Often an Arbitrary Number

Thresholds such as

look principled because they are familiar probability levels.

But without calibration they are simply hyperparameters.

A threshold of 0.95 in one model can correspond to lower empirical correctness than 0.80 in another.

Even within the same model, the meaning can vary by class, subgroup and time period.

The threshold should therefore be selected through an explicit validation objective, not because 0.95 feels safe.

Calibration Methods Help, but They Do Not Solve Everything

Post-hoc calibration methods include:

• temperature scaling,
• Platt scaling,
• isotonic regression,
• beta calibration,
• and more flexible calibration models.

For neural classifiers, temperature scaling is often a useful baseline.

If logits are

temperature scaling replaces them with

before applying softmax.

A temperature

usually softens overconfident predictions.

This can improve calibration on validation data without changing the predicted class ranking.

But post-hoc calibration does not solve distribution shift automatically.

A calibration map fitted on one labelled distribution can fail on another.

Nor does it remove model misspecification.

It improves the interpretation of scores under the conditions where the calibration relationship remains valid.

Use Trusted Labels to Measure Selective Accuracy

If enough labelled validation data exist, evaluate accuracy conditionally on the same acceptance rule that will be used for pseudo-labelling.

For threshold $\tau$, estimate

Also estimate coverage,

Plotting

against

is more informative than quoting one confidence threshold.

It directly evaluates the selection rule.

Confidence becomes operational only after we know how it behaves empirically.

Confidence Bins Are Useful but Can Hide the Tail

Reliability diagrams group predictions into bins.

For bin $B_m$, compare

with

That is helpful.

But pseudo-labelling often depends almost entirely on the extreme right tail of the confidence distribution.

A model can have a reasonable global calibration error while the bin

is poorly calibrated because it contains few validation observations.

For self-training, tail calibration matters disproportionately.

Report uncertainty there.

A Confidence Threshold Should Have an Interval Around Its Accuracy

Suppose only 80 labelled validation predictions satisfy

If 76 are correct, the observed selective accuracy is

That does not mean the true selective accuracy is exactly 0.95.

The estimate itself has sampling uncertainty.

This matters because high thresholds often leave few labelled validation examples.

A pseudo-label rule should therefore be judged using confidence or credible intervals for selective accuracy, not only the point estimate.

The irony is easy to miss:

the more selective the confidence threshold becomes, the fewer labelled observations may remain for estimating how trustworthy that threshold actually is.

Agreement Between Models Can Add Information

One way to reduce reliance on a single model's confidence is to require agreement across independently trained models or perturbations.

Suppose models

produce predictions

A pseudo-label might be accepted only when

and the confidence criteria are also satisfied.

This can reduce some idiosyncratic errors.

But agreement is not independence.

Models trained on the same data, features and architecture can share the same systematic bias.

Ten identical mistakes do not become correct because ten models agree.

Diversity of errors matters.

Consistency Regularisation Changes the Problem but Not the Principle

Modern semi-supervised methods often combine pseudo-labels with consistency regularisation.

The model is encouraged to produce similar predictions under perturbations of the same observation.

If

and

are two augmentations, the method may penalise disagreement between

and

This can be powerful when the augmentation is label-preserving.

But label preservation is itself an assumption.

An augmentation that changes the target class turns consistency into a wrong constraint.

Once again, the unlabelled method works by introducing structure that is not contained in the labels alone.

The structural assumption should be tested where possible.

Thresholding Can Hide Hard Regions

Confidence-based self-training preferentially selects easy observations.

That can be useful early in training.

It also means that hard regions may remain unlabelled indefinitely.

Suppose the true decision boundary passes through a region where confidence remains around

If the threshold stays at

those observations never enter the pseudo-labelled set.

The model repeatedly reinforces regions it already understands while receiving no new supervision near the boundary where its errors are concentrated.

This is one reason active learning and pseudo-labelling solve different information problems.

Pseudo-labelling tends to exploit confident regions.

Active learning often seeks informative uncertain regions for human labelling.

The two can complement each other.

Monitoring Only Final Test Accuracy Misses the Mechanism

Suppose a semi-supervised model improves test accuracy by one percentage point.

That is useful evidence.

But if we want to understand whether the pipeline is robust, we should inspect the pseudo-label process itself.

Track at least:

and

A final metric can hide a pipeline that succeeds only because one early pseudo-label batch happened to be favourable.

Repeated-seed behaviour is especially important.

A Better Experimental Contract

For confidence-based self-training, I would keep the following controls.

1. Preserve a purely supervised baseline

Train the same model family on the labelled sample only.

This isolates the contribution of pseudo-labels.

2. Keep a trusted validation set untouched

Do not use every available label for initial training.

Some real labels are needed to estimate calibration, selective accuracy and pseudo-label risk.

3. Evaluate multiple thresholds

Do not report only the threshold that produced the best final result.

Show the trade-off across

4. Record pseudo-label accuracy where ground truth is available

In simulation or retrospective experiments, hide labels rather than deleting them.

That allows direct measurement of

5. Track class and subgroup composition

Measure who gets pseudo-labelled, not only how many observations are accepted.

6. Repeat across seeds

Pseudo-label feedback can make the path dependent on the initial labelled sample.

Report the distribution of gains and failures.

7. Stress-test distribution shift

Alter the unlabelled pool while keeping the labelled test problem fixed.

This reveals whether confidence continues to mean the same thing when the unlabelled distribution changes.

Confidence Should Be Treated as a Measurement

A model confidence score is a measurement produced by an instrument.

The instrument has assumptions.

It can be biased.

Its calibration can drift.

Its error can depend on population and operating conditions.

Seen this way, a threshold such as

is not fundamentally different from any other measurement threshold.

Before using it to create training data, ask:

• What does this score measure?
• On which population was that interpretation validated?
• How uncertain is the calibration estimate?
• Does the interpretation hold by class and subgroup?
• What happens under plausible shift?
• How much training weight will accepted pseudo-labels receive?

That framing is more useful than treating confidence as an intrinsic property of a prediction.

Pseudo-Labels Are Not Labels

The notation itself can encourage overconfidence.

Once we write

next to

the two targets can look interchangeable.

They are not.

A human or experimentally observed label is itself imperfect in many applications, but its error mechanism is different from that of a pseudo-label generated by the model being trained.

A pseudo-label carries model dependence.

Its uncertainty should not disappear simply because it has been converted to an integer class ID.

If the model estimates class probabilities, retaining soft targets can sometimes preserve more information than collapsing immediately to

Even then, the probabilities remain model outputs and should not be confused with known conditional probabilities.

The Link to Negative Transfer

In an earlier article, I showed that unlabelled covariate shift can turn a small semi-supervised gain into negative transfer while the supervised baseline remains unchanged.

The article is available here:

When Unlabelled Data Makes Semi-Supervised Learning Worse

Confidence-based pseudo-labelling is one mechanism through which that failure can occur.

If distribution shift changes the relationship between

and correctness, the acceptance rule can continue selecting many observations while silently admitting more wrong pseudo-labels.

The threshold has not changed.

Its meaning has.

That is why confidence calibration should be monitored as part of the semi-supervised system rather than treated as a one-time model property.

The Main Distinction

Three quantities are easy to confuse:

Reported confidence

This is what the model outputs.

Conditional correctness probability

This is what calibration connects to the model score.

Selective pseudo-label accuracy

This is what matters directly when thresholded pseudo-labels are turned into training targets.

They are related.

They are not identical.

A sound pseudo-labelling pipeline measures the third quantity rather than inferring it from the first.

Conclusion

Confidence thresholding is a sensible idea.

If a model is well calibrated on the relevant population, high-confidence predictions are natural candidates for pseudo-labelling.

But the number on the probability output is not a correctness guarantee.

A threshold such as

means only that the fitted model assigned at least 0.95 probability to its preferred class.

To interpret that threshold operationally, we need evidence about

That quantity can change with model misspecification, class imbalance, subgroup composition, distribution shift and retraining.

And because pseudo-labels are fed back into the model, their errors are not passive.

They become supervision.

The important distinction is therefore

unless calibration makes the connection valid.

For self-training, I would go one step further:

That means measuring selective accuracy, coverage, calibration uncertainty, class composition, subgroup behaviour and iteration-to-iteration stability.

A pseudo-label should enter the training set because its reliability has been demonstrated under the operating conditions of the pipeline.

Not because the model printed a large number beside it.

References

• Arazo, E., Ortego, D., Albert, P., O'Connor, N. E., & McGuinness, K. (2020). Pseudo-labeling and confirmation bias in deep semi-supervised learning. International Joint Conference on Neural Networks. https://doi.org/10.1109/IJCNN48605.2020.9207304
• Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning, 1321–1330.
• Lee, D.-H. (2013). Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. ICML 2013 Workshop on Challenges in Representation Learning.
• Sohn, K., Berthelot, D., Li, C.-L., Zhang, Z., Carlini, N., Cubuk, E. D., Kurakin, A., Zhang, H., & Raffel, C. (2020). FixMatch: Simplifying semi-supervised learning with consistency and confidence. Advances in Neural Information Processing Systems, 33, 596–608.