Supervised learning depends on a simple promise: each training example has a target value that the model should learn to predict. In classification, that target is a class label. In regression, it is a number. In ranking, it may be a preference or relevance judgment. In survival analysis, it may be an event indicator and event time.
That promise is often weaker than it looks.
Labels can be wrong, delayed, ambiguous, biased, inconsistently applied, or produced by a process that changes over time. A hospital readmission label may miss patients who returned to a different hospital. A diagnosis code may reflect billing practice more than clinical truth. A maintenance failure label may depend on which technician wrote the work order. A fraud label may arrive months after the transaction. A customer churn label may confuse voluntary cancellation with failed payment.
This is label noise.
Label noise is not just another messy-data inconvenience. It corrupts the target the model is trained to imitate. If the target is unreliable, a more flexible model may simply learn the noise more efficiently. More data may amplify the bias. Better optimization may make the wrong answer more confident.
Understanding label noise is therefore central to trustworthy supervised learning.
What Is Label Noise?
Label noise occurs when the observed target ( \tilde{y} ) differs from the true target ( y ):
\[\tilde{y} \neq y\]The true target may be unobserved, expensive to measure, subjective, delayed, or defined only through a proxy. The model trains on ( \tilde{y} ), not on ( y ). If the observed labels are wrong in a systematic way, the model learns that systematic error.
In classification, label noise may mean a positive case is labeled negative or a negative case is labeled positive. In regression, it may mean the numeric outcome is measured with error. In event prediction, it may mean the event time is wrong, the event type is miscoded, or the event was not observed.
The simplest noisy-label model is:
\[P(\tilde{Y} \neq Y) = \eta\]where ( \eta ) is the label error rate.
This is useful as a starting point, but real label noise is rarely uniform. Some classes, groups, annotators, sites, devices, time periods, and edge cases usually have more label error than others.
Random Noise and Systematic Noise
Random label noise occurs when labels are wrong for reasons unrelated to the features or true class. For example, an annotator accidentally clicks the wrong option, a file is misread, or a rare data-entry error occurs.
Systematic label noise is more dangerous. It occurs when errors follow a pattern.
Examples include:
- A clinic consistently undercodes a diagnosis.
- One factory site uses a different failure taxonomy.
- An annotator is stricter than another annotator.
- A fraud team labels only investigated cases as fraud.
- A sensor alarm is treated as a failure label even when inspection finds no damage.
- A support team changes routing categories after a product reorganization.
Random noise often reduces model performance. Systematic noise can create biased models that perform well against the recorded labels while failing against the real-world objective.
The difference matters. If noise is random, robustness methods and more data may help. If noise is systematic, the label-generation process must be understood and corrected.
Health Examples
Healthcare data contains many forms of label noise.
Diagnosis codes are often used as labels, but they are not pure clinical truth. They may reflect reimbursement rules, documentation habits, coding workflows, and local practice. A patient may have a condition that is not coded. Another may receive a code during rule-out evaluation but not truly have the disease.
Readmission labels can be noisy when patients return to a different hospital network. Mortality labels can be delayed or incomplete. Disease recurrence may require imaging, pathology, or specialist review, so the recorded date may lag the biological event. Treatment toxicity may be underreported if symptoms are documented in free text rather than structured fields.
Labels can also reflect access to care. If a model uses historical diagnosis as the target, patients with less access to specialists may be mislabeled as negative because the disease was never detected. The model may then learn patterns of healthcare access rather than disease risk.
In health analytics, label noise is not only a technical problem. It can become a fairness, safety, and clinical validity problem.
Maintenance Examples
Predictive maintenance has its own label-noise problems.
Failure labels often come from work orders, maintenance logs, inspection notes, alarm systems, or technician-entered codes. These records are operational artifacts, not laboratory measurements.
A component may be replaced preventively and recorded as failed. A failure may be recorded under the symptom rather than the root cause. A technician may choose “other” because the taxonomy is too narrow. A sensor alarm may trigger a maintenance action even when no physical fault is found. A machine may fail, but the relevant work order may be closed days later with incomplete details.
Different sites may use different coding habits. One plant may distinguish bearing wear, lubrication failure, misalignment, and overheating. Another may record all of them as mechanical failure. A model trained across both sites may learn site-specific documentation style instead of mechanical degradation.
The result is a model that predicts the label process, not necessarily the failure process.
That distinction is critical. A useful maintenance model should support better interventions. If the labels encode inconsistent human documentation, the model may simply automate inconsistency.
Delayed Labels
Some labels are not wrong, but late.
Fraud may be confirmed weeks after a transaction. Equipment failure may be diagnosed after teardown. A medical outcome may require follow-up. A customer churn label may be known only after a billing cycle. A warranty claim may arrive months after product shipment.
Delayed labels create two risks.
First, recent data may look negative because positive labels have not matured yet. If the model is retrained too quickly, it may learn that recent risky cases are safe.
Second, evaluation may be optimistic or pessimistic depending on which labels have arrived. A validation period with incomplete outcomes is not a reliable validation period.
The correct design depends on label maturity. If labels require 90 days, then training and evaluation windows must respect that delay. A label is not available just because the row exists in the database.
Ambiguous Labels
Not all label disagreement is error. Some cases are genuinely ambiguous.
A radiology image may show borderline findings. A maintenance inspection may find early wear that one technician calls normal and another calls degradation. A customer complaint may fit several categories. A document may be both legal correspondence and a contract amendment.
In such cases, forcing one hard label may erase uncertainty. The model is asked to learn a boundary that experts themselves do not agree on.
Alternatives include:
- Multiple labels
- Probabilistic labels
- Adjudicated labels
- Uncertain or deferred labels
- Hierarchical taxonomies
- Label sets rather than single labels
Ambiguity should be modeled when it is part of the domain. Treating every disagreement as a mistake can create false confidence.
How Label Noise Affects Training
Label noise changes what the model learns.
With clean labels, empirical risk minimization tries to find a model ( f ) that minimizes:
\[\frac{1}{n}\sum_{i=1}^n L(f(x_i), y_i)\]With noisy labels, the model minimizes:
\[\frac{1}{n}\sum_{i=1}^n L(f(x_i), \tilde{y}_i)\]The objective has changed. The model is rewarded for fitting the observed labels, even when those labels are wrong.
Flexible models can eventually memorize noisy labels, especially when trained for many epochs or when the dataset is small. In deep learning, training loss can keep falling after validation performance stops improving because the model begins fitting idiosyncratic label errors.
In tree ensembles, noisy labels can create unnecessary splits that isolate mislabeled records. In logistic regression, systematic label noise can bias coefficients. In survival models, misclassified event indicators or wrong event times can distort hazard estimates.
Label noise is therefore not model-neutral. Its effect depends on model class, loss function, sample size, regularization, and the structure of the noise.
Evaluation Can Be Corrupted Too
Noisy training labels are bad. Noisy evaluation labels are worse.
If the test set contains label errors, the measured performance may not reflect true performance. A model may be penalized for correcting a bad label. Another model may be rewarded for reproducing labeling bias.
This is especially damaging when comparing models. A model that better captures real outcomes may look worse if the benchmark labels are noisy proxies. A simpler model may appear more stable because it aligns with the label-generation shortcut.
High-quality evaluation labels are worth more than large quantities of weak evaluation labels. A small expert-reviewed test set can be more informative than a large noisy holdout when the decision is important.
For high-stakes systems, evaluation data should often receive stricter review than training data.
Detecting Label Noise
Label noise cannot always be detected from data alone, but several signals help.
High-loss examples are a starting point. If a model repeatedly assigns low probability to the recorded label across cross-validation folds, the example may be mislabeled, unusual, or underrepresented.
Disagreement between models can help. If several strong models trained on different samples disagree with the label in the same way, the label deserves review.
Nearest-neighbor checks can help. If an example has a label that differs from very similar examples, it may be mislabeled or represent an important edge case.
Temporal checks can help. If labels suddenly change after a policy update, workflow change, or new coding guideline, the shift may be administrative rather than real.
Human review remains essential. Statistical flags identify candidates. Domain experts decide whether the label is wrong, ambiguous, rare but valid, or evidence of a deeper taxonomy issue.
Robust Training Strategies
There is no universal fix for label noise, but several strategies help.
Clean the highest-value labels. Review labels in the validation set, high-impact classes, rare outcomes, and examples that drive important decisions.
Use robust losses. Some losses reduce the influence of examples that appear inconsistent with the learned pattern. This can help with random noise, although it may also downweight rare but real cases.
Use regularization and early stopping. These can reduce memorization of noisy labels, especially in flexible models.
Use label smoothing when appropriate. Instead of treating labels as perfectly certain, label smoothing assigns less than full probability to the recorded class. This can reduce overconfidence, but it should not hide systematic labeling problems.
Model annotator reliability. When multiple annotators label the same examples, estimate annotator-specific error rates rather than assuming all labels are equally reliable.
Use probabilistic labels. If experts disagree or uncertainty is intrinsic, represent that uncertainty rather than forcing a single hard target.
Use noise-aware validation. Evaluate on a cleaner holdout set so that model selection is not dominated by noisy targets.
The right strategy depends on whether noise is random, systematic, delayed, ambiguous, or caused by a flawed target definition.
Do Not Clean Away the Hard Cases
Label cleaning has a danger: it can remove examples that are difficult but real.
A high-loss example is not automatically mislabeled. It may be a rare presentation of disease, an early sign of failure, a new fraud pattern, a minority subgroup, or a scenario missing from the model’s features.
If every difficult example is corrected, removed, or downweighted, the model may become cleaner in training and weaker in reality.
Good label review should distinguish:
- Incorrect labels
- Ambiguous labels
- Rare but valid cases
- Out-of-distribution cases
- Missing-feature cases
- Taxonomy failures
Each category implies a different action. Incorrect labels can be fixed. Ambiguous labels may need adjudication. Rare valid cases may need more data. Taxonomy failures may require redesign. Missing-feature cases may require new data collection.
Label Noise and Fairness
Label noise can be uneven across groups.
In healthcare, some populations may be underdiagnosed because they have less access to care or because symptoms are interpreted differently. In maintenance, some sites may document failures more thoroughly than others. In customer support, some languages or regions may receive lower-quality annotations. In fraud detection, only investigated cases may receive confirmed labels, and investigation itself may be biased.
If label quality differs across groups, model performance can appear unequal for reasons hidden inside the target.
Fairness evaluation should therefore ask:
- Are labels equally reliable across groups?
- Are labels equally likely to be missing?
- Are positive cases equally likely to be detected?
- Does the label reflect outcome risk or institutional attention?
- Are annotator disagreement rates different by group?
A model trained on biased labels can reproduce bias even when sensitive attributes are removed.
Governance and Metadata
Label governance is often less mature than feature governance, but it is just as important.
A useful labeling record should include:
- Label value
- Label source
- Label timestamp
- Labeling guideline version
- Annotator or system source
- Adjudication status
- Confidence or uncertainty
- Event time, when relevant
- Whether the label was later revised
This metadata makes it possible to audit changes, evaluate label delay, compare annotators, and understand why model performance changes over time.
Without label metadata, teams often discover too late that the target definition changed months ago.
Practical Workflow
A practical label-noise workflow begins before modeling.
Define the target carefully. Write down what the label means, when it becomes observable, who records it, and what can make it wrong.
Profile label sources. Compare labels across sites, annotators, systems, time periods, and subgroups.
Create a clean evaluation set. Use expert review, adjudication, or higher-quality data for the benchmark when the stakes justify it.
Train baseline models. Use cross-validation or temporal validation to identify high-loss and high-disagreement examples.
Review flagged examples. Do not automatically delete them. Classify the reason for disagreement.
Choose a mitigation strategy. Fix labels, redesign the taxonomy, use robust training, model uncertainty, add features, delay retraining until labels mature, or change the decision workflow.
Monitor label quality after deployment. Track label delay, revision rates, disagreement, source changes, and performance on clean audits.
This workflow treats labels as data products, not as unquestionable truth.
Conclusion
Label noise is one of the most important failure modes in supervised learning because it attacks the target itself.
In healthcare, labels can reflect coding practice, access to care, delayed outcomes, and clinical ambiguity. In predictive maintenance, labels can reflect technician documentation, preventive replacement, inconsistent taxonomies, and delayed failure diagnosis. In both domains, a model trained blindly on noisy targets may learn the record-keeping process instead of the real phenomenon.
The solution is not simply more data or a larger model. It is better target definition, stronger label metadata, clean evaluation sets, robust training, careful review of suspicious examples, and monitoring of the labeling process over time.
Supervised learning starts with labels. If those labels are not trustworthy, model quality is built on weak ground.
References
- Hosmer, D. W., Lemeshow, S., & Sturdivant, R. X. (2013). Applied Logistic Regression (3rd ed.). Wiley.
- Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed.). Springer.
