Machine learning models are often monitored for feature drift. Teams compare the distribution of inputs in production with the distribution seen during training. If age, transaction amount, sensor temperature, customer tenure, or device type changes, the monitoring system raises a warning.
Feature drift matters, but it is not the only way a model can become unreliable.
Sometimes the input distribution looks stable while the outcome distribution changes. A disease becomes more common after an outbreak. Fraud prevalence rises after attackers discover a loophole. Equipment failures drop after a maintenance campaign. Churn increases after a price change. A hospital changes admission policy, altering the share of high-risk patients. A factory replaces an aging component, reducing the base rate of a specific failure mode.
This is prevalence shift, also called base-rate drift or label shift in many machine learning settings.
Prevalence shift occurs when:
\[P(Y)\]changes over time, even if the conditional feature distribution within each class remains similar.
This can break deployed systems in ways that ordinary model metrics may miss. A classifier may still rank cases well, but its predicted probabilities may become miscalibrated. A threshold may produce too many alerts or too few. Precision may collapse even if recall stays stable. A clinical screening workflow may overload staff. A maintenance alert system may become noisy after the base rate of failures falls.
Base rates are not background details. They are part of the decision system.
What Is Prevalence?
Prevalence is the proportion of cases with the outcome of interest.
For binary classification:
\[\pi = P(Y = 1)\]In healthcare, ( \pi ) might be the share of patients with sepsis, readmission, disease recurrence, or treatment failure. In maintenance, it might be the share of assets that fail in the next 30 days. In fraud detection, it is the share of transactions that are fraudulent. In churn modeling, it is the share of customers who cancel.
Prevalence is not the same as model accuracy. It describes the environment in which the model operates.
If prevalence changes, decision metrics change. This happens even when the model’s ability to separate positives from negatives is unchanged.
Why Base Rates Affect Precision
Precision answers:
\[P(Y = 1 \mid \hat{Y} = 1)\]In words: among the cases the model flags as positive, how many are truly positive?
Precision depends strongly on prevalence. When the positive class is rare, even a classifier with strong discrimination can produce many false positives relative to true positives. When the positive class becomes more common, precision can improve even if the model itself has not changed.
This is one reason AUC can be misleading in operations. AUC measures ranking ability across thresholds. It can remain stable while precision, alert volume, and workload change dramatically.
Suppose a maintenance model flags machines for inspection. At a fixed threshold, it catches 80 percent of failures and falsely flags 5 percent of healthy machines.
If failure prevalence is 10 percent, the flagged set has a reasonable concentration of true failures. If prevalence drops to 1 percent after a reliability improvement program, the same false-positive rate may dominate the inspection queue.
The model did not necessarily become worse at ranking. The environment changed.
Label Shift
In machine learning, label shift usually refers to a setting where the class prior changes:
\[P_{\text{train}}(Y) \neq P_{\text{prod}}(Y)\]while the class-conditional feature distributions are assumed stable:
\[P_{\text{train}}(X \mid Y) \approx P_{\text{prod}}(X \mid Y)\]This assumption is not always true, but it is useful. It describes cases where the meaning of a positive or negative case has not changed much, but the mix of positives and negatives has.
For example, during flu season, the proportion of patients with respiratory infection may increase. The clinical features of infection may be similar, but positive cases are more common. In a factory after a component redesign, the features of bearing failure may be similar, but bearing failures may be less common.
Under label shift, the model score may still contain useful information, but the posterior probability needs adjustment because the prior has changed.
Bayes’ Rule and Base Rates
Base rates matter because probabilities combine evidence with priors.
Bayes’ rule says:
\[P(Y=1 \mid X) = \frac{P(X \mid Y=1)P(Y=1)} {P(X)}\]The term ( P(Y=1) ) is the prevalence. If it changes, the posterior probability changes, even if the evidence ( X ) has the same meaning within each class.
This is why a diagnostic test cannot be interpreted without disease prevalence. A positive test result has different positive predictive value in a high-prevalence clinic than in a low-prevalence general screening population.
The same logic applies to machine learning. A risk score trained under one base rate may not be calibrated under another.
Calibration Under Prevalence Shift
A calibrated model has predicted probabilities that match observed frequencies. If a model assigns many cases a probability near 0.20, about 20 percent of those cases should be positive.
Prevalence shift can break calibration.
Imagine a model trained when failure prevalence was 8 percent. After a maintenance intervention, true failure prevalence falls to 3 percent. If the model still outputs probabilities on the old scale, it may overestimate risk across many score ranges.
This affects decisions. A threshold chosen for the old environment may trigger unnecessary interventions. A dashboard may exaggerate risk. A resource allocation process may send inspectors to assets that are no longer as risky as their scores suggest.
Calibration should therefore be monitored over time, not checked once at training.
Useful diagnostics include:
- Calibration curves by time period
- Observed-to-expected ratios
- Brier score by period
- Expected calibration error by period
- Calibration by subgroup or site
- Score distribution alongside outcome prevalence
When labels are delayed, calibration monitoring must wait until outcomes mature. That delay should be built into the monitoring design.
Thresholds Are Not Portable
A classification threshold is a policy choice. It translates scores into actions.
For example:
- Alert if predicted sepsis risk exceeds 0.15
- Inspect if failure risk exceeds 0.10
- Review if fraud probability exceeds 0.03
- Contact if churn probability exceeds 0.25
These thresholds depend on prevalence, cost, capacity, and calibration. If prevalence changes, the threshold may no longer produce the intended precision, recall, workload, or expected value.
In production, a threshold should often be monitored by its consequences:
- How many cases are flagged?
- What is the precision among flagged cases?
- What is the false negative rate?
- Is review capacity exceeded?
- Are interventions still cost-effective?
- Do thresholds behave differently across groups?
The threshold that was optimal during validation may be wrong after base-rate drift.
Healthcare Example
Consider a hospital model that predicts 30-day readmission risk at discharge.
The model was trained during a period when readmission prevalence was 14 percent. After a new follow-up care program, readmissions fall to 9 percent. The model may still rank patients reasonably: the highest-risk patients are still more likely to be readmitted than the lowest-risk patients. But the predicted probabilities may be too high.
If the hospital uses a fixed threshold to assign follow-up calls, the program may call too many patients relative to the new risk level. That may sound harmless, but follow-up capacity is limited. Overcalling lower-risk patients may reduce attention to patients who need more intensive support.
Now consider the opposite situation. During a respiratory outbreak, readmission prevalence rises. A threshold chosen under normal conditions may miss too many patients if the model probabilities are not updated. Staff may also be overloaded if alert volume rises faster than capacity.
In healthcare, base-rate drift is not just a statistical issue. It affects staffing, patient prioritization, and trust in risk scores.
Maintenance Example
Consider a predictive maintenance model that estimates whether a machine will fail in the next 14 days.
The model was trained when a fleet was aging and failure prevalence was high. After a replacement campaign, the fleet becomes healthier. The feature patterns associated with failure may remain valid, but failures are less common.
At the old threshold, the model may generate many inspections with low yield. Technicians may start ignoring alerts because too few lead to real findings. The model becomes operationally ineffective even if its ranking performance is still acceptable.
Alternatively, suppose operating conditions become harsher because production volume increases. Failure prevalence rises. The old threshold may generate too few alerts or too late an intervention. Spare parts planning may also fail because expected failure volume is underestimated.
Maintenance systems need base-rate monitoring because reliability changes are often the point of the program. A successful maintenance strategy should change the prevalence of failures. The model must adapt to that success.
Fraud and Abuse Example
Fraud detection is especially sensitive to base-rate drift.
Fraud prevalence can change after holidays, policy updates, new attack strategies, product launches, or enforcement actions. Attackers adapt. A model trained during a quiet period may underestimate fraud during an attack. A model trained during an attack may overflag after the attack ends.
Precision is central because review teams have limited capacity. If prevalence rises, the same threshold may send many more cases to review. If prevalence falls, the same threshold may produce a queue dominated by false positives.
Fraud teams often manage this through dynamic thresholds, review budgets, active learning, and rapid label feedback. But those strategies work only if the team measures the base rate and understands label delay.
Detecting Prevalence Shift
Detecting prevalence shift requires labels, or at least reliable proxies for labels.
This is harder than feature drift monitoring because labels often arrive late. A healthcare outcome may take 30 days. A maintenance failure may take weeks to confirm. A fraud label may require investigation.
Useful approaches include:
- Monitor mature outcome prevalence by period.
- Track proxy outcomes when labels are delayed.
- Compare predicted positive volume with realized positive volume.
- Monitor observed-to-expected ratios.
- Track precision among reviewed or audited cases.
- Use random audits to estimate prevalence outside selected review queues.
- Separate true prevalence changes from label-process changes.
The last point is important. A change in recorded prevalence may reflect a real-world shift, but it may also reflect a new coding policy, investigation workflow, sensor rule, or label delay.
Prevalence monitoring should always ask whether the event changed or the measurement process changed.
Selection Bias in Observed Labels
Observed labels may not represent all production cases.
In fraud detection, only reviewed transactions may receive reliable labels. In healthcare, only patients who return to the same network may have observed outcomes. In maintenance, only inspected assets may receive confirmed failure diagnoses. In customer churn, only certain contract types may have clean cancellation labels.
If labels are observed selectively, prevalence estimates can be biased.
For example, if a fraud model sends high-score cases to review, then the labeled set will overrepresent risky cases. Estimating production prevalence from that reviewed set will exaggerate the base rate unless the selection process is accounted for.
Random audits are one practical solution. A small random sample of production cases can be labeled independently of the model’s decisions. This provides an anchor for prevalence estimation, calibration, and bias detection.
Without random or representative labeling, teams may confuse review prevalence with population prevalence.
Adjusting Probabilities Under Label Shift
When label shift is plausible, probabilities can sometimes be adjusted using the new class prior.
Suppose a model estimates odds under the training prevalence. If the class-conditional feature distributions are stable, a prior correction can update the odds:
\[\text{odds}_{\text{new}}(Y=1 \mid X) = \text{odds}_{\text{old}}(Y=1 \mid X) \times \frac{\pi_{\text{new}}/(1-\pi_{\text{new}})} {\pi_{\text{old}}/(1-\pi_{\text{old}})}\]This is not a universal fix. It depends on assumptions. If the relationship between features and labels has also changed, prior correction alone is insufficient.
Still, the formula gives a useful lesson: changing base rates should change probabilities. If the deployed system has no way to adjust probabilities or thresholds when prevalence changes, it is likely brittle.
Base-Rate Drift and Decision Value
The value of a model depends on prevalence.
If the positive class becomes extremely rare, many interventions may no longer be cost-effective. If it becomes common, more aggressive intervention may be justified. If prevalence differs by site or subgroup, a global threshold may allocate resources poorly.
For example, a maintenance intervention costing 1,000 euros may be justified when failure risk is 20 percent and failure cost is high. It may not be justified when failure risk falls to 2 percent. A clinical outreach program may need stricter prioritization when prevalence rises and staff are overloaded.
This connects prevalence shift to decision theory. The model score is not the final answer. The decision depends on current risk, cost, and capacity.
Monitoring Dashboard Design
A good monitoring dashboard should not show only feature drift and model score distributions. It should also show base-rate behavior.
Useful panels include:
- Outcome prevalence by time period
- Outcome prevalence by subgroup, site, or product line
- Score distribution by period
- Alert volume by threshold
- Precision and recall when labels mature
- Observed-to-expected ratio
- Calibration by score band
- Review capacity and queue size
- Label delay distribution
- Random-audit prevalence estimates
These metrics should be interpreted together. A rising alert volume may reflect rising risk, score drift, a threshold bug, or a feature pipeline issue. A falling observed prevalence may reflect a real improvement, delayed labels, or reduced detection.
Monitoring should help diagnose, not merely alarm.
Common Mistakes
The first mistake is assuming stable prevalence because input features look stable.
The second mistake is relying on AUC alone. AUC can remain stable while precision, calibration, and workload change.
The third mistake is keeping thresholds fixed after major changes in prevalence, cost, or capacity.
The fourth mistake is estimating prevalence from selected labels, such as reviewed cases only.
The fifth mistake is treating recorded prevalence as real prevalence without checking whether the label process changed.
The sixth mistake is recalibrating too quickly using immature labels.
The seventh mistake is ignoring subgroup base rates. A global prevalence estimate can hide local changes that matter operationally.
Practical Workflow
A practical base-rate drift workflow starts with the target definition.
Define the event, prediction horizon, and label maturity period. Decide when a label is reliable enough for monitoring.
Track prevalence over mature periods. Compare it with training prevalence and recent historical baselines.
Monitor score distributions and observed-to-expected ratios. This helps distinguish model score movement from outcome base-rate movement.
Evaluate thresholds by workload and decision value. Do not assume the old threshold remains useful.
Use representative audits where possible. Avoid estimating population prevalence only from cases selected by the model.
Recalibrate or adjust thresholds when evidence supports it. If feature-label relationships changed, retraining may be needed rather than prior correction alone.
Document the change. Was the base-rate shift caused by seasonality, policy, intervention, label process, data source, or real-world behavior?
This workflow keeps monitoring connected to the actual decision environment.
Conclusion
Prevalence shift is easy to overlook because it does not always appear as a dramatic feature change. The inputs may look familiar while the outcome becomes more or less common.
That change matters. It affects probability calibration, precision, thresholds, workload, decision value, and trust. A model can still rank cases well and still become poorly calibrated or operationally expensive because the base rate moved.
Healthcare screening, predictive maintenance, fraud detection, churn prevention, and alerting systems all depend on current prevalence. The base rate is not a static training-set detail. It is a live property of the environment.
Reliable machine learning systems monitor it, reason about it, and update decisions when it changes.
