Feature selection is often treated as a predictive exercise: keep the variables that improve cross-validation score and remove the rest. That works when the only goal is passive prediction under stable conditions. It becomes fragile when the model informs interventions.

In observational machine learning, the data was not generated by a randomized experiment. Customers self-select, clinicians triage, operators intervene, and policies change who receives treatment. A feature can be strongly predictive because it is a proxy for past decisions rather than a stable driver of the outcome.

Predictive Features Are Not Always Causal Features

Suppose a hospital readmission model uses “scheduled follow-up call” as a feature. The variable may predict lower readmission risk, but it is also a downstream consequence of clinical judgment. If the model recommends fewer follow-up calls for patients without that feature, it may reinforce the original allocation pattern rather than improve care.

The problem is not that predictive features are bad. The problem is that their role matters.

Feature role Example Risk
Cause of outcome machine temperature before failure usually useful for intervention
Proxy for cause ZIP code as proxy for access may create fairness or stability issues
Consequence of decision treatment assigned by clinician can encode historical policy
Collider customer complaint after both defect and support exposure can induce spurious associations
Leakage value recorded after prediction time invalid offline performance

Start With a Causal Diagram

A causal diagram does not need to be perfect to be useful. Even a rough directed acyclic graph forces the team to state assumptions:

  • What variables exist before the prediction time?
  • Which features are influenced by prior human decisions?
  • Which variables are proxies for unavailable causes?
  • Which variables are consequences of the outcome?
  • Which planned interventions will use the model output?

This exercise often finds more risk than automated feature importance does.

Selection Rules for Intervention-Aware Models

When the model is used to allocate resources, prioritize features that are available before the decision and stable under the intervention.

Good candidates:

  • pre-decision measurements;
  • domain variables linked to plausible mechanisms;
  • historical summaries computed without future leakage;
  • features with consistent meaning across groups and time;
  • variables that remain observable after deployment.

Features to challenge:

  • post-treatment variables;
  • administrative codes that reflect billing or workflow;
  • variables created by the same policy the model may change;
  • high-cardinality identifiers with unclear causal meaning;
  • proxies for sensitive attributes when the use case affects opportunity or access.

Validation Must Match the Causal Question

Random cross-validation can make a model look strong while hiding policy dependence. Better validation designs include:

  • temporal validation across policy periods;
  • group validation across hospitals, sites, regions, or teams;
  • sensitivity analysis after removing decision-proxy features;
  • subgroup calibration for populations affected by the intervention;
  • backtesting against periods where the intervention rule changed.

If a feature is only useful because it captures an old decision process, its value may disappear once the model changes that process.

Practical Checklist

Before approving a feature set, ask:

  1. Is the feature observed before the prediction is made?
  2. Can a user act on the model without changing the meaning of the feature?
  3. Does the feature encode a past decision that may be biased or inconsistent?
  4. Does the feature remain available in production with the same definition?
  5. Does performance survive after removing obvious proxies and leakage candidates?

This checklist will not prove causality, but it prevents many avoidable mistakes.

Conclusion

Causal feature selection is not a replacement for predictive validation. It is a layer of reasoning that makes predictive models safer when decisions and interventions are involved.

The best feature set is not always the one with the highest offline score. It is the one whose relationships remain meaningful when the model is deployed, acted upon, and evaluated in the real system.

References

  • Pearl, J. (2009). Causality (2nd ed.). Cambridge University Press.
  • Hernan, M. A., & Robins, J. M. (2020). Causal Inference: What If. Chapman & Hall/CRC.
  • Guyon, I., & Elisseeff, A. (2003). An introduction to variable and feature selection. Journal of Machine Learning Research, 3, 1157-1182.
  • Veitch, V., et al. (2021). Counterfactual invariance to spurious correlations in text classification. NeurIPS.

You may also enjoy