Our paper, "Split Conformal Prediction under Data Contamination," received the Best Paper Award at the 13th Symposium on Conformal and Probabilistic Prediction with Applications (COPA 2024). The work is co-authored by Sentinel Assurance co-founder Wenkai Xu, together with Jason Clarkson, Mihai Cucuringu, and Gesine Reinert (University of Oxford).
What the paper does
- Split conformal prediction is a widely used, distribution-free method for producing calibrated prediction sets, and its guarantees rely on the calibration data being clean.
- The paper quantifies exactly how contaminated calibration data degrades that guarantee, in both coverage (how often the truth actually lands in the predicted set) and efficiency (how tight the set is).
- Introduces Contamination Robust Conformal Prediction, an adjustment for classification tasks that restores reliable coverage even when a fraction of the calibration data is contaminated.
- Validated on both synthetic and real-world datasets.
Our take
This is directly the kind of guarantee worth caring about: not just whether a model performs well, but whether you can trust the confidence it reports, and under what conditions that trust breaks down. Calibration data is rarely as clean in production as it is in a benchmark, and knowing precisely how much that costs you is the difference between a system that's calibrated and one that just looks calibrated.