ICML 2026 Tutorial

Calibration, Decisions, and Collaboration in Learning

A tutorial on calibration, decision calibration, multicalibration, and related ideas in learning, downstream decision-making, and information aggregation.

Aaron Roth University of Pennsylvania Natalie Collina University of Pennsylvania / MIT Ira Globus-Harris Cornell University
Date and time Mon, Jul 6, 2026 1:30 PM-4:00 PM KST
Location Auditorium ICML 2026
Conference page ICML listing Schedule and live content
Calibration curve sketch A reliability diagram with a diagonal reference line and a calibrated curve approaching it. Predicted probability Empirical frequency

Abstract

Probabilities that downstream users can trust

In this tutorial we will learn about a powerful framework to make probabilistic predictions in ways that "look like real probabilities" in all of the ways that matter for downstream applications. We'll see how to do this efficiently even in difficult, adversarial environments, and then focus on two concrete applications.

First we'll see how to make predictions that are "trustworthy" for downstream decision makers. Many downstream decision makers, each with different objectives and actions, will be able to act optimally as if our predictions are correct, and get strong guarantees about their performance. Next, we'll see how to make predictions that allow for efficient collaboration between two differently informed parties, like an AI and a human user, who can't easily share their observations, while still obtaining the complementary benefits of their individual knowledge. We'll end with a quick survey of many other applications of this technique.

Tutorial Roadmap

From empirical probability tests to decisions and collaboration

The tutorial follows one thread: identify the empirical properties of probabilities that make them useful, learn forecasts satisfying those properties, and use them for decisions and collaboration between agents with different information.

1

Why Calibration?

Individual probabilities are hard to verify directly, so we start with auditable empirical tests that real probabilities should pass.

Foundations Metrics
2

Multicalibration and Outcome Indistinguishability

Calibration becomes much more useful when it holds on rich, overlapping subsequences defined by context, groups, benchmarks, or interaction history.

Algorithms Complexity
3

Efficient Online Algorithms

The minimax and approachability viewpoint turns a large family of calibration-style constraints into online learning algorithms with adversarial guarantees.

Interactive demo Online multicalibration
5

Agreement and Information Aggregation

Conditional calibration also gives a tractable way to reason about communication between agents with different information, replacing idealized Bayesian common-prior assumptions with enforceable behavioral conditions.

Agreement readings Reference class readings

Interactive

Stylized Sepsis Decision Game

A companion to the calibration and multicalibration part of the tutorial: predict sepsis risk online, let simple treatment rules act on the forecasts, and watch calibration error and downstream regret evolve.

Disclaimer: We are not doctors. Do not use this to predict Sepsis!

Round 1
Auto patients

The forecaster predicts whether a patient will develop sepsis. Clinicians use the forecast to decide when to treat. The main score is how much the worst clinician regrets trusting the forecasts; lower is better.

Try first: press Run 50, then switch the algorithm goal and compare worst clinician regret with calibration error.

Forecaster character
Forecaster chooses a randomized sepsis-risk forecast
Adversary character
Adversary chooses sepsis or no sepsis from the history

Patient pattern guide
  • Manual Choose sepsis or no sepsis yourself each day.
  • No sepsis The outcome is no sepsis every day.
  • Always sepsis The outcome is sepsis every day.
  • ICU high risk, ward low risk ICU patients develop sepsis; ward patients do not.
  • Unstable vitals high risk Patients with unstable vitals develop sepsis; patients with stable vitals do not.
  • Alternating Outcomes alternate day by day, starting with sepsis.
  • Condition shift at round 25 The first 25 days have no sepsis; later days develop sepsis.
  • React to last forecast The first day has no sepsis; later days develop sepsis if the previous sampled forecast was below 55%.
  • Contrarian Sepsis occurs when the current expected forecast is at most 50%, and does not otherwise.

Next patient context

Use Next patient to watch the context change; batch runs jump through many contexts.

Sepsis-risk forecast

Hidden

No completed rounds yet

Clinicians being evaluated

Each clinician follows a simple treatment rule on the days when they are active.

Materials

Lecture notes and tutorial materials

Tutorial slides and comprehensive lecture notes are available for download.

Tutorial slides

Final ICML 2026 tutorial slides for Calibration, Decisions, and Collaboration in Learning.

Download slides PDF Download slides PowerPoint

Comprehensive lecture notes

Uncertain: Modern Topics in Uncertainty Estimation is a working draft covering calibration, multicalibration, conformal prediction, distribution shift, omniprediction, and related topics.

Download notes PDF Browse readings

Recording

A recording link will be added here after ICML posts the tutorial video.

ICML tutorial page

Resources

Reading List

Papers are grouped by theme and annotated with a brief description of their main contribution. Public versions, including arXiv and open proceedings pages, are linked when available.

Foundations of Calibration

  • The Well-Calibrated Bayesian

    Authors: A. P. Dawid. Publication: Journal of the American Statistical Association, 1982.

    Formalizes calibration for sequential probability forecasts and proves that a coherent Bayesian expects their forecasts to be well calibrated.

  • Calibration-Based Empirical Probability

    Authors: A. P. Dawid. Publication: Annals of Statistics, 1985.

    Develops a calibration-based account of empirical probability, using long-run forecasting behavior to ground frequency-style probability assessments.

  • Calibrated Learning and Correlated Equilibrium

    Authors: Dean Foster, Rakesh Vohra. Publication: Games and Economic Behavior, 1997.

    Shows that players who best respond to calibrated forecasts generate play whose limit points are correlated equilibria.

  • Asymptotic Calibration

    Authors: Dean Foster, Rakesh Vohra. Publication: Biometrika, 1998.

    Shows that randomized forecasting can guarantee asymptotic calibration against arbitrary outcome sequences, while deterministic forecasting cannot.

  • Deterministic Calibration and Nash Equilibrium

    Authors: Sham Kakade, Dean Foster. Publication: COLT 2004; Journal of Computer and System Sciences, 2008.

    Introduces weak calibration, a deterministic Lipschitz-test-function relaxation later shown essentially equivalent to smooth calibration, and uses it for Nash-equilibrium learning dynamics.

  • Smooth Calibration, Leaky Forecasts, Finite Recall, and Nash Dynamics

    Authors: Dean Foster, Sergiu Hart. Publication: Games and Economic Behavior, 2018.

    Introduces smooth calibration, showing it can be achieved deterministically with leaked finite-memory forecasts and yields approximate Nash play in repeated games.

  • Multicalibration: Calibration for the Computationally-Identifiable Masses

    Authors: Ursula Hebert-Johnson, Michael Kim, Omer Reingold, Guy Rothblum. Publication: ICML 2018.

    Introduces multicalibration and multiaccuracy, gives algorithms for learning multicalibrated predictors, and connects the task to agnostic learning.

Multiaccuracy, Relaxations, and Subgroup Auditing

  • Multiaccuracy: Black-Box Post-Processing for Fairness in Classification

    Authors: Michael P. Kim, Amirata Ghorbani, James Zou. Publication: AIES 2019.

    Develops multiaccuracy auditing and black-box post-processing as a practical way to remove systematic subgroup residual bias without retraining or white-box access to the original predictor.

  • Low-Degree Multicalibration

    Authors: Parikshit Gopalan, Michael Kim, Mihir Singhal, Shengjia Zhao. Publication: COLT 2022.

    Defines low-degree multicalibration as an efficient hierarchy between multiaccuracy and full multicalibration that retains key fairness and accuracy guarantees.

  • How Global Calibration Strengthens Multiaccuracy

    Authors: Silvia Casacuberta, Parikshit Gopalan, Varun Kanade, Omer Reingold. Publication: FOCS 2025.

    Clarifies the hierarchy between multiaccuracy, calibrated multiaccuracy, and multicalibration by showing that global calibration makes multiaccuracy much more powerful.

  • Multiaccuracy and Multicalibration via Proxy Groups

    Authors: Beepul Bharti, Mary Versa Clemens-Sewall, Paul H. Yi, Jeremias Sulam. Publication: ICML 2025.

    Shows how proxy-sensitive attributes can upper-bound and help mitigate multiaccuracy and multicalibration violations when true sensitive-group labels are missing.

Fairness, Risk Scores, and Subgroup Guarantees

  • Inherent Trade-Offs in the Fair Determination of Risk Scores

    Authors: Jon Kleinberg, Sendhil Mullainathan, Manish Raghavan. Publication: ITCS 2017.

    Establishes incompatibility results showing that natural fairness requirements for risk scores, including calibration, cannot generally be simultaneously satisfied.

  • Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments

    Authors: Alexandra Chouldechova. Publication: Big Data, 2017.

    Analyzes calibration and disparate impact in recidivism prediction, giving an influential statistical account of fairness tradeoffs.

  • On Fairness and Calibration

    Authors: Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, Kilian Q. Weinberger. Publication: NeurIPS 2017.

    Shows that calibrated probability estimates are compatible with only limited error-rate parity constraints, highlighting a sharp tension between calibration and error-disparity goals.

Multicalibration Algorithms, Extensions, and Complexity

  • Sample Complexity of Uniform Convergence for Multicalibration

    Authors: Eliran Shabat, Lee Cohen, Yishay Mansour. Publication: NeurIPS 2020.

    Gives uniform-convergence sample-complexity bounds that control population multicalibration error from empirical multicalibration error independently of prediction error.

  • Moment Multicalibration for Uncertainty Estimation

    Authors: Christopher Jung, Changhwa Lee, Mallesh M. Pai, Aaron Roth, Rakesh Vohra. Publication: COLT 2021.

    Extends multicalibration from means to higher moments, enabling calibrated uncertainty estimates such as variances.

  • Online Minimax Multiobjective Optimization: Multicalibeating and Other Applications

    Authors: Daniel D. Lee, Georgy Noarov, Mallesh Pai, Aaron Roth. Publication: NeurIPS 2022.

    Develops a vector-valued online optimization framework that recovers multicalibration guarantees and enables multicalibeating against collections of forecasters.

  • Multicalibrated Partitions for Importance Weights

    Authors: Parikshit Gopalan, Omer Reingold, Vatsal Sharan, Udi Wieder. Publication: ALT 2022.

    Introduces multicalibrated partitions to compute importance weights from samples with set-wise sandwiching guarantees.

  • Calibeating: Beating Forecasters at Their Own Game

    Authors: Dean P. Foster, Sergiu Hart. Publication: Theoretical Economics, 2023.

    Introduces calibeating, showing how to post-process external forecasts online to improve calibration without losing refinement or Brier-score performance.

  • Multicalibrated Regression for Downstream Fairness

    Authors: Ira Globus-Harris, Varun Gupta, Christopher Jung, Michael Kearns, Jamie Morgenstern, Aaron Roth. Publication: AIES 2023; arXiv.

    Shows how a multicalibrated regression function can be post-processed without labeled data to solve many downstream fairness-constrained classification tasks.

  • The Statistical Scope of Multicalibration

    Authors: Georgy Noarov, Aaron Roth. Publication: ICML 2023.

    Characterizes which continuous scalar distributional properties can be multicalibrated by showing that multicalibration is possible exactly for elicitable properties.

  • A Unifying Perspective on Multi-Calibration: Game Dynamics for Multi-Objective Learning

    Authors: Nika Haghtalab, Michael I. Jordan, Eric Zhao. Publication: NeurIPS 2023.

    Places multicalibration in a multi-objective learning framework and uses game dynamics to simplify analyses and improve guarantees for several variants.

  • Improved and Oracle-Efficient Online L1-Multicalibration

    Authors: Rohan Ghuge, Vidya Muthukumar, Sahil Singla. Publication: ICML 2025.

    Gives direct online L1-multicalibration algorithms with improved rates and an oracle-efficient variant for large or infinite group families.

  • Discretization-free Multicalibration through Loss Minimization over Tree Ensembles

    Authors: Hongyi Henry Jin, Zijun Ding, Dung Daniel Ngo, Zhiwei Steven Wu. Publication: NeurIPS 2025.

    Gives an ERM-based, discretization-free multicalibration method over depth-two tree ensembles that can be implemented with off-the-shelf gradient-boosted tree methods.

  • Optimal Lower Bounds for Online Multicalibration

    Authors: Natalie Collina, Jiuyao Lu, Georgy Noarov, Aaron Roth. Publication: FOCS 2026.

    Establishes the minimax rate for online multicalibration by proving lower bounds that match existing upper bounds up to logarithmic factors.

  • Calibeating Made Simple

    Authors: Yurong Chen, Zhiyi Huang, Michael I. Jordan, Haipeng Luo. Publication: arXiv, 2026.

    Reduces calibeating and multi-calibeating to standard regret-minimization problems, yielding optimal rates for broad classes of proper losses.

  • An Efficient Black-Box Reduction from Online Learning to Multicalibration, and a New Route to Phi-Regret Minimization

    Authors: Gabriele Farina, Juan Carlos Perdomo. Publication: arXiv, 2026.

    Gives a GGM-style black-box reduction showing that online multicalibration can be achieved by combining a no-regret learner over test functions with an expected variational inequality solver, yielding oracle-efficient sqrt(T)-type guarantees.

  • The Sample Complexity of Multicalibration

    Authors: Natalie Collina, Jiuyao Lu, Georgy Noarov, Aaron Roth. Publication: arXiv, 2026.

    Establishes the minimax sample complexity of batch multicalibration, showing that epsilon^{-3} samples are necessary and sufficient up to polylogarithmic factors for polynomial-size group families.

  • Instance-Adaptive Online Multicalibration

    Authors: Zhiming Huang, Jamie Morgenstern, Aaron Roth, Claire Jie Zhang. Publication: arXiv, 2026.

    Gives an efficient online multicalibration algorithm that recovers worst-case-optimal rates while adapting to easier stochastic and piecewise-stationary instances.

  • Adaptive Calibration in Non-Stationary Environments

    Authors: Junyan Liu, Haipeng Luo, Lillian J. Ratliff. Publication: arXiv, 2026.

    Develops online calibration algorithms whose guarantees adapt to the degree of non-stationarity, interpolating between stationary, piecewise-stationary, and adversarial regimes under multiple calibration measures.

  • Optimal Deterministic Multicalibration and Omniprediction

    Authors: Georgy Noarov, Aaron Roth. Publication: arXiv, 2026.

    Gives minimax-optimal deterministic algorithms for multicalibration and outcome indistinguishability, yielding deterministic omnipredictors and panpredictors with optimal sample complexity.

Adaptation, Distribution Shift, and Causality

Multivalid Conformal Prediction and Prediction Intervals

  • Online Multivalid Learning: Means, Moments, and Prediction Intervals

    Authors: Varun Gupta, Christopher Jung, Georgy Noarov, Mallesh M. Pai, Aaron Roth. Publication: ITCS 2022.

    Introduces adversarial online multivalid learning for means, moments, and prediction intervals, giving subgroup-conditional uncertainty guarantees for black-box predictors.

  • Practical Adversarial Multivalid Conformal Prediction

    Authors: Osbert Bastani, Varun Gupta, Christopher Jung, Georgy Noarov, Ramya Ramalingam, Aaron Roth. Publication: NeurIPS 2022.

    Gives a lightweight sequential conformal method with threshold-conditional and subgroup-conditional empirical coverage guarantees without a held-out calibration set.

  • Batch Multivalid Conformal Prediction

    Authors: Christopher Jung, Georgy Noarov, Ramya Ramalingam, Aaron Roth. Publication: ICLR 2023.

    Develops batch conformal algorithms whose coverage guarantees hold simultaneously conditional on group membership and nonconformity threshold.

Omniprediction and Loss Minimization

  • Outcome Indistinguishability

    Authors: Cynthia Dwork, Michael Kim, Omer Reingold, Guy Rothblum, Gal Yona. Publication: STOC 2021.

    Introduces outcome indistinguishability, requiring a predictor-induced generative model of outcomes to be hard for efficient tests to refute from real observations.

  • Omnipredictors

    Authors: Parikshit Gopalan, Adam Tauman Kalai, Omer Reingold, Vatsal Sharan, Udi Wieder. Publication: ITCS 2022.

    Introduces omniprediction as loss-oblivious learning, where one predictor can be post-processed to compete with a hypothesis class for many downstream losses.

  • Loss Minimization through the Lens of Outcome Indistinguishability

    Authors: Parikshit Gopalan, Lunjia Hu, Michael Kim, Omer Reingold, Udi Wieder. Publication: ITCS 2023; arXiv.

    Shows that loss outcome indistinguishability implies omniprediction and can be decomposed into calibration plus calibrated multiaccuracy.

  • When Does Optimizing a Proper Loss Yield Calibration?

    Authors: Jaroslaw Blasiok, Parikshit Gopalan, Lunjia Hu, Preetum Nakkiran. Publication: NeurIPS 2023.

    Shows that local optimality for proper-loss minimization under Lipschitz post-processing is equivalent, up to approximation, to smooth calibration.

  • Swap Agnostic Learning, or Characterizing Omniprediction via Multicalibration

    Authors: Parikshit Gopalan, Michael Kim, Omer Reingold. Publication: NeurIPS 2023.

    Introduces swap agnostic learning and proves equivalences among swap omniprediction, multicalibration variants, and outcome indistinguishability notions.

  • Omnipredictors for Constrained Optimization

    Authors: Lunjia Hu, Inbal Livni Navon, Omer Reingold, Chutong Yang. Publication: ICML 2023.

    Extends omniprediction to settings where both the downstream loss and constraints are unknown at learning time but later imposed through known subpopulations.

  • Multicalibration as Boosting for Regression

    Authors: Ira Globus-Harris, Declan Harrison, Michael Kearns, Aaron Roth, Jessica Sorrell. Publication: ICML 2023.

    Characterizes multicalibration through a squared-error swap-regret condition and gives a regression-oracle boosting algorithm with agnostic guarantees.

  • Loss Minimization Yields Multicalibration for Large Neural Networks

    Authors: Jaroslaw Blasiok, Parikshit Gopalan, Lunjia Hu, Adam Tauman Kalai, Preetum Nakkiran. Publication: ITCS 2024.

    Shows that minimizing squared loss over sufficiently expressive neural networks can imply multicalibration with respect to smaller neural-network group classes.

  • Oracle Efficient Online Multicalibration and Omniprediction

    Authors: Sumegha Garg, Christopher Jung, Omer Reingold, Aaron Roth. Publication: SODA 2024.

    Develops oracle-efficient online multicalibration for infinite benchmark classes and derives the first oracle-efficient online omnipredictor.

  • Near-Optimal Algorithms for Omniprediction

    Authors: Princewill Okoroafor, Robert Kleinberg, Michael P. Kim. Publication: FOCS 2025.

    Gives near-optimal online and offline omniprediction algorithms whose online regret nearly matches single-loss learning rates.

  • Sample Efficient Omniprediction and Downstream Swap Regret for Non-Linear Losses

    Authors: Jiuyao Lu, Aaron Roth, Mirah Shi. Publication: arXiv, 2025.

    Defines decision swap regret, generalizing omniprediction and downstream swap regret, and gives online and batch algorithms for multidimensional nonlinear Lipschitz losses.

  • Improved Bounds for Swap Multicalibration and Swap Omniprediction

    Authors: Haipeng Luo, Spandan Senapati, Vatsal Sharan. Publication: NeurIPS 2025.

    Improves online swap multicalibration and swap omniprediction rates and translates them into better distributional sample-complexity bounds.

  • Online Omniprediction with Long-Term Constraints

    Authors: Yahav Bechavod, Jiuyao Lu, Aaron Roth. Publication: COLT 2026.

    Introduces online omniprediction with long-term constraints, where one prediction stream lets many downstream agents obtain no-regret guarantees while satisfying cumulative constraints.

  • Efficient Swap Multicalibration of Elicitable Properties

    Authors: Lunjia Hu, Haipeng Luo, Spandan Senapati, Vatsal Sharan. Publication: arXiv, 2025.

    Extends swap multicalibration from means to general elicitable properties and gives oracle-efficient online algorithms with improved rates.

Pseudorandomness and Complexity Connections

  • From Pseudorandomness to Multi-Group Fairness and Back

    Authors: Cynthia Dwork, Daniel Lee, Huijia Lin, Pranay Tankala. Publication: COLT 2023.

    Connects multi-group fairness to pseudorandomness through statistical-distance variants of multicalibration, yielding algorithms and a real-valued hardcore lemma.

  • Complexity-Theoretic Implications of Multicalibration

    Authors: Silvia Casacuberta, Cynthia Dwork, Salil Vadhan. Publication: STOC 2024.

    Uses multicalibration to strengthen regularity-lemma applications including hardcore lemmas, dense-model theorems, and conditional pseudo-min-entropy equivalences.

Decision Calibration, Calibration Metrics, and Downstream Decision-Making

  • Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration

    Authors: Shengjia Zhao, Michael P. Kim, Roshni Sahoo, Tengyu Ma, Stefano Ermon. Publication: NeurIPS 2021.

    Introduces decision calibration for multiclass prediction, replacing infeasible distribution calibration with indistinguishability to bounded-action downstream decision-makers.

  • A Unifying Theory of Distance from Calibration

    Authors: Jaroslaw Blasiok, Parikshit Gopalan, Lunjia Hu, Preetum Nakkiran. Publication: STOC 2023.

    Defines distance to the nearest calibrated predictor as a ground-truth calibration metric and identifies efficiently estimable calibration measures consistent with it.

  • U-Calibration: Forecasting for an Unknown Agent

    Authors: Bobby Kleinberg, Renato Paes Leme, Jon Schneider, Yifeng Teng. Publication: COLT 2023.

    Introduces U-calibration as the maximum regret of forecasts under any bounded scoring rule, showing it is necessary and sufficient for all unknown agents to obtain sublinear regret.

  • On the Distance from Calibration in Sequential Prediction

    Authors: Mingda Qiao, Letian Zheng. Publication: COLT 2024.

    Extends distance to calibration to sequential prediction and nonconstructively establishes the existence of an online forecasting algorithm with O(sqrt(T)) adversarial distance to calibration.

  • On Computationally Efficient Multi-Class Calibration

    Authors: Parikshit Gopalan, Lunjia Hu, Guy N. Rothblum. Publication: COLT 2024.

    Introduces projected smooth calibration for multiclass prediction, giving polynomial-in-k recalibration algorithms while proving barriers for stronger multiclass calibration notions.

  • Forecasting for Swap Regret for All Downstream Agents

    Authors: Aaron Roth, Mirah Shi. Publication: EC 2024.

    Shows that event-unbiased forecasts can give all downstream best-responding agents low swap regret with better dimension dependence than calibrated forecasts.

  • Calibration Error for Decision Making

    Authors: Lunjia Hu, Yifan Wu. Publication: FOCS 2024.

    Introduces Calibration Decision Loss as the worst-case downstream decision-payoff improvement from recalibrating forecasts and gives an efficient online algorithm with near-optimal expected error.

  • Truthfulness of Calibration Measures

    Authors: Nika Haghtalab, Mingda Qiao, Kunhe Yang, Eric Zhao. Publication: NeurIPS 2024.

    Shows that many existing calibration measures are not truthful and introduces subsampled smooth calibration error as a more incentive-compatible alternative.

  • An Elementary Predictor Obtaining 2 sqrt(T) + 1 Distance to Calibration

    Authors: Eshwar Ram Arunachaleswaran, Natalie Collina, Aaron Roth, Mirah Shi. Publication: SODA 2025.

    Gives a simple deterministic online predictor achieving distance to calibration at most 2 sqrt(T) + 1 in the adversarial setting.

  • High-Dimensional Prediction for Sequential Decision Making

    Authors: Georgy Noarov, Ramya Ramalingam, Aaron Roth, Stephan Xie. Publication: ICML 2025.

    Gives efficient high-dimensional event-unbiased forecasts that support downstream decision-makers, conditional regret guarantees, and online multicalibration rates.

  • Truthfulness of Decision-Theoretic Calibration Measures

    Authors: Mingda Qiao, Eric Zhao. Publication: COLT 2025.

    Introduces a decision-theoretic calibration measure that is both truthful and useful for downstream no-regret decision-making, while proving limits for such measures.

  • Smooth Calibration and Decision Making

    Authors: Jason Hartline, Yifan Wu, Yunran Yang. Publication: FORC 2025.

    Shows how post-processing low distance-to-calibration forecasts can achieve decision-theoretic calibration measures such as ECE and calibration decision loss.

  • A Perfectly Truthful Calibration Measure

    Authors: Jason Hartline, Lunjia Hu, Yifan Wu. Publication: arXiv, 2025.

    Introduces averaged two-bin calibration error, a perfectly truthful batch calibration measure that is sound, complete, continuous, and efficiently computable.

  • Auditability and the Landscape of Distance to Multicalibration

    Authors: Nathan Derhake, Siddartha Devic, Dutch Hansen, Kuan Liu, Vatsal Sharan. Publication: ITCS 2026.

    Studies metrics for distance to multicalibration, showing that natural dCE-style generalizations fail either auditability or geometric desiderata and proposing auditable alternatives.

Agreement, Collaboration, and Information Aggregation

  • Tractable Agreement Protocols

    Authors: Natalie Collina, Surbhi Goel, Varun Gupta, Aaron Roth. Publication: STOC 2025.

    Turns learning algorithms into efficient agreement protocols using tractable calibration relaxations of Bayesian rationality.

  • Collaborative Prediction: Tractable Information Aggregation via Agreement

    Authors: Natalie Collina, Ira Globus-Harris, Surbhi Goel, Varun Gupta, Aaron Roth, Mirah Shi. Publication: SODA 2026.

    Gives efficient collaboration protocols that aggregate predictions from parties with different features without requiring either party to reveal those features.

  • Networked Information Aggregation via Machine Learning

    Authors: Michael Kearns, Aaron Roth, Emily Ryu. Publication: SODA 2026.

    Studies distributed learning over DAGs of agents who observe different features and their parents' predictions, identifying graph depth as the key parameter governing when information aggregation is possible.

Reference Class Problem and Individual Probabilities

  • The Logic of Chance

    Authors: John Venn. Publication: Macmillan, 1866; linked public scan of 1888 edition.

    Gives an early frequency-theoretic account of probability and foreshadows the reference class problem by emphasizing that individual cases can be grouped in many different ways.

  • The Theory of Probability

    Authors: Hans Reichenbach. Publication: University of California Press, 1949; second edition 1971.

    Introduces the modern framing of the reference class problem for single-case probabilities and motivates choosing the most specific reference class supported by reliable statistics.

  • Objectively Homogeneous Reference Classes

    Authors: Wesley C. Salmon. Publication: Synthese, 1977.

    Proposes objective homogeneity as a criterion for legitimate reference classes: no further statistically relevant partition should change the conditional probability.

  • The Reference Class Problem Is Your Problem Too

    Authors: Alan Hajek. Publication: Synthese, 2007.

    Argues that the reference class problem is not merely a frequentist difficulty, but recurs across major interpretations of probability, including Bayesian prior choice.

  • On Individual Risk

    Authors: A. Philip Dawid. Publication: Synthese, 2017.

    Surveys interpretations of individual risk, distinguishing group-to-individual and individual-to-group approaches while explaining why individual probabilities remain elusive but pragmatically useful.

  • Reconciling Individual Probability Forecasts

    Authors: Aaron Roth, Alexander Tolbert, Scott Weinstein. Publication: FAccT 2023.

    Introduces model reconciliation, showing that substantial disagreement between two individual-probability models yields a reference class that can empirically falsify and improve at least one of them.

  • Resolving the Reference Class Problem at Scale

    Authors: Aaron Roth, Alexander Williams Tolbert. Publication: Philosophy of Science, 2025.

    Distinguishes the individual reference class problem from the at-scale version faced by machine-learning systems, arguing that multicalibration-style statistical tools can mitigate the latter.

Applications and Empirical Studies

  • When is Multicalibration Post-Processing Necessary?

    Authors: Dutch Hansen, Siddartha Devic, Preetum Nakkiran, Vatsal Sharan. Publication: NeurIPS 2024.

    Empirically maps when multicalibration post-processing helps, finding that already calibrated models are often relatively multicalibrated while uncalibrated large models can benefit.

  • Multicalibration for Confidence Scoring in LLMs

    Authors: Gianluca Detommaso, Martin Andres Bertran, Riccardo Fogliato, Aaron Roth. Publication: ICML 2024.

    Applies multicalibration to LLM confidence scoring using embedding clusters and self-annotation groups, with algorithmic variants that reduce overfitting.

  • MCGrad: Multicalibration at Web Scale

    Authors: Niek Tax, Lorenzo Perini, Fridolin Linder, Daniel Haimovich, Dima Karamshuk, Nastaran Okati, Milan Vojnovic, Pavlos Athanasios Apostolopoulos. Publication: KDD 2026.

    Proposes MCGrad, a scalable multicalibration algorithm that avoids manually specifying subgroups and has been deployed in large-scale production systems.

Practical Calibration in Machine Learning