ConformalClsAPS#

class torch_uncertainty.post_processing.ConformalClsAPS(alpha, model=None, randomized=True, ts_init_val=1, ts_lr=0.1, ts_max_iter=100, enable_ts=True, device=None)[source]#

Conformal classification with Adaptive Prediction Sets (APS; Romano, Sesia & Candès, NeurIPS 2020).

Uses as non-conformity score the cumulative probability mass needed to reach the true class once the predictions are sorted by decreasing probability. For a sample with predicted probabilities \(\hat{\mathbf{p}}\) whose true class ranks at position \(k\) after sorting,

\[s(\mathbf{x}, y) = \sum_{i=1}^{k} \hat{p}_{(i)} - U \cdot \hat{p}_{(k)},\]

where \(U \sim \mathrm{Uniform}(0, 1)\) smooths the cumulative score when randomized is True. The calibrated quantile \(\hat{q}\) defines the test-time prediction set

\[\mathcal{C}(\mathbf{x}) = \{ c : s(\mathbf{x}, c) \leq \hat{q} \},\]

which adapts in size to the difficulty of each example: easy points get tight sets, ambiguous points get larger ones.

Parameters:

alpha (float) – Target mis-coverage level \(\alpha \in (0, 1)\).
model (Module | None) – Trained classification model. Defaults to None.
randomized (bool) – Whether to use randomised tie-breaking in APS. Defaults to True.
ts_init_val (float) – Initial value for the temperature. Defaults to 1.0.
ts_lr (float) – Learning rate for the temperature scaling optimizer. Defaults to 0.1.
ts_max_iter (int) – Maximum number of iterations for the temperature scaling optimizer. Defaults to 100.
enable_ts (bool) – Whether to apply temperature scaling to the logits before computing the conformal scores. Defaults to True.
device (Union[Literal['cpu', 'cuda'], device, None]) – Device to use. Defaults to None.

Warning

This implementation only works in the multiclass setting. Raise an issue if binary support is needed.

Reference:

Romano, Y., Sesia, M., & Candès, E. (2020). Classification with Valid and Adaptive Coverage. NeurIPS 2020.

Code inspired by TorchCP.

conformal(inputs)[source]#

Compute the prediction set for each input.

Return type:: Tensor

fit(dataloader)[source]#

Calibrate the APS threshold q_hat on a calibration set.

Return type:: None

model_forward(inputs)[source]#

Apply the model and return the scores.

Return type:: Tensor