ConformalClsAPS#

class torch_uncertainty.post_processing.ConformalClsAPS(alpha, model=None, randomized=True, ts_init_val=1, ts_lr=0.1, ts_max_iter=100, enable_ts=True, device=None)[source]#

Conformal classification with Adaptive Prediction Sets (APS; Romano, Sesia & Candès, NeurIPS 2020).

Uses as non-conformity score the cumulative probability mass needed to reach the true class once the predictions are sorted by decreasing probability. For a sample with predicted probabilities \(\hat{\mathbf{p}}\) whose true class ranks at position \(k\) after sorting,

\[s(\mathbf{x}, y) = \sum_{i=1}^{k} \hat{p}_{(i)} - U \cdot \hat{p}_{(k)},\]

where \(U \sim \mathrm{Uniform}(0, 1)\) smooths the cumulative score when randomized is True. The calibrated quantile \(\hat{q}\) defines the test-time prediction set

\[\mathcal{C}(\mathbf{x}) = \{ c : s(\mathbf{x}, c) \leq \hat{q} \},\]

which adapts in size to the difficulty of each example: easy points get tight sets, ambiguous points get larger ones.

Parameters:

alpha (float) – Target mis-coverage level \(\alpha \in (0, 1)\).
model (Module | None) – Trained classification model. Defaults to None.
randomized (bool) – Whether to use randomised tie-breaking in APS. Defaults to True.
ts_init_val (float) – Initial value for the temperature. Defaults to 1.0.
ts_lr (float) – Learning rate for the temperature scaling optimizer. Defaults to 0.1.
ts_max_iter (int) – Maximum number of iterations for the temperature scaling optimizer. Defaults to 100.
enable_ts (bool) – Whether to apply temperature scaling to the logits before computing the conformal scores. Defaults to True.
device (Union[Literal['cpu', 'cuda'], device, None]) – Device to use. Defaults to None.

Warning

This implementation only works in the multiclass setting. Raise an issue if binary support is needed.

Reference:

Romano, Y., Sesia, M., & Candès, E. (2020). Classification with Valid and Adaptive Coverage. NeurIPS 2020.

Code inspired by TorchCP.

conformal(inputs)[source]#

Compute the prediction set for each input.

Return type:: Tensor
Returns:: A probability vector for each sample. Its support is the conformal prediction set, with equal probability assigned to every included class. If thresholding produces an empty prediction set, the class with the lowest non-conformity score is included before normalization. Consequently, every returned row is finite and sums to one.

fit(dataloader)[source]#

Calibrate the APS threshold q_hat on a calibration set.

Return type:: None

model_forward(inputs)[source]#

Apply the model and return the scores.

Return type:: Tensor