ConformalClsRAPS

class torch_uncertainty.post_processing.ConformalClsRAPS(alpha, model=None, randomized=True, penalty=0.1, regularization_rank=1, ts_init_val=1.0, ts_lr=0.1, ts_max_iter=100, enable_ts=False, device=None)

Conformal classification with Regularised Adaptive Prediction Sets (RAPS; Angelopoulos, Bates, Jordan & Malik, 2021).

A regularised variant of ConformalClsAPS that penalises the inclusion of classes with a low predicted rank to produce smaller prediction sets without sacrificing coverage. The non-conformity score adds a rank-based regulariser to the APS score:

\[s(\mathbf{x}, y) = \underbrace{\sum_{i=1}^{k} \hat{p}_{(i)} - U \cdot \hat{p}_{(k)}}_{\text{APS}} + \lambda \cdot (k - k_\text{reg})_{+},\]

where \(k\) is the rank of class \(y\), \(\lambda\) is penalty, \(k_\text{reg}\) is regularization_rank, and \((\cdot)_+ = \max(\cdot, 0)\). Larger \(\lambda\) and smaller \(k_\text{reg}\) produce tighter sets at the cost of a coarser score.

Parameters:

• alpha (float) – Target mis-coverage level \(\alpha \in (0, 1)\).

• model (Module | None) – Trained classification model. Defaults to None.

• randomized (bool) – Whether to use randomised tie-breaking. Defaults to True.

• penalty (float) – Regularisation weight \(\lambda\). Defaults to 0.1.

• regularization_rank (int) – Rank threshold \(k_\text{reg}\) above which the penalty is applied. Defaults to 1.

• ts_init_val (float) – Initial value for the temperature. Defaults to 1.0.

• ts_lr (float) – Learning rate for the temperature scaling optimizer. Defaults to 0.1.

• ts_max_iter (int) – Maximum number of iterations for the temperature scaling optimizer. Defaults to 100.

• enable_ts (bool) – Whether to apply temperature scaling before computing the conformal scores. Defaults to False.

• device (Union[Literal['cpu', 'cuda'], device, None]) – Device to use. Defaults to None.

Warning

This implementation only works in the multiclass setting. Raise an issue if binary support is needed.

Note

Outputs are uniform probability vectors over the selected classes. Empty thresholded sets are replaced by the lowest-score singleton before normalization.

Reference:

• Angelopoulos, A. N., Bates, S., Jordan, M., & Malik, J. (2021). Uncertainty Sets for Image Classifiers using Conformal Prediction. ICLR 2021.

Code inspired by TorchCP.

conformal(inputs)

Compute the prediction set for each input.

Return type:

Tensor

Returns:

A probability vector for each sample. Its support is the conformal prediction set, with equal probability assigned to every included class. If thresholding produces an empty prediction set, the class with the lowest non-conformity score is included before normalization. Consequently, every returned row is finite and sums to one.

fit(dataloader)

Calibrate the APS threshold q_hat on a calibration set.

Return type:

None

model_forward(inputs)

Apply the model and return the scores.

Return type:

Tensor