ConformalClsTHR#

class torch_uncertainty.post_processing.ConformalClsTHR(alpha, model=None, ts_init_val=1.0, ts_lr=0.1, ts_max_iter=100, enable_ts=True, device=None)[source]#

Threshold-based conformal classifier (THR; Sadinle et al., 2019).

The simplest conformal classification rule. Defines the non-conformity score of class \(c\) as \(s(\mathbf{x}, c) = 1 - \hat{p}_c(\mathbf{x})\), calibrates the empirical \((1 - \alpha)\)-quantile \(\hat{q}\) of the scores at the true class on a held-out calibration set, and at test time outputs the prediction set

\[\mathcal{C}(\mathbf{x}) = \{ c : \hat{p}_c(\mathbf{x}) \geq 1 - \hat{q} \},\]

guaranteeing marginal coverage of \(1 - \alpha\). The top-1 class is always included to avoid empty sets. Probabilities can optionally be calibrated first via temperature scaling (enable_ts=True), which usually yields smaller prediction sets.

Parameters:

  • alpha (float) – Target mis-coverage level \(\alpha \in (0, 1)\).

  • model (Module | None) – Model to be calibrated. Defaults to None.

  • ts_init_val (float) – Initial value for the temperature. Defaults to 1.0.

  • ts_lr (float) – Learning rate for the temperature scaling optimizer. Defaults to 0.1.

  • ts_max_iter (int) – Maximum number of iterations for the temperature scaling optimizer. Defaults to 100.

  • enable_ts (bool) – Whether to scale the logits via temperature scaling before computing the conformal scores. Defaults to True.

  • device (Union[Literal['cpu', 'cuda'], device, None]) – Device to use. Defaults to None.

Warning

This implementation only works in the multiclass setting. Raise an issue if binary support is needed.

Reference:

Code inspired by TorchCP.

conformal(inputs)[source]#

Perform conformal prediction on the test set.

Return type:

Tensor

model_forward(inputs)#

Apply the model and return the scores.

Return type:

Tensor