SCODAUGRC#

class torch_uncertainty.metrics.classification.SCODAUGRC(ood_cost=0.5, **kwargs)[source]#

Calculate the Area Under the Generalized SCOD Risk-Coverage curve.

This metric applies the generalized risk-coverage construction of AUGRC to the joint SCOD loss defined by SCODAURC.

Let \(r(\kappa)\) denote the SCOD selective risk at coverage \(\kappa\). The generalized SCOD risk is \(\kappa r(\kappa)\), and its area is

\[\operatorname{SCOD\text{-}AUGRC} = \int_0^1 \kappa r(\kappa)\,\mathrm{d}\kappa.\]

At empirical coverage \(\kappa_k=k/N\), this uses

\[\kappa_k r(\kappa_k) = \frac{1}{N}\sum_{j=1}^{k}\ell_{\sigma(j)},\]

where \(\ell_i\) is the SCOD loss and \(\sigma\) orders samples from lowest to highest OOD score.

As input to forward and update, the metric accepts:

ood_scores (Tensor): Float tensor containing OOD scores, where larger values indicate more OOD-like samples.
classification_errors (Tensor): Boolean or binary tensor indicating misclassified ID samples.
is_ood (Tensor): Boolean or binary tensor indicating OOD samples.

As output to forward and compute, the metric returns:

scod_augrc (Tensor): Scalar tensor containing the area under the generalized SCOD risk-coverage curve. Lower values are better.

Parameters:

ood_cost (float) – Relative cost \(c_{\mathrm{OOD}}\) of accepting an OOD sample. The cost of accepting a misclassified ID sample is 1 - ood_cost. Defaults to 0.5.
kwargs – Additional keyword arguments passed to torchmetrics.Metric.

Note

SCOD-AUGRC is the Torch-Uncertainty generalized risk-coverage metric applied to the SCOD loss. It is a natural SCOD extension of AUGRC, but it is not introduced under this name in the original SCOD papers.

References

[1] Xia & Bouganis. Augmenting Softmax Information for Selective Classification with Out-of-Distribution Data. ACCV, 2022..

[2] Narasimhan et al. Plugin Estimators for Selective Classification with Out-of-Distribution Detection..

[3] Traub et al. Overcoming Common Flaws in the Evaluation of Selective Classification Systems..