SCODAURC

class torch_uncertainty.metrics.classification.SCODAURC(ood_cost=0.5, **kwargs)

Calculate the Area Under the SCOD Risk-Coverage curve.

Selective Classification with Out-of-Distribution Detection (SCOD) evaluates a classifier and a rejection rule jointly. Unlike ordinary OOD detection, SCOD penalizes both accepted OOD samples and accepted, misclassified in-distribution (ID) samples.

Let \(s_i\) be an OOD score for sample \(i\), where larger values indicate that a sample is more likely to be OOD. Let \(o_i \in \{0, 1\}\) indicate whether the sample is OOD and let \(e_i \in \{0, 1\}\) indicate whether an ID sample is misclassified. For an OOD acceptance cost \(c_{\mathrm{OOD}} \in [0, 1]\), the per-sample SCOD loss is

\[\begin{split}\ell_i = \begin{cases} 0, & o_i = 0,\ e_i = 0,\\ 1 - c_{\mathrm{OOD}}, & o_i = 0,\ e_i = 1,\\ c_{\mathrm{OOD}}, & o_i = 1. \end{cases}\end{split}\]

Equivalently,

\[\ell_i = c_{\mathrm{OOD}} o_i + (1 - c_{\mathrm{OOD}})(1-o_i)e_i.\]

This convention corresponds to the cost \(c_{\mathrm{fn}}\) used by Narasimhan et al. The parameter \(\beta\) used by Xia and Bouganis instead denotes the ID misclassification cost, and therefore satisfies

\[\beta = 1 - c_{\mathrm{OOD}}.\]

Let \(\sigma\) be the permutation sorting the \(N\) samples by increasing OOD score,

\[s_{\sigma(1)} \leq \cdots \leq s_{\sigma(N)},\]

so that samples most likely to be accepted appear first. At empirical coverage \(\kappa_k = k/N\), the SCOD selective risk is

\[r(\kappa_k) = \frac{1}{k}\sum_{j=1}^{k}\ell_{\sigma(j)}.\]

The SCOD-AURC is the area under this joint risk-coverage curve:

\[\operatorname{SCOD\text{-}AURC} = \int_0^1 r(\kappa)\,\mathrm{d}\kappa.\]

The discrete integration and finite-sample normalization follow AURC.

As input to forward and update, the metric accepts:

• ood_scores (Tensor): Float tensor containing one OOD score per sample. Larger values must indicate more OOD-like samples.

• classification_errors (Tensor): Boolean or binary tensor indicating misclassified ID samples. Values corresponding to OOD samples are ignored.

• is_ood (Tensor): Boolean or binary tensor indicating which samples are OOD.

As output to forward and compute, the metric returns:

• scod_aurc (Tensor): Scalar tensor containing the area under the SCOD risk-coverage curve. Lower values are better.

Parameters:

• ood_cost (float) – Relative cost \(c_{\mathrm{OOD}}\) of accepting an OOD sample. The cost of accepting a misclassified ID sample is 1 - ood_cost. Defaults to 0.5.

• kwargs – Additional keyword arguments passed to torchmetrics.Metric.

Note

The empirical ratio of ID to OOD samples determines the mixture proportion evaluated by this metric. Results obtained with different ID/OOD ratios are therefore not directly comparable unless the mixture proportions are controlled.

References

[1] Xia & Bouganis. Augmenting Softmax Information for Selective Classification with Out-of-Distribution Data. ACCV, 2022..

[2] Narasimhan et al. Plugin Estimators for Selective Classification with Out-of-Distribution Detection..

[3] Geifman & El-Yaniv. Selective Classification for Deep Neural Networks. NeurIPS, 2017..