Disagreement#
- class torch_uncertainty.metrics.classification.Disagreement(reduction='mean', **kwargs)[source]#
Calculate the Disagreement Metric.
The Disagreement Metric estimates the confidence of an ensemble of estimators. Given the predicted classes \(\hat{y}_{b,n} = \arg\max_c \hat{p}_{b,n,c}\) for sample \(b\) and estimator \(n\), the disagreement is the fraction of estimator pairs that predict different classes:
\[\text{Disagreement} = \frac{2}{N(N-1)} \sum_{1 \le n < m \le N} \mathbf{1}\!\left[ \hat{y}_{b,n} \neq \hat{y}_{b,m} \right]\]where \(N\) is the number of estimators. Equivalently, this equals
\[\text{Disagreement} = 1 - \frac{1}{\binom{N}{2}} \sum_{c=1}^{C} \binom{n_c}{2}\]where \(n_c = \sum_{n=1}^{N} \mathbf{1}[\hat{y}_{b,n} = c]\) is the number of estimators predicting class \(c\).
- Parameters:
reduction (
Optional[Literal['mean','sum','none']]) –Determines how to reduce over the \(B\)/batch dimension:
'mean'[default]: Averages score across samples'sum': Sum score across samples'none'orNone: Returns score per sample
kwargs (
Any) – Additional keyword arguments, see Advanced metric settings.
- Inputs:
probs: \((B, N, C)\)
where \(B\) is the batch size, \(C\) is the number of classes and \(N\) is the number of estimators.
Note
A higher disagreement means a lower confidence.
Warning
Make sure that the probabilities in
probsare normalized to sum to one.- Raises:
ValueError – If
reductionis not one of'mean','sum','none'orNone.
Example:
from torch_uncertainty.metrics.classification import Disagreement probs = torch.tensor( [ [[0.7, 0.3], [0.6, 0.4], [0.8, 0.2]], # Example 1, 3 estimators [[0.4, 0.6], [0.5, 0.5], [0.3, 0.7]], # Example 2, 3 estimators ] ) ds = Disagreement(reduction="mean") ds.update(probs) result = ds.compute() print(result) # output: tensor(0.3333)