IntervalScore#
- class torch_uncertainty.metrics.regression.IntervalScore(coverage, **kwargs)[source]#
Interval (Winkler) score for a central prediction interval.
A proper scoring rule that rewards sharp intervals while penalizing targets that fall outside them. For a central prediction interval \([\hat{l}_i, \hat{u}_i]\) built for a nominal coverage \(1 - \alpha\) (so that \(\hat{l}_i\) and \(\hat{u}_i\) are the \(\alpha/2\) and \(1 - \alpha/2\) predictive quantiles), the score is
\[S_\alpha(\hat{l}_i, \hat{u}_i; y_i) = (\hat{u}_i - \hat{l}_i) + \frac{2}{\alpha} (\hat{l}_i - y_i)\, \mathbf{1}\!\left[y_i < \hat{l}_i\right] + \frac{2}{\alpha} (y_i - \hat{u}_i)\, \mathbf{1}\!\left[y_i > \hat{u}_i\right],\]and the metric returns its mean over all elements. Lower is better. Unlike coverage and width taken separately, the interval score penalizes width and miscoverage jointly, so a trivially wide interval no longer scores well.
- Parameters:
coverage (
float) – The nominal coverage \(1 - \alpha\) of the interval, in \((0, 1)\) (e.g.0.9for a 90% interval). Sets the miscoverage penalty factor \(2 / \alpha\).kwargs (
Any) – Additional keyword arguments, see Advanced metric settings.
Note
Inputs of any shape are accepted and flattened, so the metric returns the mean score over all elements.
- Reference:
[1] Gneiting & Raftery, Strictly Proper Scoring Rules, Prediction, and Estimation, JASA 2007 (Eq. 43).
Example:
import torch from torch_uncertainty.metrics.regression import IntervalScore lower = torch.tensor([0.0, 1.0]) upper = torch.tensor([2.0, 3.0]) target = torch.tensor([1.0, 5.0]) # 2nd target misses the upper bound by 2 metric = IntervalScore(coverage=0.9) metric.update(lower, upper, target) print(metric.compute()) # width mean = 2; penalty on 2nd = (2/0.1)*2 = 40; mean = (2 + 42)/2 = 22 # tensor(22.)