QuantileCalibrationError#

class torch_uncertainty.metrics.regression.QuantileCalibrationError(num_bins=15, norm='l1', ignore_index=None, validate_args=True, **kwargs)[source]#

Quantile Calibration Error for regression tasks.

For each confidence level \(\alpha \in (0, 1)\), a well-calibrated probabilistic regressor should ensure that a fraction \(\alpha\) of the ground-truth targets lies inside the centered \(\alpha\)-credible interval of the predicted distribution \(p_\theta(\cdot \mid x)\). Concretely, let

\[\hat{c}(\alpha) = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}\!\left[ y_i \in \left[ F^{-1}_{\theta, x_i}\!\left(\tfrac{1 - \alpha}{2}\right), F^{-1}_{\theta, x_i}\!\left(\tfrac{1 + \alpha}{2}\right) \right] \right].\]

The metric evaluates this coverage on num_bins confidence levels \(\alpha_k\) regularly spaced between 0.05 and 0.95. It returns

\[\operatorname{QCE}_{L_1} = \frac{1}{K}\sum_{k=1}^{K} \left|\hat{c}(\alpha_k)-\alpha_k\right|,\]

with analogous root-mean-square and maximum variants for norm="l2" and norm="max".

For Independent distributions, calibration is evaluated marginally: every scalar event component contributes one coverage observation.

Parameters:

  • num_bins (int) – Number of confidence levels. Defaults to 15.

  • norm (Literal['l1', 'l2', 'max']) – Norm used to aggregate the calibration gaps. One of "l1", "l2", or "max". Defaults to "l1".

  • ignore_index (int | None) – Optional target value to ignore. Defaults to None.

  • validate_args (bool) – Whether to validate input shapes. Defaults to True.

  • kwargs – Additional keyword arguments, see Advanced metric settings.

compute()[source]#

Compute the Quantile Calibration Error.

Return type:

Tensor

plot()[source]#

Plot empirical coverage against nominal coverage.

Return type:

tuple[Figure, Union[Axes, ndarray]]

update(dist, target, ignore_mask=None)[source]#

Update the metric with predictive distributions and targets.

Parameters:

  • dist (Distribution) – Predicted distribution. It must implement icdf.

  • target (Tensor) – Ground-truth values, with one value per predictive distribution.

  • ignore_mask (Tensor | None) – Boolean mask of targets to ignore. A mask over only the batch dimensions is expanded over trailing event dimensions. Defaults to None.

Return type:

None