QuantileCalibrationError

class torch_uncertainty.metrics.regression.QuantileCalibrationError(num_bins=15, norm='l1', ignore_index=None, validate_args=True, **kwargs)

Quantile Calibration Error for regression tasks.

For each confidence level \(\alpha \in (0, 1)\), a well-calibrated probabilistic regressor should ensure that a fraction \(\alpha\) of the ground-truth targets lies inside the centered \(\alpha\)-credible interval of the predicted distribution \(p_\theta(\cdot \mid x)\). Concretely, let

\[\hat{c}(\alpha) = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}\!\left[ y_i \in \left[ F^{-1}_{\theta, x_i}\!\left(\tfrac{1 - \alpha}{2}\right), F^{-1}_{\theta, x_i}\!\left(\tfrac{1 + \alpha}{2}\right) \right] \right],\]

where \(F^{-1}_{\theta, x_i}\) is the predictive inverse CDF (computed via the distribution’s icdf method). The Quantile Calibration Error then aggregates the gap \(|\hat{c}(\alpha) - \alpha|\) over a regular grid of num_bins confidence levels using the chosen norm — the regression counterpart of the CalibrationError.

Parameters:

• num_bins (int) – Number of bins to use for calibration. Defaults to 15.

• norm (Literal['l1', 'l2', 'max']) – Norm to use for calibration error computation. Defaults to "l1".

• ignore_index – Index to ignore during calibration. Defaults to None.

• validate_args – Whether to validate the input arguments. Defaults to True.

• kwargs – Additional keyword arguments, see Advanced metric settings.

compute()

Compute the quantile calibration error.

Returns:

The quantile calibration error.

Return type:

Tensor

Warning

If the distribution does not support icdf(), this returns nan values.

plot()

Plot the quantile calibration reliability diagram.

Raises:

NotImplementedError – If the distribution does not support icdf().

Return type:

tuple[Figure, Union[Axes, ndarray]]

update(dist, target, padding_mask=None)

Update the metric with new predictions and targets.

Parameters:

• dist (Distribution) – The predicted distribution.

• target (Tensor) – The ground truth values.

• padding_mask (Tensor | None) – A mask to ignore certain values. Defaults to None.

Return type:

None