AdaptiveCalibrationError#

class torch_uncertainty.metrics.classification.AdaptiveCalibrationError(task, num_bins=10, norm='l1', num_classes=None, ignore_index=None, validate_args=True, **kwargs)[source]#

Computes the Adaptive Top-label Calibration Error (ACE) for classification tasks.

The Adaptive Calibration Error is a metric designed to measure the calibration of predicted probabilities by dividing the probability space into bins that adapt to the distribution of predicted probabilities. Unlike uniform binning, adaptive binning ensures a more balanced representation of predictions across bins.

This metric is particularly useful for datasets or models where predictions are concentrated in certain regions of the probability space.

Parameters:

task (str) – Specifies the task type, either "binary" or "multiclass".
num_bins (int, optional) – Number of bins to divide the probability space. Defaults to 10.
norm (str, optional) – Specifies the type of norm to use: "l1", "l2", or "max". Defaults to "l1".
num_classes (int, optional) – Number of classes for "multiclass" tasks. Required when task is "multiclass".
ignore_index (int, optional) – Index to ignore during calculations. Defaults to None.
validate_args (bool, optional) – Whether to validate input arguments. Defaults to True.
**kwargs (Any) – Additional keyword arguments passed to the metric.

Example

from torch_uncertainty.metrics.classification.adaptive_calibration_error import (
    AdaptiveCalibrationError,
)

# Binary classification example
predicted_probs = torch.tensor([0.95, 0.85, 0.15, 0.05])
true_labels = torch.tensor([1, 1, 0, 0])

metric = CalibrationError(
    task="binary",
    num_bins=5,
    norm="l1",
)

calibration_error = metric(predicted_probs, true_labels)
print(f"Calibration Error (Binary): {calibration_error}")
# Output : Calibration Error (Binary): 0.1

Note

Adaptive binning adjusts the size of bins to ensure a more uniform distribution of samples across bins.
If task=”multiclass”, num_classes must be provided; otherwise, a TypeError will be raised.

Warning

Ensure that num_classes matches the actual number of classes in the dataset for multiclass tasks.

References

[1] Nixon et al., Measuring calibration in deep learning, CVPR Workshops, 2019.

AdaptiveCalibrationError#

This Page