BrierScore

class torch_uncertainty.metrics.BrierScore(num_classes, top_class=False, reduction='mean', **kwargs)

The Brier Score Metric.

Parameters:

• num_classes (int) – Number of classes

• top_class (bool, optional) – If true, compute the Brier score for the top class only. Defaults to False.

• reduction (str, optional) –

  Determines how to reduce over the \(B\)/batch dimension:

  • 'mean' [default]: Averages score across samples

  • 'sum': Sum score across samples

  • 'none' or None: Returns score per sample

• kwargs – Additional keyword arguments, see Advanced metric settings.

Inputs:

• probs: \((B, C)\) or \((B, N, C)\)

• target: \((B)\) or \((B, C)\)

where \(B\) is the batch size, \(C\) is the number of classes and \(N\) is the number of estimators.

Note

If probs is a 3d tensor, then the metric computes the mean of the Brier score over the estimators ie. \(t = \frac{1}{N} \sum_{i=0}^{N-1} BrierScore(probs[:,i,:], target)\).

Warning

Make sure that the probabilities in probs are normalized to sum to one.

Raises:

ValueError – If reduction is not one of 'mean', 'sum', 'none' or None.

compute()

Compute the final Brier score based on inputs passed to update.

Returns:

The final value(s) for the Brier score

Return type:

Tensor

update(probs, target)

Update the current Brier score with a new tensor of probabilities.

Parameters:

• probs (Tensor) – A probability tensor of shape (batch, num_estimators, num_classes) or (batch, num_classes)

• target (Tensor) – A tensor of ground truth labels of shape (batch, num_classes) or (batch)