ConfidencePenaltyLoss

class torch_uncertainty.losses.ConfidencePenaltyLoss(reg_weight=1, reduction='mean', eps=1e-06)

The Confidence Penalty loss.

Augments the standard cross-entropy loss with a regulariser that penalises low-entropy (over-confident) predictive distributions:

\[\mathcal{L} = \text{CE}(\mathbf{z}, y) + \lambda \left( \log C + \sum_{i=1}^{C} p_i \log(p_i + \varepsilon) \right),\]

where \(\mathbf{p} = \mathrm{softmax}(\mathbf{z})\), \(C\) is the number of classes, \(\lambda\) is reg_weight, and \(\varepsilon\) is eps. The regulariser equals the negative entropy of \(\mathbf{p}\) shifted by \(\log C\) so that it is non-negative.

Parameters:

• reg_weight (float) – The weight \(\lambda\) of the regularization term.

• reduction (str | None) – Specifies the reduction to apply to the output: 'none' | 'mean' | 'sum'. Defaults to 'mean'.

• eps (float) – A small value to avoid numerical instability. Defaults to 1e-6.

References

[1] Pereyra, G., et al. (2017). Regularizing neural networks by penalizing confident output distributions.

forward(logits, targets)

Compute the Confidence Penalty loss.

Parameters:

• logits (Tensor) – The inputs of the Bayesian Neural Network

• targets (Tensor) – The target values

Returns:

The Confidence Penalty loss

Return type:

Tensor