References#

Please find an exhaustive list of the references of the models, metrics, and datasets used in this library in the sections below.

Uncertainty Models#

The following uncertainty models are implemented:

Deep Evidential Classification#

For Deep Evidential Classification, consider citing:

Evidential Deep Learning to Quantify Classification Uncertainty

Authors: Murat Sensoy, Lance Kaplan, and Melih Kandemir
Paper: NeurIPS 2018.

Beta NLL in Deep Regression#

For Beta NLL in Deep Regression, consider citing:

On the Pitfalls of Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networks

Authors: Maximilian Seitzer, Arash Tavakoli, Dimitrije Antic, and Georg Martius
Paper: ICLR 2022.

Deep Evidential Regression#

For Deep Evidential Regression, consider citing:

Deep Evidential Regression

Authors: Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus
Paper: NeurIPS 2020.

Variational Inference Bayesian Neural Networks#

For Variational Inference Bayesian Neural Networks, consider citing:

Weight Uncertainty in Neural Networks

Authors: Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra
Paper: ICML 2015.

Deep Ensembles#

For Deep Ensembles, consider citing:

Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles

Authors: Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell
Paper: NeurIPS 2017.

Monte-Carlo Dropout#

For Monte-Carlo Dropout, consider citing:

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Authors: Yarin Gal and Zoubin Ghahramani
Paper: ICML 2016.

Stochastic Weight Averaging#

For Stochastic Weight Averaging, consider citing:

Averaging Weights Leads to Wider Optima and Better Generalization

Authors: Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson
Paper: UAI 2018.

Stochastic Weight Averaging Gaussian#

For Stochastic Weight Averaging Gaussian, consider citing:

A simple baseline for Bayesian uncertainty in deep learning

Authors: Wesley Maddox, Timur Garipov, Pavel Izmailov, Dmitry Vetrov, Andrew Gordon Wilson
Paper: NeurIPS 2019.

CheckpointCollector#

For the SGD ensembling version of CheckpointCollector, consider citing:

Checkpoint Ensembles: Ensemble Methods from a Single Training Process

Authors: Hugh Chen, Scott Lundberg, and Su-In Lee
Paper: ArXiv.

SnapshotEnsemble#

For SnapshotEnsemble, consider citing:

Snapshot Ensembles: Train 1, get M for free

Authors: Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, and Kilian Q. Weinberger
Paper: ICLR 2017.

BatchEnsemble#

For BatchEnsemble, consider citing:

BatchEnsemble: An alternative approach to Efficient Ensemble and Lifelong Learning

Authors: Yeming Wen, Dustin Tran, and Jimmy Ba
Paper: ICLR 2020.

Masksembles#

For Masksembles, consider citing:

Masksembles for Uncertainty Estimation

Authors: Nikita Durasov, Timur Bagautdinov, Pierre Baque, and Pascal Fua
Paper: CVPR 2021.

MIMO#

For MIMO, consider citing:

Training independent subnetworks for robust prediction

Authors: Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshminarayanan, Andrew M. Dai, and Dustin Tran
Paper: ICLR 2021.

Packed-Ensembles#

For Packed-Ensembles, consider citing:

Packed-Ensembles for Efficient Uncertainty Estimation

Authors: Olivier Laurent, Adrien Lafage, Enzo Tartaglione, Geoffrey Daniel, Jean-Marc Martinez, Andrei Bursuc, and Gianni Franchi
Paper: ICLR 2023.

LPBNN#

For LPBNN, consider citing:

Encoding the latent posterior of Bayesian Neural Networks for uncertainty quantification

Authors: Gianni Franchi, Andrei Bursuc, Emanuel Aldea, Severine Dubuisson, Isabelle Bloch
Paper: IEEE TPAMI 2024.

Stochastic Gradient Hamiltonian Monte Carlo#

For Stochastic Gradient Hamiltonian Monte Carlo (SGHMC), consider citing:

Stochastic Gradient Hamiltonian Monte Carlo

Authors: Tianqi Chen, Emily B. Fox, and Carlos Guestrin
Paper: ICML 2014.

And, for the robust version,

Bayesian Optimization with Robust Bayesian Neural Networks

Authors: Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter
Paper: NeurIPS 2016.

Stochastic Gradient Langevin Dynamics#

For Stochastic Gradient Langevin Dynamics (SGLD), consider citing:

Bayesian Learning via Stochastic Gradient Langevin Dynamics

Authors: Max Welling, and Yee Whye Teh
Paper: ICML 2011.

Data Augmentation Methods#

Mixup#

For Mixup, consider citing:

mixup: Beyond Empirical Risk Minimization

Authors: Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz
Paper: ICLR 2018.

RegMixup#

For RegMixup, consider citing:

RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy and Out Distribution Robustness

Authors: Francesco Pinto, Harry Yang, Ser-Nam Lim, Philip H.S. Torr, and Puneet K. Dokania
Paper: NeurIPS 2022.

MixupIO#

For MixupIO, consider citing:

On the Pitfall of Mixup for Uncertainty Calibration

Authors: Deng-Bao Wang, Lanqing Li, Peilin Zhao, Pheng-Ann Heng, and Min-Ling Zhang
Paper: CVPR 2023 <https://openaccess.thecvf.com/content/CVPR2023/papers/Wang_On_the_Pitfall_of_Mixup_for_Uncertainty_Calibration_CVPR_2023_paper.pdf>__

Warping Mixup#

For Warping Mixup, consider citing:

Tailoring Mixup to Data using Kernel Warping functions

Authors: Quentin Bouniot, Pavlo Mozharovskyi, and Florence d’Alché-Buc
Paper: ArXiv 2023.

Test-Time-Adaptation with ZERO#

For ZERO, consider citing:

Frustratingly Easy Test-Time Adaptation of Vision-Language Models

Authors: Matteo Farina, Gianni Franchi, Giovanni Iacca, Massimiliano Mancini and Elisa Ricci
Paper: NeurIPS 2024.

Post-Processing Methods#

Temperature, Vector, & Matrix scaling#

For temperature, vector, & matrix scaling, consider citing:

On Calibration of Modern Neural Networks

Authors: Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger
Paper: ICML 2017.

Monte-Carlo Batch Normalization#

For Monte-Carlo Batch Normalization, consider citing:

Bayesian Uncertainty Estimation for Batch Normalized Deep Networks

Authors: Mathias Teye, Hossein Azizpour, and Kevin Smith
Paper: ICML 2018.

Laplace Approximation#

For Laplace Approximation, consider citing:

Laplace Redux - Effortless Bayesian Deep Learning

Authors: Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig
Paper: NeurIPS 2021.

Losses#

Focal Loss#

For the focal loss, consider citing:

Focal Loss for Dense Object Detection

Authors: Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár
Paper: TPAMI 2020.

Conflictual Loss#

For the conflictual loss, consider citing:

On the Calibration of Epistemic Uncertainty: Principles, Paradoxes and Conflictual Loss

Authors: Mohammed Fellaji, Frédéric Pennerath, Brieuc Conan-Guez, and Miguel Couceiro
Paper: ArXiv 2024.

Binary Cross-Entropy with Logits Loss with Label Smoothing#

For the binary cross-entropy with logits loss with label smoothing, consider citing:

Rethinking the Inception Architecture for Computer Vision

Authors: Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna
Paper: CVPR 2016.

MaxSup: Fixing Label Smoothing for Improved Feature Representation#

For the cross-entropy with maximum suppression loss, consider citing:

MaxSup: Fixing Label Smoothing for Improved Feature Representation

Authors: Yuxuan Zhou, Heng Li, Zhi-Qi Cheng, Xudong Yan, Mario Fritz, and Margret Keuper
Paper: ArXiv 2024.

Metrics#

The following metrics are used/implemented.

Expected Calibration Error#

For the expected calibration error, consider citing:

Obtaining Well Calibrated Probabilities Using Bayesian Binning

Authors: Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht
Paper: AAAI 2015.

Adaptive Calibration Error#

For the adaptive calibration error, consider citing:

Measuring Calibration in Deep Learning

Authors: Jeremy Nixon, Mike Dusenberry, Ghassen Jerfel, Timothy Nguyen, Jeremiah Liu, Linchuan Zhang, and Dustin Tran
Paper: CVPRW 2019.

Area Under the Risk-Coverage curve#

For the area under the risk-coverage curve, consider citing:

Selective classification for deep neural networks

Authors: Yonatan Geifman and Ran El-Yaniv
Paper: NeurIPS 2017.

Area Under the Generalized Risk-Coverage curve#

For the area under the generalized risk-coverage curve, consider citing:

Overcoming Common Flaws in the Evaluation of Selective Classification Systems

Authors: Jeremias Traub, Till J. Bungert, Carsten T. Lüth, Michael Baumgartner, Klaus H. Maier-Hein, Lena Maier-Hein, and Paul F Jaeger
Paper: ArXiv.

Grouping Loss#

For the grouping loss, consider citing:

Beyond Calibration: Estimating the Grouping Loss of Modern Neural Networks

Authors: Alexandre Perez-Lebel, Marine Le Morvan, and Gaël Varoquaux
Paper: ICLR 2023.

Datasets#

The following datasets are used/implemented.

MNIST#

Gradient-based learning applied to document recognition

Authors: Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner
Paper: Proceedings of the IEEE 1998.

MNIST-C#

MNIST-C: A Robustness Benchmark for Computer Vision

Authors: Norman Mu, and Justin Gilmer
Paper: ICMLW 2019.

Not-MNIST#

Author: Yaroslav Bulatov

CIFAR-10 & CIFAR-100#

Learning multiple layers of features from tiny images

Authors: Alex Krizhevsky
Paper: MIT Tech Report.

CIFAR-C, Tiny-ImageNet-C, ImageNet-C#

Benchmarking neural network robustness to common corruptions and perturbations

Authors: Dan Hendrycks and Thomas Dietterich
Paper: ICLR 2019.

CIFAR-10 H#

Human uncertainty makes classification more robust

Authors: Joshua C. Peterson, Ruairidh M. Battleday, Thomas L. Griffiths, and Olga Russakovsky
Paper: ICCV 2019.

CIFAR-10 N / CIFAR-100 N#

Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations

Authors: Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu
Paper: ICLR 2022.

SVHN#

Reading digits in natural images with unsupervised feature learning

Authors: Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng
Paper: NeurIPS Workshops 2011.

ImageNet#

Imagenet: A large-scale hierarchical image database

Authors: Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei
Paper: CVPR 2009.

ImageNet-A & ImageNet-0#

Natural adversarial examples

Authors: Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song
Paper: CVPR 2021.

ImageNet-R#

The many faces of robustness: A critical analysis of out-of-distribution generalization

Authors: Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al.
Paper: ICCV 2021.

Textures#

ViM: Out-of-distribution with virtual-logit matching

Authors: Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang
Paper: CVPR 2022.

OpenImage-O#

Curation: