Note
Go to the end to download the full example code.
Conformal Prediction on CIFAR-10 with TorchUncertainty#
We evaluate the model’s performance both before and after applying different conformal predictors (THR, APS, RAPS), and visualize how conformal prediction estimates the prediction sets.
We use the pretrained ResNet models we provide on Hugging Face.
import matplotlib.pyplot as plt
import numpy as np
import torch
from huggingface_hub import hf_hub_download
from torch_uncertainty import TUTrainer
from torch_uncertainty.datamodules import CIFAR10DataModule
from torch_uncertainty.models.classification.resnet import resnet
from torch_uncertainty.post_processing import ConformalClsAPS, ConformalClsRAPS, ConformalClsTHR
from torch_uncertainty.routines import ClassificationRoutine
1. Load pretrained model from Hugging Face repository#
We use a ResNet18 model trained on CIFAR-10, provided by the TorchUncertainty team
ckpt_path = hf_hub_download(repo_id="torch-uncertainty/resnet18_c10", filename="resnet18_c10.ckpt")
model = resnet(in_channels=3, num_classes=10, arch=18, conv_bias=False, style="cifar")
ckpt = torch.load(ckpt_path, weights_only=True)
model.load_state_dict(ckpt)
model = model.cuda().eval()
2. Load CIFAR-10 Dataset & Define Dataloaders#
We set eval_ood to True to evaluate the performance of Conformal scores for detecting out-of-distribution samples. In this case, since we use a model trained on the full training set, we use the test set to as calibration set for the Conformal methods and for its evaluation. This is not a proper way to evaluate the coverage.
BATCH_SIZE = 128
datamodule = CIFAR10DataModule(
root="./data",
batch_size=BATCH_SIZE,
num_workers=8,
eval_ood=True,
postprocess_set="test",
)
datamodule.prepare_data()
datamodule.setup()
0%| | 0.00/170M [00:00<?, ?B/s]
0%| | 65.5k/170M [00:00<07:09, 397kB/s]
0%| | 229k/170M [00:00<03:49, 743kB/s]
0%| | 590k/170M [00:00<01:39, 1.70MB/s]
1%| | 918k/170M [00:00<01:16, 2.20MB/s]
1%| | 1.44M/170M [00:00<01:00, 2.80MB/s]
2%|▏ | 2.79M/170M [00:00<00:28, 5.82MB/s]
2%|▏ | 4.00M/170M [00:00<00:21, 7.63MB/s]
3%|▎ | 5.73M/170M [00:00<00:15, 10.5MB/s]
5%|▍ | 8.00M/170M [00:01<00:11, 14.0MB/s]
6%|▌ | 10.3M/170M [00:01<00:09, 16.6MB/s]
8%|▊ | 13.1M/170M [00:01<00:07, 20.0MB/s]
9%|▉ | 15.4M/170M [00:01<00:07, 20.8MB/s]
11%|█ | 18.2M/170M [00:01<00:06, 22.9MB/s]
12%|█▏ | 20.5M/170M [00:01<00:06, 22.9MB/s]
14%|█▎ | 23.2M/170M [00:01<00:06, 24.0MB/s]
15%|█▌ | 25.6M/170M [00:01<00:06, 23.9MB/s]
16%|█▋ | 28.0M/170M [00:01<00:05, 23.8MB/s]
18%|█▊ | 30.4M/170M [00:01<00:05, 23.7MB/s]
19%|█▉ | 32.8M/170M [00:02<00:05, 23.1MB/s]
21%|██ | 35.2M/170M [00:02<00:06, 22.5MB/s]
22%|██▏ | 37.4M/170M [00:02<00:06, 20.8MB/s]
23%|██▎ | 39.6M/170M [00:02<00:06, 19.6MB/s]
24%|██▍ | 41.5M/170M [00:02<00:06, 19.5MB/s]
26%|██▌ | 43.6M/170M [00:02<00:06, 19.7MB/s]
27%|██▋ | 46.0M/170M [00:02<00:05, 20.9MB/s]
29%|██▊ | 48.7M/170M [00:02<00:05, 22.4MB/s]
30%|███ | 51.2M/170M [00:02<00:05, 23.2MB/s]
32%|███▏ | 53.9M/170M [00:03<00:04, 24.0MB/s]
33%|███▎ | 56.4M/170M [00:03<00:04, 24.4MB/s]
35%|███▌ | 59.7M/170M [00:03<00:04, 26.9MB/s]
37%|███▋ | 62.4M/170M [00:03<00:04, 26.4MB/s]
39%|███▊ | 65.7M/170M [00:03<00:03, 28.2MB/s]
40%|████ | 68.6M/170M [00:03<00:03, 28.3MB/s]
42%|████▏ | 71.8M/170M [00:03<00:03, 28.7MB/s]
44%|████▍ | 74.6M/170M [00:03<00:03, 27.9MB/s]
45%|████▌ | 77.5M/170M [00:03<00:03, 27.4MB/s]
47%|████▋ | 80.2M/170M [00:04<00:03, 25.4MB/s]
49%|████▊ | 82.8M/170M [00:04<00:03, 23.9MB/s]
50%|████▉ | 85.2M/170M [00:04<00:03, 22.6MB/s]
51%|█████▏ | 87.5M/170M [00:04<00:03, 22.2MB/s]
53%|█████▎ | 89.8M/170M [00:04<00:03, 20.6MB/s]
54%|█████▍ | 91.9M/170M [00:04<00:03, 20.3MB/s]
55%|█████▌ | 93.9M/170M [00:04<00:03, 19.5MB/s]
56%|█████▋ | 95.9M/170M [00:04<00:03, 19.2MB/s]
57%|█████▋ | 97.8M/170M [00:04<00:03, 18.9MB/s]
59%|█████▊ | 99.9M/170M [00:05<00:03, 19.3MB/s]
60%|█████▉ | 102M/170M [00:05<00:03, 19.4MB/s]
61%|██████ | 104M/170M [00:05<00:03, 19.8MB/s]
62%|██████▏ | 106M/170M [00:05<00:03, 19.4MB/s]
63%|██████▎ | 108M/170M [00:05<00:03, 18.8MB/s]
64%|██████▍ | 110M/170M [00:05<00:03, 17.3MB/s]
65%|██████▌ | 112M/170M [00:05<00:03, 16.5MB/s]
66%|██████▋ | 113M/170M [00:05<00:03, 15.6MB/s]
67%|██████▋ | 115M/170M [00:05<00:03, 15.5MB/s]
68%|██████▊ | 117M/170M [00:06<00:03, 16.0MB/s]
69%|██████▉ | 118M/170M [00:06<00:03, 16.1MB/s]
70%|███████ | 120M/170M [00:06<00:03, 16.1MB/s]
71%|███████▏ | 122M/170M [00:06<00:03, 16.1MB/s]
72%|███████▏ | 123M/170M [00:06<00:02, 15.9MB/s]
73%|███████▎ | 125M/170M [00:06<00:02, 15.6MB/s]
74%|███████▍ | 126M/170M [00:06<00:02, 15.4MB/s]
75%|███████▌ | 128M/170M [00:06<00:02, 15.2MB/s]
76%|███████▌ | 130M/170M [00:06<00:02, 14.9MB/s]
77%|███████▋ | 131M/170M [00:07<00:02, 14.7MB/s]
78%|███████▊ | 133M/170M [00:07<00:02, 14.4MB/s]
79%|███████▊ | 134M/170M [00:07<00:02, 14.3MB/s]
79%|███████▉ | 135M/170M [00:07<00:02, 14.2MB/s]
80%|████████ | 137M/170M [00:07<00:02, 14.3MB/s]
81%|████████ | 138M/170M [00:07<00:02, 14.3MB/s]
82%|████████▏ | 140M/170M [00:07<00:02, 14.2MB/s]
83%|████████▎ | 142M/170M [00:07<00:01, 15.1MB/s]
84%|████████▍ | 143M/170M [00:07<00:01, 15.9MB/s]
85%|████████▌ | 145M/170M [00:07<00:01, 16.3MB/s]
86%|████████▌ | 147M/170M [00:08<00:01, 16.6MB/s]
87%|████████▋ | 149M/170M [00:08<00:01, 16.6MB/s]
88%|████████▊ | 150M/170M [00:08<00:01, 16.7MB/s]
89%|████████▉ | 152M/170M [00:08<00:01, 16.9MB/s]
90%|█████████ | 154M/170M [00:08<00:01, 16.8MB/s]
91%|█████████ | 155M/170M [00:08<00:00, 16.5MB/s]
92%|█████████▏| 157M/170M [00:08<00:00, 16.4MB/s]
93%|█████████▎| 159M/170M [00:08<00:00, 16.2MB/s]
94%|█████████▍| 160M/170M [00:08<00:00, 16.0MB/s]
95%|█████████▍| 162M/170M [00:08<00:00, 15.7MB/s]
96%|█████████▌| 164M/170M [00:09<00:00, 15.5MB/s]
97%|█████████▋| 165M/170M [00:09<00:00, 15.4MB/s]
98%|█████████▊| 167M/170M [00:09<00:00, 16.2MB/s]
99%|█████████▉| 169M/170M [00:09<00:00, 16.4MB/s]
100%|█████████▉| 170M/170M [00:09<00:00, 16.4MB/s]
100%|██████████| 170M/170M [00:09<00:00, 18.0MB/s]
0%| | 0.00/64.3M [00:00<?, ?B/s]
0%| | 32.8k/64.3M [00:00<04:27, 240kB/s]
0%| | 65.5k/64.3M [00:00<04:28, 240kB/s]
0%| | 98.3k/64.3M [00:00<04:27, 240kB/s]
0%| | 131k/64.3M [00:00<04:27, 239kB/s]
0%| | 197k/64.3M [00:00<03:16, 326kB/s]
0%| | 295k/64.3M [00:00<02:19, 459kB/s]
1%| | 426k/64.3M [00:00<01:42, 621kB/s]
1%| | 557k/64.3M [00:01<01:27, 728kB/s]
1%| | 721k/64.3M [00:01<01:12, 874kB/s]
1%|▏ | 852k/64.3M [00:01<01:10, 897kB/s]
2%|▏ | 1.18M/64.3M [00:01<00:46, 1.35MB/s]
2%|▏ | 1.41M/64.3M [00:01<00:43, 1.45MB/s]
3%|▎ | 1.64M/64.3M [00:01<00:41, 1.52MB/s]
3%|▎ | 1.87M/64.3M [00:01<00:39, 1.56MB/s]
3%|▎ | 2.10M/64.3M [00:02<00:38, 1.60MB/s]
4%|▎ | 2.36M/64.3M [00:02<00:36, 1.69MB/s]
4%|▍ | 2.59M/64.3M [00:02<00:36, 1.69MB/s]
4%|▍ | 2.85M/64.3M [00:02<00:34, 1.76MB/s]
5%|▍ | 3.11M/64.3M [00:02<00:33, 1.80MB/s]
5%|▌ | 3.38M/64.3M [00:02<00:33, 1.84MB/s]
6%|▌ | 3.64M/64.3M [00:02<00:32, 1.86MB/s]
6%|▌ | 3.93M/64.3M [00:03<00:30, 1.95MB/s]
7%|▋ | 4.19M/64.3M [00:03<00:31, 1.94MB/s]
7%|▋ | 4.49M/64.3M [00:03<00:29, 2.00MB/s]
7%|▋ | 4.78M/64.3M [00:03<00:29, 2.05MB/s]
8%|▊ | 5.05M/64.3M [00:03<00:29, 2.01MB/s]
8%|▊ | 5.34M/64.3M [00:03<00:28, 2.05MB/s]
9%|▉ | 5.64M/64.3M [00:03<00:28, 2.08MB/s]
9%|▉ | 5.93M/64.3M [00:03<00:27, 2.10MB/s]
10%|▉ | 6.23M/64.3M [00:04<00:27, 2.12MB/s]
10%|█ | 6.52M/64.3M [00:04<00:24, 2.31MB/s]
11%|█ | 6.82M/64.3M [00:04<00:23, 2.44MB/s]
11%|█ | 7.11M/64.3M [00:04<00:24, 2.37MB/s]
12%|█▏ | 7.41M/64.3M [00:04<00:22, 2.49MB/s]
12%|█▏ | 7.67M/64.3M [00:04<00:22, 2.47MB/s]
12%|█▏ | 7.93M/64.3M [00:04<00:24, 2.29MB/s]
13%|█▎ | 8.19M/64.3M [00:04<00:25, 2.17MB/s]
13%|█▎ | 8.42M/64.3M [00:05<00:27, 2.02MB/s]
14%|█▎ | 8.68M/64.3M [00:05<00:28, 1.98MB/s]
14%|█▍ | 8.98M/64.3M [00:05<00:27, 2.04MB/s]
14%|█▍ | 9.31M/64.3M [00:05<00:25, 2.14MB/s]
15%|█▍ | 9.60M/64.3M [00:05<00:25, 2.15MB/s]
15%|█▌ | 9.93M/64.3M [00:05<00:24, 2.22MB/s]
16%|█▌ | 10.2M/64.3M [00:05<00:24, 2.20MB/s]
16%|█▋ | 10.5M/64.3M [00:05<00:22, 2.38MB/s]
17%|█▋ | 10.8M/64.3M [00:06<00:21, 2.51MB/s]
17%|█▋ | 11.1M/64.3M [00:06<00:20, 2.60MB/s]
18%|█▊ | 11.4M/64.3M [00:06<00:20, 2.64MB/s]
18%|█▊ | 11.7M/64.3M [00:06<00:21, 2.47MB/s]
19%|█▊ | 12.0M/64.3M [00:06<00:22, 2.29MB/s]
19%|█▉ | 12.2M/64.3M [00:06<00:23, 2.17MB/s]
19%|█▉ | 12.5M/64.3M [00:06<00:25, 2.02MB/s]
20%|█▉ | 12.7M/64.3M [00:06<00:25, 1.99MB/s]
20%|██ | 13.0M/64.3M [00:07<00:25, 2.04MB/s]
21%|██ | 13.3M/64.3M [00:07<00:23, 2.15MB/s]
21%|██ | 13.6M/64.3M [00:07<00:23, 2.15MB/s]
22%|██▏ | 13.9M/64.3M [00:07<00:21, 2.34MB/s]
22%|██▏ | 14.2M/64.3M [00:07<00:20, 2.48MB/s]
23%|██▎ | 14.5M/64.3M [00:07<00:19, 2.58MB/s]
23%|██▎ | 14.8M/64.3M [00:07<00:18, 2.62MB/s]
24%|██▎ | 15.1M/64.3M [00:07<00:19, 2.46MB/s]
24%|██▍ | 15.4M/64.3M [00:08<00:21, 2.28MB/s]
24%|██▍ | 15.6M/64.3M [00:08<00:22, 2.17MB/s]
25%|██▍ | 15.9M/64.3M [00:08<00:23, 2.02MB/s]
25%|██▌ | 16.1M/64.3M [00:08<00:24, 1.99MB/s]
26%|██▌ | 16.4M/64.3M [00:08<00:21, 2.22MB/s]
26%|██▌ | 16.7M/64.3M [00:08<00:19, 2.40MB/s]
26%|██▋ | 17.0M/64.3M [00:08<00:18, 2.51MB/s]
27%|██▋ | 17.3M/64.3M [00:08<00:18, 2.50MB/s]
27%|██▋ | 17.5M/64.3M [00:09<00:20, 2.30MB/s]
28%|██▊ | 17.8M/64.3M [00:09<00:21, 2.17MB/s]
28%|██▊ | 18.0M/64.3M [00:09<00:22, 2.02MB/s]
28%|██▊ | 18.3M/64.3M [00:09<00:20, 2.25MB/s]
29%|██▉ | 18.6M/64.3M [00:09<00:21, 2.12MB/s]
30%|██▉ | 19.0M/64.3M [00:09<00:20, 2.20MB/s]
30%|███ | 19.3M/64.3M [00:09<00:19, 2.26MB/s]
31%|███ | 19.6M/64.3M [00:10<00:19, 2.30MB/s]
31%|███ | 20.0M/64.3M [00:10<00:19, 2.32MB/s]
32%|███▏ | 20.3M/64.3M [00:10<00:18, 2.34MB/s]
32%|███▏ | 20.6M/64.3M [00:10<00:17, 2.55MB/s]
33%|███▎ | 20.9M/64.3M [00:10<00:15, 2.72MB/s]
33%|███▎ | 21.3M/64.3M [00:10<00:16, 2.62MB/s]
34%|███▎ | 21.6M/64.3M [00:10<00:15, 2.76MB/s]
34%|███▍ | 21.9M/64.3M [00:10<00:15, 2.75MB/s]
35%|███▍ | 22.2M/64.3M [00:10<00:16, 2.56MB/s]
35%|███▍ | 22.4M/64.3M [00:11<00:17, 2.35MB/s]
35%|███▌ | 22.7M/64.3M [00:11<00:18, 2.22MB/s]
36%|███▌ | 23.1M/64.3M [00:11<00:17, 2.35MB/s]
36%|███▋ | 23.4M/64.3M [00:11<00:16, 2.44MB/s]
37%|███▋ | 23.8M/64.3M [00:11<00:14, 2.71MB/s]
38%|███▊ | 24.2M/64.3M [00:11<00:13, 2.92MB/s]
38%|███▊ | 24.5M/64.3M [00:11<00:13, 2.99MB/s]
39%|███▊ | 24.8M/64.3M [00:11<00:13, 3.01MB/s]
39%|███▉ | 25.1M/64.3M [00:12<00:13, 2.80MB/s]
40%|███▉ | 25.4M/64.3M [00:12<00:14, 2.59MB/s]
40%|████ | 25.7M/64.3M [00:12<00:14, 2.68MB/s]
41%|████ | 26.1M/64.3M [00:12<00:13, 2.93MB/s]
41%|████ | 26.5M/64.3M [00:12<00:11, 3.16MB/s]
42%|████▏ | 26.8M/64.3M [00:12<00:11, 3.13MB/s]
42%|████▏ | 27.1M/64.3M [00:12<00:12, 2.88MB/s]
43%|████▎ | 27.4M/64.3M [00:12<00:13, 2.65MB/s]
43%|████▎ | 27.8M/64.3M [00:12<00:12, 2.97MB/s]
44%|████▍ | 28.2M/64.3M [00:13<00:11, 3.22MB/s]
45%|████▍ | 28.6M/64.3M [00:13<00:10, 3.47MB/s]
45%|████▌ | 29.0M/64.3M [00:13<00:10, 3.44MB/s]
46%|████▌ | 29.4M/64.3M [00:13<00:11, 3.16MB/s]
46%|████▌ | 29.7M/64.3M [00:13<00:11, 2.95MB/s]
47%|████▋ | 30.1M/64.3M [00:13<00:10, 3.33MB/s]
48%|████▊ | 30.6M/64.3M [00:13<00:09, 3.65MB/s]
48%|████▊ | 31.1M/64.3M [00:13<00:08, 3.86MB/s]
49%|████▉ | 31.5M/64.3M [00:13<00:08, 3.90MB/s]
50%|████▉ | 31.9M/64.3M [00:14<00:08, 3.63MB/s]
50%|█████ | 32.3M/64.3M [00:14<00:09, 3.40MB/s]
51%|█████ | 32.8M/64.3M [00:14<00:09, 3.41MB/s]
52%|█████▏ | 33.3M/64.3M [00:14<00:08, 3.62MB/s]
53%|█████▎ | 33.9M/64.3M [00:14<00:07, 4.05MB/s]
54%|█████▎ | 34.4M/64.3M [00:14<00:06, 4.35MB/s]
54%|█████▍ | 35.0M/64.3M [00:14<00:06, 4.62MB/s]
55%|█████▌ | 35.5M/64.3M [00:14<00:06, 4.60MB/s]
56%|█████▌ | 35.9M/64.3M [00:15<00:06, 4.27MB/s]
57%|█████▋ | 36.4M/64.3M [00:15<00:06, 3.99MB/s]
58%|█████▊ | 37.0M/64.3M [00:15<00:06, 4.38MB/s]
58%|█████▊ | 37.6M/64.3M [00:15<00:05, 4.77MB/s]
59%|█████▉ | 38.2M/64.3M [00:15<00:05, 5.10MB/s]
60%|██████ | 38.7M/64.3M [00:15<00:05, 5.04MB/s]
61%|██████ | 39.2M/64.3M [00:15<00:05, 4.64MB/s]
62%|██████▏ | 39.7M/64.3M [00:15<00:05, 4.43MB/s]
63%|██████▎ | 40.5M/64.3M [00:16<00:04, 5.08MB/s]
64%|██████▍ | 41.1M/64.3M [00:16<00:04, 5.46MB/s]
65%|██████▌ | 41.8M/64.3M [00:16<00:03, 5.86MB/s]
66%|██████▌ | 42.5M/64.3M [00:16<00:03, 5.85MB/s]
67%|██████▋ | 43.1M/64.3M [00:16<00:03, 5.42MB/s]
68%|██████▊ | 43.6M/64.3M [00:16<00:04, 5.03MB/s]
69%|██████▉ | 44.5M/64.3M [00:16<00:03, 5.78MB/s]
70%|███████ | 45.2M/64.3M [00:16<00:03, 6.16MB/s]
72%|███████▏ | 46.0M/64.3M [00:16<00:02, 6.70MB/s]
73%|███████▎ | 46.7M/64.3M [00:17<00:02, 6.63MB/s]
74%|███████▍ | 47.4M/64.3M [00:17<00:02, 6.09MB/s]
75%|███████▍ | 48.1M/64.3M [00:17<00:02, 5.81MB/s]
76%|███████▋ | 49.0M/64.3M [00:17<00:02, 6.60MB/s]
78%|███████▊ | 49.9M/64.3M [00:17<00:02, 7.09MB/s]
79%|███████▉ | 50.8M/64.3M [00:17<00:01, 7.64MB/s]
80%|████████ | 51.6M/64.3M [00:17<00:01, 7.56MB/s]
82%|████████▏ | 52.4M/64.3M [00:17<00:01, 6.96MB/s]
83%|████████▎ | 53.2M/64.3M [00:17<00:01, 6.72MB/s]
84%|████████▍ | 54.2M/64.3M [00:18<00:01, 7.52MB/s]
86%|████████▌ | 55.2M/64.3M [00:18<00:01, 8.14MB/s]
88%|████████▊ | 56.3M/64.3M [00:18<00:00, 8.72MB/s]
89%|████████▉ | 57.2M/64.3M [00:18<00:00, 8.67MB/s]
90%|█████████ | 58.1M/64.3M [00:18<00:00, 7.94MB/s]
92%|█████████▏| 59.0M/64.3M [00:18<00:00, 7.71MB/s]
94%|█████████▎| 60.2M/64.3M [00:18<00:00, 8.66MB/s]
95%|█████████▌| 61.3M/64.3M [00:18<00:00, 9.31MB/s]
97%|█████████▋| 62.6M/64.3M [00:18<00:00, 10.0MB/s]
99%|█████████▉| 63.6M/64.3M [00:19<00:00, 9.92MB/s]
100%|██████████| 64.3M/64.3M [00:19<00:00, 3.36MB/s]
3. Define the Lightning Trainer#
trainer = TUTrainer(accelerator="gpu", devices=1, max_epochs=5, enable_progress_bar=False)
4. Function to Visualize the Prediction Sets#
def visualize_prediction_sets(inputs, labels, confidence_scores, classes, num_examples=5) -> None:
_, axs = plt.subplots(2, num_examples, figsize=(15, 5))
for i in range(num_examples):
ax = axs[0, i]
img = np.clip(
inputs[i].permute(1, 2, 0).cpu().numpy() * datamodule.std + datamodule.mean, 0, 1
)
ax.imshow(img)
ax.set_title(f"True: {classes[labels[i]]}")
ax.axis("off")
ax = axs[1, i]
for j in range(len(classes)):
ax.barh(classes[j], confidence_scores[i, j], color="blue")
ax.set_xlim(0, 1)
ax.set_xlabel("Confidence Score")
plt.tight_layout()
plt.show()
5. Estimate Prediction Sets with ConformalClsTHR#
Using alpha=0.01, we aim for a 1% error rate.
print("[Phase 2]: ConformalClsTHR calibration")
conformal_model = ConformalClsTHR(alpha=0.01, device="cuda")
routine_thr = ClassificationRoutine(
num_classes=10,
model=model,
loss=None, # No loss needed for evaluation
eval_ood=True,
post_processing=conformal_model,
ood_criterion="post_processing",
)
perf_thr = trainer.test(routine_thr, datamodule=datamodule)
[Phase 2]: ConformalClsTHR calibration
0%| | 0/79 [00:00<?, ?it/s]
1%|▏ | 1/79 [00:00<00:20, 3.85it/s]
10%|█ | 8/79 [00:00<00:02, 26.66it/s]
18%|█▊ | 14/79 [00:00<00:01, 37.04it/s]
25%|██▌ | 20/79 [00:00<00:01, 43.99it/s]
33%|███▎ | 26/79 [00:00<00:01, 48.87it/s]
41%|████ | 32/79 [00:00<00:00, 52.21it/s]
48%|████▊ | 38/79 [00:00<00:00, 54.53it/s]
56%|█████▌ | 44/79 [00:00<00:00, 56.17it/s]
63%|██████▎ | 50/79 [00:01<00:00, 57.20it/s]
72%|███████▏ | 57/79 [00:01<00:00, 58.15it/s]
80%|███████▉ | 63/79 [00:01<00:00, 58.66it/s]
87%|████████▋ | 69/79 [00:01<00:00, 59.04it/s]
95%|█████████▍| 75/79 [00:01<00:00, 59.30it/s]
100%|██████████| 79/79 [00:01<00:00, 50.13it/s]
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ Classification ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Acc │ 93.380% │
│ Brier │ 0.10812 │
│ Entropy │ 0.08849 │
│ NLL │ 0.26405 │
└──────────────┴───────────────────────────┘
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ Calibration ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ ECE │ 3.537% │
│ aECE │ 3.500% │
└──────────────┴───────────────────────────┘
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ OOD Detection ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ AUPR │ 86.585% │
│ AUROC │ 79.256% │
│ Entropy │ 0.08849 │
│ FPR95 │ 100.000% │
└──────────────┴───────────────────────────┘
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ Selective Classification ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ AUGRC │ 0.779% │
│ AURC │ 0.959% │
│ Cov@5Risk │ 96.510% │
│ Risk@80Cov │ 1.200% │
└──────────────┴───────────────────────────┘
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ Post-Processing ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ CoverageRate │ 0.99000 │
│ SetSize │ 1.52320 │
└──────────────┴───────────────────────────┘
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ Complexity ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ flops │ 142.19 G │
│ params │ 11.17 M │
└──────────────┴───────────────────────────┘
6. Visualization of ConformalClsTHR prediction sets#
inputs, labels = next(iter(datamodule.test_dataloader()[0]))
conformal_model.cuda()
confidence_scores = conformal_model.conformal(inputs.cuda())
classes = datamodule.test.classes
visualize_prediction_sets(inputs, labels, confidence_scores[:5].cpu(), classes)
7. Estimate Prediction Sets with ConformalClsAPS#
print("[Phase 3]: ConformalClsAPS calibration")
conformal_model = ConformalClsAPS(alpha=0.01, device="cuda", enable_ts=True)
routine_aps = ClassificationRoutine(
num_classes=10,
model=model,
loss=None, # No loss needed for evaluation
eval_ood=True,
post_processing=conformal_model,
ood_criterion="post_processing",
)
perf_aps = trainer.test(routine_aps, datamodule=datamodule)
conformal_model.cuda()
confidence_scores = conformal_model.conformal(inputs.cuda())
visualize_prediction_sets(inputs, labels, confidence_scores[:5].cpu(), classes)
[Phase 3]: ConformalClsAPS calibration
0%| | 0/79 [00:00<?, ?it/s]
1%|▏ | 1/79 [00:00<00:20, 3.80it/s]
9%|▉ | 7/79 [00:00<00:03, 23.37it/s]
16%|█▋ | 13/79 [00:00<00:01, 35.25it/s]
24%|██▍ | 19/79 [00:00<00:01, 43.05it/s]
32%|███▏ | 25/79 [00:00<00:01, 48.24it/s]
39%|███▉ | 31/79 [00:00<00:00, 51.71it/s]
47%|████▋ | 37/79 [00:00<00:00, 54.10it/s]
54%|█████▍ | 43/79 [00:00<00:00, 55.74it/s]
62%|██████▏ | 49/79 [00:01<00:00, 56.88it/s]
70%|██████▉ | 55/79 [00:01<00:00, 57.67it/s]
77%|███████▋ | 61/79 [00:01<00:00, 58.24it/s]
85%|████████▍ | 67/79 [00:01<00:00, 58.66it/s]
92%|█████████▏| 73/79 [00:01<00:00, 58.94it/s]
100%|██████████| 79/79 [00:01<00:00, 59.07it/s]
100%|██████████| 79/79 [00:01<00:00, 50.08it/s]
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ Classification ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Acc │ 93.380% │
│ Brier │ 0.10812 │
│ Entropy │ 0.08849 │
│ NLL │ 0.26405 │
└──────────────┴───────────────────────────┘
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ Calibration ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ ECE │ 3.537% │
│ aECE │ 3.500% │
└──────────────┴───────────────────────────┘
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ OOD Detection ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ AUPR │ 82.359% │
│ AUROC │ 73.184% │
│ Entropy │ 0.08849 │
│ FPR95 │ 100.000% │
└──────────────┴───────────────────────────┘
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ Selective Classification ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ AUGRC │ 0.779% │
│ AURC │ 0.959% │
│ Cov@5Risk │ 96.510% │
│ Risk@80Cov │ 1.200% │
└──────────────┴───────────────────────────┘
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ Post-Processing ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ CoverageRate │ 0.99340 │
│ SetSize │ 2.25320 │
└──────────────┴───────────────────────────┘
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ Complexity ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ flops │ 142.19 G │
│ params │ 11.17 M │
└──────────────┴───────────────────────────┘
8. Estimate Prediction Sets with ConformalClsRAPS#
print("[Phase 4]: ConformalClsRAPS calibration")
conformal_model = ConformalClsRAPS(
alpha=0.01, regularization_rank=3, penalty=0.002, model=model, device="cuda", enable_ts=True
)
routine_raps = ClassificationRoutine(
num_classes=10,
model=model,
loss=None, # No loss needed for evaluation
eval_ood=True,
post_processing=conformal_model,
ood_criterion="post_processing",
)
perf_raps = trainer.test(routine_raps, datamodule=datamodule)
conformal_model.cuda()
confidence_scores = conformal_model.conformal(inputs.cuda())
visualize_prediction_sets(inputs, labels, confidence_scores[:5].cpu(), classes)
[Phase 4]: ConformalClsRAPS calibration
0%| | 0/79 [00:00<?, ?it/s]
1%|▏ | 1/79 [00:00<00:17, 4.43it/s]
9%|▉ | 7/79 [00:00<00:02, 25.57it/s]
16%|█▋ | 13/79 [00:00<00:01, 37.23it/s]
24%|██▍ | 19/79 [00:00<00:01, 44.60it/s]
32%|███▏ | 25/79 [00:00<00:01, 49.30it/s]
39%|███▉ | 31/79 [00:00<00:00, 52.46it/s]
47%|████▋ | 37/79 [00:00<00:00, 54.44it/s]
54%|█████▍ | 43/79 [00:00<00:00, 56.03it/s]
62%|██████▏ | 49/79 [00:01<00:00, 57.08it/s]
70%|██████▉ | 55/79 [00:01<00:00, 57.74it/s]
77%|███████▋ | 61/79 [00:01<00:00, 58.18it/s]
85%|████████▍ | 67/79 [00:01<00:00, 58.53it/s]
92%|█████████▏| 73/79 [00:01<00:00, 58.77it/s]
100%|██████████| 79/79 [00:01<00:00, 58.86it/s]
100%|██████████| 79/79 [00:01<00:00, 51.05it/s]
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ Classification ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Acc │ 93.380% │
│ Brier │ 0.10812 │
│ Entropy │ 0.08849 │
│ NLL │ 0.26405 │
└──────────────┴───────────────────────────┘
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ Calibration ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ ECE │ 3.537% │
│ aECE │ 3.500% │
└──────────────┴───────────────────────────┘
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ OOD Detection ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ AUPR │ 82.663% │
│ AUROC │ 73.428% │
│ Entropy │ 0.08849 │
│ FPR95 │ 100.000% │
└──────────────┴───────────────────────────┘
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ Selective Classification ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ AUGRC │ 0.779% │
│ AURC │ 0.959% │
│ Cov@5Risk │ 96.510% │
│ Risk@80Cov │ 1.200% │
└──────────────┴───────────────────────────┘
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ Post-Processing ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ CoverageRate │ 0.99330 │
│ SetSize │ 2.16840 │
└──────────────┴───────────────────────────┘
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ Complexity ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ flops │ 142.19 G │
│ params │ 11.17 M │
└──────────────┴───────────────────────────┘
Summary#
In this tutorial, we explored how to apply conformal prediction to a pretrained ResNet on CIFAR-10. We evaluated three methods: Thresholding (THR), Adaptive Prediction Sets (APS), and Regularized APS (RAPS). For each, we calibrated on a validation set, evaluated OOD performance, and visualized prediction sets. You can explore further by adjusting alpha, changing the model, or testing on other datasets.
Total running time of the script: (1 minutes 19.935 seconds)