Note
Go to the end to download the full example code.
Evaluating Model Performance Under Distribution Shift with TorchUncertainty#
In this tutorial, we explore how to assess a model’s robustness when faced with distribution shifts. Specifically, we will:
Shortly train a ResNet18 model on the standard CIFAR-10 dataset.
Evaluate its performance on both the original CIFAR-10 test set and a corrupted version of CIFAR-10 to simulate distribution shift.
Analyze the model’s performance and robustness under these conditions.
By the end of this tutorial, you will understand how to use TorchUncertainty to evaluate and interpret model behavior under distribution shifts.
Imports and Setup#
First, we need to import the necessary libraries and set up our environment. This includes importing PyTorch, TorchUncertainty components, and TorchUncertainty’s Trainer (built on top of Lightning’s).
from torch import nn, optim
from torch_uncertainty import TUTrainer
from torch_uncertainty.datamodules import CIFAR10DataModule
from torch_uncertainty.models.classification.resnet import resnet
from torch_uncertainty.routines.classification import ClassificationRoutine
DataModule Setup#
TorchUncertainty provides convenient DataModules for standard datasets like CIFAR-10. DataModules handle data loading, preprocessing, and batching, simplifying the data pipeline. Each datamodule also include the corresponding out-of-distribution and distribution shift datasets, which are then used by the routine. For CIFAR-10, the corresponding distribution-shift dataset is CIFAR-10C as used in the community. To enable Distribution Shift evaluation, activate the eval_shift flag as done below.
# Initialize the CIFAR-10 DataModule
datamodule = CIFAR10DataModule(
root="./data",
batch_size=512,
num_workers=8,
eval_shift=True,
shift_severity=5, # Set severity level of the corruption (1 to 5): max-strength!
)
CIFAR-10C#
CIFAR-10C is a transformed version of CIFAR-10 test set. Dan Hendrycks and Thomas Dietterich applied computer vision transforms, known as corruptions to degrade the quality of the image and test deep learning models in adverse conditions. There are 15 (+4 optional) corruptions in total, including noise, blur, weather effects, etc. Each corruption has 5 different levels of severity ranging from small corruptions to very strong effects on the image. You can set the desired corruption level with the shift-severity argument. We refer to [1] for more details. You can get a more detailed overview and examples of the corruptions on the corresponding tutorial.
# These lines are usually not necessary (they are called by the Trainer),
# but we want to get access to the dataset before training
datamodule.prepare_data()
datamodule.setup("test")
# Let's check the CIFAR-10C, it should contain (15+4)*10000 images for the selected severity level.
print(datamodule.shift)
0%| | 0.00/170M [00:00<?, ?B/s]
0%| | 32.8k/170M [00:00<10:27, 272kB/s]
0%| | 98.3k/170M [00:00<06:48, 417kB/s]
0%| | 393k/170M [00:00<02:30, 1.13MB/s]
1%| | 983k/170M [00:00<01:18, 2.16MB/s]
1%| | 2.13M/170M [00:00<00:41, 4.05MB/s]
2%|▏ | 3.67M/170M [00:00<00:24, 6.88MB/s]
4%|▎ | 6.39M/170M [00:00<00:14, 11.7MB/s]
5%|▌ | 9.11M/170M [00:01<00:10, 15.1MB/s]
7%|▋ | 12.5M/170M [00:01<00:07, 20.0MB/s]
10%|▉ | 16.8M/170M [00:01<00:05, 26.3MB/s]
12%|█▏ | 21.1M/170M [00:01<00:04, 31.0MB/s]
14%|█▍ | 24.5M/170M [00:01<00:05, 28.5MB/s]
17%|█▋ | 28.9M/170M [00:01<00:04, 32.5MB/s]
19%|█▉ | 33.1M/170M [00:01<00:03, 35.0MB/s]
22%|██▏ | 36.7M/170M [00:01<00:03, 35.4MB/s]
24%|██▎ | 40.3M/170M [00:01<00:04, 32.3MB/s]
26%|██▌ | 44.2M/170M [00:02<00:03, 33.9MB/s]
28%|██▊ | 47.7M/170M [00:02<00:03, 32.7MB/s]
30%|██▉ | 51.1M/170M [00:02<00:03, 32.8MB/s]
32%|███▏ | 54.4M/170M [00:02<00:03, 33.1MB/s]
34%|███▍ | 57.9M/170M [00:02<00:03, 33.5MB/s]
36%|███▌ | 61.3M/170M [00:02<00:03, 31.9MB/s]
38%|███▊ | 64.8M/170M [00:02<00:03, 32.6MB/s]
40%|███▉ | 68.1M/170M [00:02<00:03, 32.2MB/s]
42%|████▏ | 71.4M/170M [00:02<00:03, 32.4MB/s]
44%|████▍ | 74.7M/170M [00:03<00:03, 30.9MB/s]
46%|████▌ | 77.9M/170M [00:03<00:03, 29.4MB/s]
47%|████▋ | 80.8M/170M [00:03<00:03, 28.0MB/s]
49%|████▉ | 83.7M/170M [00:03<00:03, 27.1MB/s]
51%|█████ | 86.4M/170M [00:03<00:03, 26.6MB/s]
52%|█████▏ | 89.1M/170M [00:03<00:03, 25.3MB/s]
54%|█████▍ | 91.7M/170M [00:03<00:03, 25.2MB/s]
55%|█████▌ | 94.2M/170M [00:03<00:03, 23.4MB/s]
57%|█████▋ | 96.8M/170M [00:03<00:03, 23.7MB/s]
58%|█████▊ | 99.2M/170M [00:04<00:03, 22.3MB/s]
60%|█████▉ | 101M/170M [00:04<00:03, 22.1MB/s]
61%|██████ | 104M/170M [00:04<00:03, 20.3MB/s]
62%|██████▏ | 106M/170M [00:04<00:03, 20.7MB/s]
63%|██████▎ | 108M/170M [00:04<00:03, 19.9MB/s]
65%|██████▍ | 110M/170M [00:04<00:03, 20.1MB/s]
66%|██████▌ | 112M/170M [00:04<00:02, 19.9MB/s]
67%|██████▋ | 114M/170M [00:04<00:02, 19.8MB/s]
68%|██████▊ | 116M/170M [00:04<00:02, 20.1MB/s]
69%|██████▉ | 118M/170M [00:05<00:02, 20.5MB/s]
71%|███████ | 121M/170M [00:05<00:02, 22.1MB/s]
72%|███████▏ | 123M/170M [00:05<00:02, 22.3MB/s]
74%|███████▎ | 126M/170M [00:05<00:02, 22.4MB/s]
75%|███████▍ | 128M/170M [00:05<00:01, 21.5MB/s]
76%|███████▌ | 130M/170M [00:05<00:01, 21.3MB/s]
78%|███████▊ | 132M/170M [00:05<00:01, 21.4MB/s]
79%|███████▉ | 134M/170M [00:05<00:01, 20.7MB/s]
80%|████████ | 137M/170M [00:05<00:01, 21.7MB/s]
82%|████████▏ | 139M/170M [00:05<00:01, 20.8MB/s]
83%|████████▎ | 141M/170M [00:06<00:01, 21.9MB/s]
84%|████████▍ | 144M/170M [00:06<00:01, 20.9MB/s]
86%|████████▌ | 146M/170M [00:06<00:01, 21.9MB/s]
87%|████████▋ | 148M/170M [00:06<00:01, 21.9MB/s]
88%|████████▊ | 151M/170M [00:06<00:00, 22.0MB/s]
90%|█████████ | 154M/170M [00:06<00:00, 23.9MB/s]
92%|█████████▏| 156M/170M [00:06<00:00, 22.6MB/s]
93%|█████████▎| 159M/170M [00:06<00:00, 23.7MB/s]
94%|█████████▍| 161M/170M [00:06<00:00, 21.7MB/s]
96%|█████████▌| 163M/170M [00:07<00:00, 21.8MB/s]
97%|█████████▋| 166M/170M [00:07<00:00, 22.0MB/s]
98%|█████████▊| 168M/170M [00:07<00:00, 22.2MB/s]
100%|█████████▉| 170M/170M [00:07<00:00, 21.4MB/s]
100%|██████████| 170M/170M [00:07<00:00, 23.0MB/s]
0%| | 0.00/2.92G [00:00<?, ?B/s]
0%| | 459k/2.92G [00:00<11:23, 4.27MB/s]
0%| | 1.54M/2.92G [00:00<06:22, 7.62MB/s]
0%| | 2.49M/2.92G [00:00<05:45, 8.44MB/s]
0%| | 3.60M/2.92G [00:00<05:15, 9.23MB/s]
0%| | 4.55M/2.92G [00:00<05:33, 8.73MB/s]
0%| | 5.47M/2.92G [00:00<05:37, 8.64MB/s]
0%| | 6.52M/2.92G [00:00<05:17, 9.16MB/s]
0%| | 7.57M/2.92G [00:00<05:05, 9.54MB/s]
0%| | 8.59M/2.92G [00:00<05:00, 9.68MB/s]
0%| | 9.63M/2.92G [00:01<04:54, 9.89MB/s]
0%| | 10.7M/2.92G [00:01<04:46, 10.1MB/s]
0%| | 11.8M/2.92G [00:01<04:54, 9.87MB/s]
0%| | 12.8M/2.92G [00:01<07:02, 6.87MB/s]
0%| | 13.7M/2.92G [00:01<06:30, 7.44MB/s]
0%| | 14.6M/2.92G [00:02<11:00, 4.39MB/s]
1%| | 15.5M/2.92G [00:02<09:19, 5.19MB/s]
1%| | 16.3M/2.92G [00:02<08:35, 5.63MB/s]
1%| | 17.4M/2.92G [00:02<07:21, 6.57MB/s]
1%| | 18.2M/2.92G [00:02<06:58, 6.93MB/s]
1%| | 19.1M/2.92G [00:02<06:36, 7.32MB/s]
1%| | 20.3M/2.92G [00:02<05:50, 8.26MB/s]
1%| | 21.3M/2.92G [00:02<05:25, 8.90MB/s]
1%| | 22.3M/2.92G [00:02<05:13, 9.23MB/s]
1%| | 23.4M/2.92G [00:02<05:04, 9.51MB/s]
1%| | 24.4M/2.92G [00:03<04:59, 9.67MB/s]
1%| | 25.4M/2.92G [00:03<09:42, 4.96MB/s]
1%| | 26.5M/2.92G [00:03<10:20, 4.66MB/s]
1%| | 27.4M/2.92G [00:03<09:03, 5.32MB/s]
1%| | 28.4M/2.92G [00:03<07:48, 6.17MB/s]
1%| | 29.4M/2.92G [00:04<06:55, 6.96MB/s]
1%| | 30.4M/2.92G [00:04<06:29, 7.42MB/s]
1%| | 31.5M/2.92G [00:04<05:52, 8.19MB/s]
1%| | 32.4M/2.92G [00:04<05:49, 8.25MB/s]
1%| | 33.3M/2.92G [00:04<05:40, 8.48MB/s]
1%| | 34.4M/2.92G [00:04<05:16, 9.10MB/s]
1%| | 35.5M/2.92G [00:04<05:04, 9.46MB/s]
1%|▏ | 36.5M/2.92G [00:04<04:59, 9.63MB/s]
1%|▏ | 37.6M/2.92G [00:04<04:46, 10.0MB/s]
1%|▏ | 38.7M/2.92G [00:05<04:44, 10.1MB/s]
1%|▏ | 39.7M/2.92G [00:05<07:10, 6.69MB/s]
1%|▏ | 40.6M/2.92G [00:05<08:35, 5.59MB/s]
1%|▏ | 41.6M/2.92G [00:05<07:20, 6.53MB/s]
1%|▏ | 42.4M/2.92G [00:05<08:24, 5.70MB/s]
1%|▏ | 43.2M/2.92G [00:05<07:54, 6.06MB/s]
2%|▏ | 44.3M/2.92G [00:06<06:41, 7.16MB/s]
2%|▏ | 45.4M/2.92G [00:06<06:02, 7.93MB/s]
2%|▏ | 46.4M/2.92G [00:06<05:41, 8.40MB/s]
2%|▏ | 47.4M/2.92G [00:06<05:29, 8.72MB/s]
2%|▏ | 48.5M/2.92G [00:06<05:13, 9.16MB/s]
2%|▏ | 49.5M/2.92G [00:06<05:03, 9.44MB/s]
2%|▏ | 50.6M/2.92G [00:06<05:04, 9.43MB/s]
2%|▏ | 51.5M/2.92G [00:06<05:01, 9.52MB/s]
2%|▏ | 52.6M/2.92G [00:06<04:59, 9.58MB/s]
2%|▏ | 53.6M/2.92G [00:06<04:52, 9.80MB/s]
2%|▏ | 54.6M/2.92G [00:07<04:50, 9.87MB/s]
2%|▏ | 55.6M/2.92G [00:07<07:17, 6.54MB/s]
2%|▏ | 56.5M/2.92G [00:07<11:25, 4.17MB/s]
2%|▏ | 65.8M/2.92G [00:07<02:36, 18.2MB/s]
3%|▎ | 77.6M/2.92G [00:07<01:17, 36.4MB/s]
3%|▎ | 89.4M/2.92G [00:08<00:53, 53.3MB/s]
3%|▎ | 101M/2.92G [00:08<00:41, 68.0MB/s]
4%|▍ | 113M/2.92G [00:08<00:35, 79.2MB/s]
4%|▍ | 124M/2.92G [00:08<00:31, 89.2MB/s]
5%|▍ | 135M/2.92G [00:08<00:29, 92.8MB/s]
5%|▍ | 145M/2.92G [00:08<00:28, 95.9MB/s]
5%|▌ | 155M/2.92G [00:08<00:30, 90.1MB/s]
6%|▌ | 165M/2.92G [00:09<00:51, 53.5MB/s]
6%|▌ | 172M/2.92G [00:09<01:02, 44.1MB/s]
6%|▌ | 179M/2.92G [00:09<01:09, 39.5MB/s]
6%|▋ | 184M/2.92G [00:09<01:14, 36.5MB/s]
6%|▋ | 188M/2.92G [00:09<01:18, 34.8MB/s]
7%|▋ | 192M/2.92G [00:09<01:16, 35.7MB/s]
7%|▋ | 196M/2.92G [00:10<01:14, 36.7MB/s]
7%|▋ | 201M/2.92G [00:10<01:10, 38.5MB/s]
7%|▋ | 205M/2.92G [00:10<01:11, 37.8MB/s]
7%|▋ | 209M/2.92G [00:10<01:10, 38.2MB/s]
7%|▋ | 213M/2.92G [00:10<01:08, 39.3MB/s]
7%|▋ | 217M/2.92G [00:10<01:13, 36.8MB/s]
8%|▊ | 221M/2.92G [00:10<01:12, 37.3MB/s]
8%|▊ | 225M/2.92G [00:10<01:35, 28.1MB/s]
8%|▊ | 228M/2.92G [00:11<01:37, 27.6MB/s]
8%|▊ | 231M/2.92G [00:11<01:34, 28.5MB/s]
8%|▊ | 235M/2.92G [00:11<01:30, 29.6MB/s]
8%|▊ | 238M/2.92G [00:11<01:30, 29.5MB/s]
8%|▊ | 241M/2.92G [00:11<01:29, 29.8MB/s]
8%|▊ | 244M/2.92G [00:11<01:33, 28.7MB/s]
8%|▊ | 247M/2.92G [00:11<01:30, 29.4MB/s]
9%|▊ | 250M/2.92G [00:11<01:28, 30.3MB/s]
9%|▉ | 256M/2.92G [00:11<01:09, 38.4MB/s]
9%|▉ | 268M/2.92G [00:11<00:43, 61.4MB/s]
10%|▉ | 280M/2.92G [00:12<00:33, 77.8MB/s]
10%|▉ | 291M/2.92G [00:12<00:29, 88.2MB/s]
10%|█ | 302M/2.92G [00:12<00:27, 94.5MB/s]
11%|█ | 312M/2.92G [00:12<00:27, 93.5MB/s]
11%|█ | 321M/2.92G [00:12<00:28, 90.5MB/s]
11%|█▏ | 330M/2.92G [00:12<00:29, 88.8MB/s]
12%|█▏ | 339M/2.92G [00:12<00:29, 87.3MB/s]
12%|█▏ | 348M/2.92G [00:12<00:32, 78.6MB/s]
12%|█▏ | 356M/2.92G [00:13<00:52, 48.9MB/s]
12%|█▏ | 362M/2.92G [00:13<00:58, 43.4MB/s]
13%|█▎ | 368M/2.92G [00:13<01:06, 38.6MB/s]
13%|█▎ | 372M/2.92G [00:13<01:13, 34.8MB/s]
13%|█▎ | 376M/2.92G [00:13<01:16, 33.3MB/s]
13%|█▎ | 381M/2.92G [00:13<01:12, 34.8MB/s]
13%|█▎ | 389M/2.92G [00:14<00:55, 45.4MB/s]
14%|█▎ | 399M/2.92G [00:14<00:43, 57.8MB/s]
14%|█▍ | 406M/2.92G [00:14<00:40, 61.6MB/s]
14%|█▍ | 413M/2.92G [00:14<00:41, 59.9MB/s]
14%|█▍ | 419M/2.92G [00:14<00:40, 61.5MB/s]
15%|█▍ | 426M/2.92G [00:14<00:39, 63.5MB/s]
15%|█▍ | 433M/2.92G [00:14<00:38, 65.3MB/s]
15%|█▌ | 440M/2.92G [00:14<00:37, 66.8MB/s]
15%|█▌ | 447M/2.92G [00:14<00:38, 64.0MB/s]
16%|█▌ | 453M/2.92G [00:15<00:59, 41.4MB/s]
16%|█▌ | 459M/2.92G [00:15<01:09, 35.4MB/s]
16%|█▌ | 463M/2.92G [00:15<01:10, 34.8MB/s]
16%|█▌ | 467M/2.92G [00:15<01:14, 33.1MB/s]
16%|█▌ | 471M/2.92G [00:15<01:12, 33.6MB/s]
16%|█▋ | 475M/2.92G [00:15<01:16, 32.1MB/s]
16%|█▋ | 478M/2.92G [00:16<01:25, 28.7MB/s]
17%|█▋ | 485M/2.92G [00:16<01:04, 37.7MB/s]
17%|█▋ | 493M/2.92G [00:16<00:51, 47.5MB/s]
17%|█▋ | 500M/2.92G [00:16<00:43, 55.3MB/s]
17%|█▋ | 508M/2.92G [00:16<00:39, 61.4MB/s]
18%|█▊ | 516M/2.92G [00:16<00:36, 65.9MB/s]
18%|█▊ | 524M/2.92G [00:16<00:34, 69.2MB/s]
18%|█▊ | 532M/2.92G [00:16<00:33, 71.9MB/s]
18%|█▊ | 539M/2.92G [00:16<00:34, 68.9MB/s]
19%|█▊ | 546M/2.92G [00:17<01:06, 35.9MB/s]
19%|█▉ | 551M/2.92G [00:17<01:11, 33.3MB/s]
19%|█▉ | 556M/2.92G [00:17<01:40, 23.5MB/s]
19%|█▉ | 560M/2.92G [00:18<01:36, 24.5MB/s]
19%|█▉ | 563M/2.92G [00:18<01:32, 25.5MB/s]
19%|█▉ | 566M/2.92G [00:18<01:28, 26.5MB/s]
20%|█▉ | 570M/2.92G [00:18<01:24, 27.7MB/s]
20%|█▉ | 573M/2.92G [00:18<01:21, 28.8MB/s]
20%|█▉ | 576M/2.92G [00:18<01:18, 29.8MB/s]
20%|█▉ | 580M/2.92G [00:18<01:16, 30.7MB/s]
20%|█▉ | 583M/2.92G [00:18<01:14, 31.5MB/s]
20%|██ | 586M/2.92G [00:18<01:11, 32.7MB/s]
20%|██ | 590M/2.92G [00:19<01:18, 29.8MB/s]
20%|██ | 593M/2.92G [00:19<01:15, 30.7MB/s]
20%|██ | 596M/2.92G [00:19<01:16, 30.4MB/s]
21%|██ | 600M/2.92G [00:19<01:13, 31.8MB/s]
21%|██ | 604M/2.92G [00:19<01:09, 33.2MB/s]
21%|██ | 607M/2.92G [00:19<01:11, 32.3MB/s]
21%|██ | 610M/2.92G [00:19<01:11, 32.2MB/s]
21%|██ | 614M/2.92G [00:19<01:15, 30.5MB/s]
21%|██ | 617M/2.92G [00:19<01:16, 30.0MB/s]
21%|██▏ | 620M/2.92G [00:19<01:12, 31.5MB/s]
21%|██▏ | 624M/2.92G [00:20<01:07, 33.8MB/s]
22%|██▏ | 628M/2.92G [00:20<01:04, 35.6MB/s]
22%|██▏ | 632M/2.92G [00:20<01:02, 36.8MB/s]
22%|██▏ | 636M/2.92G [00:20<01:00, 37.7MB/s]
22%|██▏ | 641M/2.92G [00:20<00:57, 39.5MB/s]
22%|██▏ | 645M/2.92G [00:20<00:57, 39.3MB/s]
22%|██▏ | 649M/2.92G [00:20<01:00, 37.8MB/s]
22%|██▏ | 653M/2.92G [00:20<00:56, 39.9MB/s]
23%|██▎ | 657M/2.92G [00:21<01:47, 21.0MB/s]
23%|██▎ | 661M/2.92G [00:21<01:36, 23.4MB/s]
23%|██▎ | 664M/2.92G [00:21<01:28, 25.4MB/s]
23%|██▎ | 667M/2.92G [00:21<01:28, 25.4MB/s]
23%|██▎ | 670M/2.92G [00:21<01:22, 27.1MB/s]
23%|██▎ | 674M/2.92G [00:21<01:24, 26.5MB/s]
23%|██▎ | 677M/2.92G [00:21<01:20, 27.8MB/s]
23%|██▎ | 680M/2.92G [00:21<01:14, 29.8MB/s]
23%|██▎ | 685M/2.92G [00:22<01:05, 34.2MB/s]
24%|██▎ | 690M/2.92G [00:22<00:58, 37.8MB/s]
24%|██▍ | 694M/2.92G [00:22<00:55, 40.3MB/s]
24%|██▍ | 698M/2.92G [00:22<00:59, 37.5MB/s]
24%|██▍ | 702M/2.92G [00:22<00:59, 37.1MB/s]
24%|██▍ | 707M/2.92G [00:22<00:57, 38.7MB/s]
24%|██▍ | 711M/2.92G [00:22<00:55, 39.8MB/s]
25%|██▍ | 716M/2.92G [00:22<00:50, 43.3MB/s]
25%|██▍ | 721M/2.92G [00:23<01:09, 31.4MB/s]
25%|██▍ | 724M/2.92G [00:23<01:10, 31.1MB/s]
25%|██▍ | 728M/2.92G [00:23<01:12, 30.1MB/s]
25%|██▌ | 731M/2.92G [00:23<01:11, 30.6MB/s]
25%|██▌ | 734M/2.92G [00:23<01:12, 30.3MB/s]
25%|██▌ | 737M/2.92G [00:23<01:11, 30.5MB/s]
25%|██▌ | 740M/2.92G [00:23<01:12, 30.1MB/s]
25%|██▌ | 744M/2.92G [00:23<01:09, 31.1MB/s]
26%|██▌ | 749M/2.92G [00:23<01:01, 35.5MB/s]
26%|██▌ | 754M/2.92G [00:24<00:54, 39.5MB/s]
26%|██▌ | 759M/2.92G [00:24<00:50, 43.1MB/s]
26%|██▌ | 764M/2.92G [00:24<00:47, 45.4MB/s]
26%|██▋ | 769M/2.92G [00:24<00:45, 46.8MB/s]
27%|██▋ | 774M/2.92G [00:24<00:45, 47.6MB/s]
27%|██▋ | 779M/2.92G [00:24<00:44, 48.3MB/s]
27%|██▋ | 784M/2.92G [00:24<00:47, 44.6MB/s]
27%|██▋ | 789M/2.92G [00:24<00:45, 46.8MB/s]
27%|██▋ | 795M/2.92G [00:24<00:58, 36.3MB/s]
27%|██▋ | 799M/2.92G [00:25<01:08, 30.7MB/s]
27%|██▋ | 802M/2.92G [00:25<01:08, 30.9MB/s]
28%|██▊ | 806M/2.92G [00:25<01:09, 30.6MB/s]
28%|██▊ | 809M/2.92G [00:25<01:11, 29.4MB/s]
28%|██▊ | 812M/2.92G [00:25<01:08, 30.5MB/s]
28%|██▊ | 815M/2.92G [00:25<01:11, 29.5MB/s]
28%|██▊ | 819M/2.92G [00:25<01:11, 29.2MB/s]
28%|██▊ | 823M/2.92G [00:25<01:04, 32.5MB/s]
28%|██▊ | 829M/2.92G [00:26<00:52, 40.1MB/s]
29%|██▊ | 835M/2.92G [00:26<00:45, 45.8MB/s]
29%|██▉ | 841M/2.92G [00:26<00:41, 49.9MB/s]
29%|██▉ | 847M/2.92G [00:26<00:39, 52.4MB/s]
29%|██▉ | 853M/2.92G [00:26<00:38, 54.2MB/s]
29%|██▉ | 858M/2.92G [00:26<00:44, 46.1MB/s]
30%|██▉ | 864M/2.92G [00:26<00:41, 48.9MB/s]
30%|██▉ | 869M/2.92G [00:26<00:40, 51.0MB/s]
30%|██▉ | 875M/2.92G [00:27<01:23, 24.4MB/s]
30%|███ | 879M/2.92G [00:27<01:18, 25.9MB/s]
30%|███ | 883M/2.92G [00:27<01:14, 27.3MB/s]
30%|███ | 886M/2.92G [00:27<01:12, 28.0MB/s]
30%|███ | 890M/2.92G [00:27<01:13, 27.7MB/s]
31%|███ | 893M/2.92G [00:27<01:18, 25.9MB/s]
31%|███ | 898M/2.92G [00:28<01:04, 31.2MB/s]
31%|███ | 903M/2.92G [00:28<00:55, 36.1MB/s]
31%|███ | 907M/2.92G [00:28<00:55, 36.2MB/s]
31%|███ | 912M/2.92G [00:28<00:50, 39.5MB/s]
31%|███▏ | 917M/2.92G [00:28<00:46, 42.9MB/s]
32%|███▏ | 922M/2.92G [00:28<00:44, 44.5MB/s]
32%|███▏ | 927M/2.92G [00:28<00:41, 47.5MB/s]
32%|███▏ | 933M/2.92G [00:28<00:39, 49.6MB/s]
32%|███▏ | 938M/2.92G [00:28<00:39, 50.4MB/s]
32%|███▏ | 943M/2.92G [00:29<01:28, 22.4MB/s]
32%|███▏ | 947M/2.92G [00:29<01:20, 24.3MB/s]
33%|███▎ | 951M/2.92G [00:29<01:15, 26.0MB/s]
33%|███▎ | 954M/2.92G [00:29<01:10, 27.8MB/s]
33%|███▎ | 958M/2.92G [00:29<01:08, 28.6MB/s]
33%|███▎ | 962M/2.92G [00:29<01:04, 30.3MB/s]
33%|███▎ | 967M/2.92G [00:30<00:53, 36.2MB/s]
33%|███▎ | 973M/2.92G [00:30<00:46, 41.8MB/s]
34%|███▎ | 979M/2.92G [00:30<00:41, 46.4MB/s]
34%|███▎ | 985M/2.92G [00:30<00:39, 49.5MB/s]
34%|███▍ | 990M/2.92G [00:30<00:38, 50.5MB/s]
34%|███▍ | 995M/2.92G [00:30<00:37, 51.0MB/s]
34%|███▍ | 1.00G/2.92G [00:30<00:37, 51.7MB/s]
34%|███▍ | 1.01G/2.92G [00:30<00:35, 53.3MB/s]
35%|███▍ | 1.01G/2.92G [00:30<00:34, 54.6MB/s]
35%|███▍ | 1.02G/2.92G [00:30<00:33, 56.6MB/s]
35%|███▌ | 1.02G/2.92G [00:31<00:54, 34.6MB/s]
35%|███▌ | 1.03G/2.92G [00:31<00:57, 33.0MB/s]
35%|███▌ | 1.03G/2.92G [00:31<01:01, 30.7MB/s]
36%|███▌ | 1.04G/2.92G [00:31<01:01, 30.7MB/s]
36%|███▌ | 1.04G/2.92G [00:31<01:01, 30.6MB/s]
36%|███▌ | 1.04G/2.92G [00:31<01:00, 30.8MB/s]
36%|███▌ | 1.05G/2.92G [00:32<01:03, 29.3MB/s]
36%|███▌ | 1.05G/2.92G [00:32<00:57, 32.5MB/s]
36%|███▌ | 1.06G/2.92G [00:32<00:49, 37.4MB/s]
36%|███▋ | 1.06G/2.92G [00:32<00:46, 39.5MB/s]
36%|███▋ | 1.06G/2.92G [00:32<00:45, 41.2MB/s]
37%|███▋ | 1.07G/2.92G [00:32<00:50, 36.5MB/s]
37%|███▋ | 1.07G/2.92G [00:32<00:52, 35.2MB/s]
37%|███▋ | 1.08G/2.92G [00:32<00:53, 34.6MB/s]
37%|███▋ | 1.08G/2.92G [00:32<00:53, 34.1MB/s]
37%|███▋ | 1.08G/2.92G [00:33<01:35, 19.3MB/s]
37%|███▋ | 1.09G/2.92G [00:33<01:28, 20.8MB/s]
37%|███▋ | 1.09G/2.92G [00:33<01:21, 22.5MB/s]
37%|███▋ | 1.09G/2.92G [00:33<01:15, 24.1MB/s]
38%|███▊ | 1.10G/2.92G [00:33<01:10, 25.7MB/s]
38%|███▊ | 1.10G/2.92G [00:33<01:11, 25.4MB/s]
38%|███▊ | 1.10G/2.92G [00:33<01:12, 25.1MB/s]
38%|███▊ | 1.10G/2.92G [00:34<01:06, 27.1MB/s]
38%|███▊ | 1.11G/2.92G [00:34<01:00, 29.7MB/s]
38%|███▊ | 1.11G/2.92G [00:34<00:55, 32.4MB/s]
38%|███▊ | 1.12G/2.92G [00:34<00:52, 34.3MB/s]
38%|███▊ | 1.12G/2.92G [00:34<00:50, 35.9MB/s]
39%|███▊ | 1.12G/2.92G [00:34<00:48, 36.7MB/s]
39%|███▊ | 1.13G/2.92G [00:34<00:47, 37.6MB/s]
39%|███▉ | 1.13G/2.92G [00:34<00:47, 37.8MB/s]
39%|███▉ | 1.14G/2.92G [00:34<00:45, 39.0MB/s]
39%|███▉ | 1.14G/2.92G [00:34<00:43, 40.5MB/s]
39%|███▉ | 1.14G/2.92G [00:35<01:36, 18.5MB/s]
39%|███▉ | 1.15G/2.92G [00:35<01:25, 20.6MB/s]
39%|███▉ | 1.15G/2.92G [00:35<01:18, 22.4MB/s]
40%|███▉ | 1.15G/2.92G [00:35<01:11, 24.8MB/s]
40%|███▉ | 1.16G/2.92G [00:35<01:10, 24.8MB/s]
40%|███▉ | 1.16G/2.92G [00:35<01:05, 26.9MB/s]
40%|███▉ | 1.16G/2.92G [00:36<01:02, 27.9MB/s]
40%|████ | 1.17G/2.92G [00:36<00:55, 31.6MB/s]
40%|████ | 1.17G/2.92G [00:36<00:49, 35.4MB/s]
40%|████ | 1.18G/2.92G [00:36<00:45, 37.9MB/s]
40%|████ | 1.18G/2.92G [00:36<00:43, 40.2MB/s]
41%|████ | 1.19G/2.92G [00:36<00:41, 41.9MB/s]
41%|████ | 1.19G/2.92G [00:36<00:40, 42.7MB/s]
41%|████ | 1.20G/2.92G [00:36<00:39, 43.3MB/s]
41%|████ | 1.20G/2.92G [00:36<00:41, 41.3MB/s]
41%|████▏ | 1.20G/2.92G [00:37<00:39, 43.7MB/s]
41%|████▏ | 1.21G/2.92G [00:37<01:16, 22.4MB/s]
42%|████▏ | 1.21G/2.92G [00:37<01:14, 23.0MB/s]
42%|████▏ | 1.22G/2.92G [00:37<01:09, 24.4MB/s]
42%|████▏ | 1.22G/2.92G [00:37<01:04, 26.5MB/s]
42%|████▏ | 1.22G/2.92G [00:37<00:59, 28.3MB/s]
42%|████▏ | 1.23G/2.92G [00:37<00:58, 29.1MB/s]
42%|████▏ | 1.23G/2.92G [00:38<00:57, 29.3MB/s]
42%|████▏ | 1.23G/2.92G [00:38<01:02, 27.0MB/s]
42%|████▏ | 1.24G/2.92G [00:38<01:03, 26.3MB/s]
42%|████▏ | 1.24G/2.92G [00:38<00:51, 32.7MB/s]
43%|████▎ | 1.24G/2.92G [00:38<00:45, 36.6MB/s]
43%|████▎ | 1.25G/2.92G [00:38<00:43, 38.4MB/s]
43%|████▎ | 1.25G/2.92G [00:38<00:40, 41.0MB/s]
43%|████▎ | 1.26G/2.92G [00:38<00:40, 41.4MB/s]
43%|████▎ | 1.26G/2.92G [00:38<00:38, 43.4MB/s]
43%|████▎ | 1.27G/2.92G [00:39<00:35, 46.6MB/s]
44%|████▎ | 1.27G/2.92G [00:39<00:54, 30.1MB/s]
44%|████▍ | 1.28G/2.92G [00:39<00:55, 29.3MB/s]
44%|████▍ | 1.28G/2.92G [00:39<00:57, 28.6MB/s]
44%|████▍ | 1.28G/2.92G [00:39<01:00, 27.1MB/s]
44%|████▍ | 1.29G/2.92G [00:39<01:02, 26.1MB/s]
44%|████▍ | 1.29G/2.92G [00:40<01:04, 25.3MB/s]
44%|████▍ | 1.29G/2.92G [00:40<01:03, 25.6MB/s]
44%|████▍ | 1.30G/2.92G [00:40<01:05, 24.9MB/s]
45%|████▍ | 1.30G/2.92G [00:40<00:51, 31.7MB/s]
45%|████▍ | 1.31G/2.92G [00:40<00:43, 36.7MB/s]
45%|████▍ | 1.31G/2.92G [00:40<00:38, 41.7MB/s]
45%|████▌ | 1.32G/2.92G [00:40<00:35, 45.4MB/s]
45%|████▌ | 1.32G/2.92G [00:40<00:37, 42.3MB/s]
45%|████▌ | 1.33G/2.92G [00:40<00:35, 44.6MB/s]
46%|████▌ | 1.33G/2.92G [00:40<00:34, 46.3MB/s]
46%|████▌ | 1.34G/2.92G [00:41<01:07, 23.4MB/s]
46%|████▌ | 1.34G/2.92G [00:41<01:08, 23.2MB/s]
46%|████▌ | 1.34G/2.92G [00:41<01:07, 23.2MB/s]
46%|████▌ | 1.35G/2.92G [00:41<01:04, 24.4MB/s]
46%|████▋ | 1.35G/2.92G [00:41<01:03, 24.7MB/s]
46%|████▋ | 1.35G/2.92G [00:42<00:57, 27.5MB/s]
47%|████▋ | 1.36G/2.92G [00:42<00:44, 35.2MB/s]
47%|████▋ | 1.37G/2.92G [00:42<00:37, 41.6MB/s]
47%|████▋ | 1.37G/2.92G [00:42<00:33, 46.1MB/s]
47%|████▋ | 1.38G/2.92G [00:42<00:30, 51.1MB/s]
47%|████▋ | 1.38G/2.92G [00:42<00:28, 54.8MB/s]
48%|████▊ | 1.39G/2.92G [00:42<00:28, 53.7MB/s]
48%|████▊ | 1.40G/2.92G [00:42<00:29, 51.5MB/s]
48%|████▊ | 1.40G/2.92G [00:43<00:40, 37.7MB/s]
48%|████▊ | 1.40G/2.92G [00:43<00:44, 34.1MB/s]
48%|████▊ | 1.41G/2.92G [00:43<00:45, 33.0MB/s]
48%|████▊ | 1.41G/2.92G [00:43<00:49, 30.4MB/s]
49%|████▊ | 1.42G/2.92G [00:43<00:49, 30.4MB/s]
49%|████▊ | 1.42G/2.92G [00:43<00:49, 30.3MB/s]
49%|████▊ | 1.42G/2.92G [00:43<00:57, 26.2MB/s]
49%|████▉ | 1.43G/2.92G [00:43<00:46, 31.8MB/s]
49%|████▉ | 1.43G/2.92G [00:44<00:39, 37.6MB/s]
49%|████▉ | 1.44G/2.92G [00:44<00:35, 41.3MB/s]
49%|████▉ | 1.44G/2.92G [00:44<00:36, 40.9MB/s]
50%|████▉ | 1.45G/2.92G [00:44<00:37, 38.9MB/s]
50%|████▉ | 1.45G/2.92G [00:44<00:37, 39.1MB/s]
50%|████▉ | 1.45G/2.92G [00:44<00:38, 38.0MB/s]
50%|████▉ | 1.46G/2.92G [00:44<00:37, 38.7MB/s]
50%|█████ | 1.46G/2.92G [00:44<00:37, 38.7MB/s]
50%|█████ | 1.47G/2.92G [00:45<00:51, 28.5MB/s]
50%|█████ | 1.47G/2.92G [00:45<00:52, 27.7MB/s]
50%|█████ | 1.47G/2.92G [00:45<00:49, 29.3MB/s]
51%|█████ | 1.48G/2.92G [00:45<00:48, 29.6MB/s]
51%|█████ | 1.48G/2.92G [00:45<00:48, 30.0MB/s]
51%|█████ | 1.48G/2.92G [00:45<00:53, 26.8MB/s]
51%|█████ | 1.49G/2.92G [00:45<00:51, 27.6MB/s]
51%|█████ | 1.49G/2.92G [00:45<00:45, 31.6MB/s]
51%|█████ | 1.49G/2.92G [00:45<00:40, 34.9MB/s]
51%|█████▏ | 1.50G/2.92G [00:46<00:37, 37.8MB/s]
52%|█████▏ | 1.50G/2.92G [00:46<00:35, 40.1MB/s]
52%|█████▏ | 1.51G/2.92G [00:46<00:34, 40.9MB/s]
52%|█████▏ | 1.51G/2.92G [00:46<00:39, 35.3MB/s]
52%|█████▏ | 1.52G/2.92G [00:46<00:37, 37.5MB/s]
52%|█████▏ | 1.52G/2.92G [00:46<00:35, 39.3MB/s]
52%|█████▏ | 1.53G/2.92G [00:46<00:49, 27.9MB/s]
52%|█████▏ | 1.53G/2.92G [00:46<00:48, 28.7MB/s]
52%|█████▏ | 1.53G/2.92G [00:47<00:48, 28.7MB/s]
53%|█████▎ | 1.53G/2.92G [00:47<00:49, 27.7MB/s]
53%|█████▎ | 1.54G/2.92G [00:47<00:49, 28.0MB/s]
53%|█████▎ | 1.54G/2.92G [00:47<00:53, 26.0MB/s]
53%|█████▎ | 1.54G/2.92G [00:47<00:54, 25.1MB/s]
53%|█████▎ | 1.55G/2.92G [00:47<00:52, 26.0MB/s]
53%|█████▎ | 1.55G/2.92G [00:47<00:53, 25.7MB/s]
53%|█████▎ | 1.55G/2.92G [00:47<00:46, 29.7MB/s]
53%|█████▎ | 1.56G/2.92G [00:47<00:38, 35.4MB/s]
54%|█████▎ | 1.56G/2.92G [00:48<00:34, 39.7MB/s]
54%|█████▎ | 1.57G/2.92G [00:48<00:32, 42.1MB/s]
54%|█████▍ | 1.57G/2.92G [00:48<00:32, 40.9MB/s]
54%|█████▍ | 1.58G/2.92G [00:48<00:30, 43.6MB/s]
54%|█████▍ | 1.58G/2.92G [00:48<00:31, 43.1MB/s]
54%|█████▍ | 1.59G/2.92G [00:48<00:33, 39.4MB/s]
54%|█████▍ | 1.59G/2.92G [00:48<00:46, 28.5MB/s]
55%|█████▍ | 1.59G/2.92G [00:48<00:49, 27.0MB/s]
55%|█████▍ | 1.60G/2.92G [00:49<00:49, 26.5MB/s]
55%|█████▍ | 1.60G/2.92G [00:49<00:50, 26.2MB/s]
55%|█████▍ | 1.60G/2.92G [00:49<00:50, 26.2MB/s]
55%|█████▌ | 1.61G/2.92G [00:49<00:50, 26.2MB/s]
55%|█████▌ | 1.61G/2.92G [00:49<00:48, 26.9MB/s]
55%|█████▌ | 1.61G/2.92G [00:49<00:41, 31.4MB/s]
55%|█████▌ | 1.62G/2.92G [00:49<00:37, 34.7MB/s]
56%|█████▌ | 1.62G/2.92G [00:49<00:34, 37.9MB/s]
56%|█████▌ | 1.63G/2.92G [00:49<00:32, 39.9MB/s]
56%|█████▌ | 1.63G/2.92G [00:50<00:30, 41.8MB/s]
56%|█████▌ | 1.64G/2.92G [00:50<00:30, 42.4MB/s]
56%|█████▌ | 1.64G/2.92G [00:50<00:29, 43.1MB/s]
56%|█████▋ | 1.64G/2.92G [00:50<00:29, 43.8MB/s]
57%|█████▋ | 1.65G/2.92G [00:50<00:28, 44.7MB/s]
57%|█████▋ | 1.65G/2.92G [00:50<00:27, 45.6MB/s]
57%|█████▋ | 1.66G/2.92G [00:50<00:34, 36.2MB/s]
57%|█████▋ | 1.66G/2.92G [00:50<00:38, 32.9MB/s]
57%|█████▋ | 1.67G/2.92G [00:51<00:39, 31.8MB/s]
57%|█████▋ | 1.67G/2.92G [00:51<00:40, 30.9MB/s]
57%|█████▋ | 1.67G/2.92G [00:51<00:46, 27.1MB/s]
57%|█████▋ | 1.68G/2.92G [00:51<00:49, 25.0MB/s]
58%|█████▊ | 1.68G/2.92G [00:51<00:49, 25.0MB/s]
58%|█████▊ | 1.68G/2.92G [00:51<00:50, 24.7MB/s]
58%|█████▊ | 1.69G/2.92G [00:51<00:38, 31.8MB/s]
58%|█████▊ | 1.69G/2.92G [00:51<00:32, 37.4MB/s]
58%|█████▊ | 1.70G/2.92G [00:51<00:29, 41.4MB/s]
58%|█████▊ | 1.70G/2.92G [00:52<00:27, 44.2MB/s]
58%|█████▊ | 1.71G/2.92G [00:52<00:26, 45.9MB/s]
59%|█████▊ | 1.71G/2.92G [00:52<00:25, 47.3MB/s]
59%|█████▉ | 1.72G/2.92G [00:52<00:24, 48.7MB/s]
59%|█████▉ | 1.72G/2.92G [00:52<00:24, 49.7MB/s]
59%|█████▉ | 1.73G/2.92G [00:52<00:24, 48.3MB/s]
59%|█████▉ | 1.73G/2.92G [00:53<00:51, 22.9MB/s]
59%|█████▉ | 1.74G/2.92G [00:53<00:50, 23.6MB/s]
60%|█████▉ | 1.74G/2.92G [00:53<00:47, 25.0MB/s]
60%|█████▉ | 1.74G/2.92G [00:53<00:46, 25.3MB/s]
60%|█████▉ | 1.75G/2.92G [00:53<00:43, 26.7MB/s]
60%|█████▉ | 1.75G/2.92G [00:53<00:38, 30.2MB/s]
60%|██████ | 1.75G/2.92G [00:53<00:34, 33.6MB/s]
60%|██████ | 1.76G/2.92G [00:53<00:32, 36.0MB/s]
60%|██████ | 1.76G/2.92G [00:53<00:31, 36.3MB/s]
61%|██████ | 1.77G/2.92G [00:54<00:33, 34.4MB/s]
61%|██████ | 1.77G/2.92G [00:54<00:34, 33.5MB/s]
61%|██████ | 1.77G/2.92G [00:54<00:33, 34.5MB/s]
61%|██████ | 1.78G/2.92G [00:54<00:31, 35.9MB/s]
61%|██████ | 1.78G/2.92G [00:54<00:30, 37.9MB/s]
61%|██████ | 1.79G/2.92G [00:54<00:57, 19.6MB/s]
61%|██████▏ | 1.79G/2.92G [00:55<00:51, 21.9MB/s]
61%|██████▏ | 1.79G/2.92G [00:55<00:47, 23.5MB/s]
62%|██████▏ | 1.80G/2.92G [00:55<00:46, 24.3MB/s]
62%|██████▏ | 1.80G/2.92G [00:55<00:44, 25.1MB/s]
62%|██████▏ | 1.80G/2.92G [00:55<00:40, 27.3MB/s]
62%|██████▏ | 1.81G/2.92G [00:55<00:32, 34.5MB/s]
62%|██████▏ | 1.81G/2.92G [00:55<00:28, 39.4MB/s]
62%|██████▏ | 1.82G/2.92G [00:55<00:25, 43.0MB/s]
62%|██████▏ | 1.82G/2.92G [00:55<00:23, 45.7MB/s]
63%|██████▎ | 1.83G/2.92G [00:55<00:22, 48.4MB/s]
63%|██████▎ | 1.83G/2.92G [00:56<00:21, 51.3MB/s]
63%|██████▎ | 1.84G/2.92G [00:56<00:20, 52.9MB/s]
63%|██████▎ | 1.85G/2.92G [00:56<00:20, 52.5MB/s]
63%|██████▎ | 1.85G/2.92G [00:56<00:20, 52.3MB/s]
64%|██████▎ | 1.86G/2.92G [00:56<00:47, 22.3MB/s]
64%|██████▎ | 1.86G/2.92G [00:57<00:45, 23.3MB/s]
64%|██████▍ | 1.86G/2.92G [00:57<00:42, 24.6MB/s]
64%|██████▍ | 1.87G/2.92G [00:57<00:42, 24.9MB/s]
64%|██████▍ | 1.87G/2.92G [00:57<00:37, 28.1MB/s]
64%|██████▍ | 1.88G/2.92G [00:57<00:32, 32.1MB/s]
64%|██████▍ | 1.88G/2.92G [00:57<00:29, 35.5MB/s]
65%|██████▍ | 1.89G/2.92G [00:57<00:27, 37.7MB/s]
65%|██████▍ | 1.89G/2.92G [00:57<00:25, 40.2MB/s]
65%|██████▍ | 1.89G/2.92G [00:57<00:24, 41.3MB/s]
65%|██████▌ | 1.90G/2.92G [00:58<00:23, 43.4MB/s]
65%|██████▌ | 1.90G/2.92G [00:58<00:22, 44.9MB/s]
65%|██████▌ | 1.91G/2.92G [00:58<00:21, 46.8MB/s]
66%|██████▌ | 1.91G/2.92G [00:58<00:43, 23.0MB/s]
66%|██████▌ | 1.92G/2.92G [00:58<00:42, 23.8MB/s]
66%|██████▌ | 1.92G/2.92G [00:58<00:41, 23.9MB/s]
66%|██████▌ | 1.92G/2.92G [00:59<00:40, 24.7MB/s]
66%|██████▌ | 1.93G/2.92G [00:59<00:40, 24.4MB/s]
66%|██████▌ | 1.93G/2.92G [00:59<00:37, 26.3MB/s]
66%|██████▋ | 1.93G/2.92G [00:59<00:33, 29.4MB/s]
66%|██████▋ | 1.94G/2.92G [00:59<00:30, 31.8MB/s]
67%|██████▋ | 1.94G/2.92G [00:59<00:28, 33.9MB/s]
67%|██████▋ | 1.95G/2.92G [00:59<00:27, 35.1MB/s]
67%|██████▋ | 1.95G/2.92G [00:59<00:26, 36.5MB/s]
67%|██████▋ | 1.95G/2.92G [00:59<00:27, 35.3MB/s]
67%|██████▋ | 1.96G/2.92G [01:00<00:28, 33.9MB/s]
67%|██████▋ | 1.96G/2.92G [01:00<00:26, 35.8MB/s]
67%|██████▋ | 1.97G/2.92G [01:00<00:25, 37.2MB/s]
68%|██████▊ | 1.97G/2.92G [01:00<00:24, 38.7MB/s]
68%|██████▊ | 1.97G/2.92G [01:00<00:32, 29.3MB/s]
68%|██████▊ | 1.98G/2.92G [01:00<00:32, 29.2MB/s]
68%|██████▊ | 1.98G/2.92G [01:00<00:32, 28.5MB/s]
68%|██████▊ | 1.98G/2.92G [01:00<00:34, 27.2MB/s]
68%|██████▊ | 1.99G/2.92G [01:01<00:36, 25.4MB/s]
68%|██████▊ | 1.99G/2.92G [01:01<00:37, 24.7MB/s]
68%|██████▊ | 1.99G/2.92G [01:01<00:39, 23.7MB/s]
68%|██████▊ | 2.00G/2.92G [01:01<00:32, 28.3MB/s]
69%|██████▊ | 2.00G/2.92G [01:01<00:27, 33.2MB/s]
69%|██████▊ | 2.01G/2.92G [01:01<00:24, 36.7MB/s]
69%|██████▉ | 2.01G/2.92G [01:01<00:29, 31.0MB/s]
69%|██████▉ | 2.01G/2.92G [01:01<00:28, 32.0MB/s]
69%|██████▉ | 2.02G/2.92G [01:01<00:25, 35.2MB/s]
69%|██████▉ | 2.02G/2.92G [01:02<00:24, 37.3MB/s]
69%|██████▉ | 2.03G/2.92G [01:02<00:22, 40.1MB/s]
70%|██████▉ | 2.03G/2.92G [01:02<00:21, 41.6MB/s]
70%|██████▉ | 2.04G/2.92G [01:02<00:20, 44.1MB/s]
70%|██████▉ | 2.04G/2.92G [01:02<00:29, 29.6MB/s]
70%|███████ | 2.04G/2.92G [01:02<00:30, 28.6MB/s]
70%|███████ | 2.05G/2.92G [01:02<00:32, 27.1MB/s]
70%|███████ | 2.05G/2.92G [01:02<00:31, 27.6MB/s]
70%|███████ | 2.05G/2.92G [01:03<00:30, 28.2MB/s]
70%|███████ | 2.06G/2.92G [01:03<00:30, 28.3MB/s]
71%|███████ | 2.06G/2.92G [01:03<00:33, 26.0MB/s]
71%|███████ | 2.06G/2.92G [01:03<00:31, 26.9MB/s]
71%|███████ | 2.07G/2.92G [01:03<00:29, 29.1MB/s]
71%|███████ | 2.07G/2.92G [01:03<00:23, 35.6MB/s]
71%|███████ | 2.08G/2.92G [01:03<00:21, 39.8MB/s]
71%|███████▏ | 2.08G/2.92G [01:03<00:20, 40.0MB/s]
71%|███████▏ | 2.08G/2.92G [01:04<00:27, 30.8MB/s]
72%|███████▏ | 2.09G/2.92G [01:04<00:26, 31.0MB/s]
72%|███████▏ | 2.09G/2.92G [01:04<00:27, 29.9MB/s]
72%|███████▏ | 2.10G/2.92G [01:04<00:29, 27.5MB/s]
72%|███████▏ | 2.10G/2.92G [01:04<00:42, 19.2MB/s]
72%|███████▏ | 2.10G/2.92G [01:04<00:40, 20.0MB/s]
72%|███████▏ | 2.10G/2.92G [01:05<00:38, 21.2MB/s]
72%|███████▏ | 2.11G/2.92G [01:05<00:35, 23.0MB/s]
72%|███████▏ | 2.11G/2.92G [01:05<00:33, 24.1MB/s]
72%|███████▏ | 2.11G/2.92G [01:05<00:27, 29.5MB/s]
73%|███████▎ | 2.12G/2.92G [01:05<00:22, 34.9MB/s]
73%|███████▎ | 2.12G/2.92G [01:05<00:21, 37.3MB/s]
73%|███████▎ | 2.13G/2.92G [01:05<00:19, 41.1MB/s]
73%|███████▎ | 2.13G/2.92G [01:05<00:18, 43.2MB/s]
73%|███████▎ | 2.14G/2.92G [01:05<00:18, 42.2MB/s]
73%|███████▎ | 2.14G/2.92G [01:05<00:17, 44.4MB/s]
74%|███████▎ | 2.15G/2.92G [01:06<00:18, 42.6MB/s]
74%|███████▍ | 2.15G/2.92G [01:06<00:16, 45.9MB/s]
74%|███████▍ | 2.16G/2.92G [01:06<00:16, 46.4MB/s]
74%|███████▍ | 2.16G/2.92G [01:06<00:22, 33.6MB/s]
74%|███████▍ | 2.17G/2.92G [01:06<00:22, 33.9MB/s]
74%|███████▍ | 2.17G/2.92G [01:06<00:22, 33.6MB/s]
74%|███████▍ | 2.17G/2.92G [01:06<00:23, 32.1MB/s]
75%|███████▍ | 2.18G/2.92G [01:06<00:24, 30.7MB/s]
75%|███████▍ | 2.18G/2.92G [01:07<00:25, 29.2MB/s]
75%|███████▍ | 2.18G/2.92G [01:07<00:25, 28.5MB/s]
75%|███████▍ | 2.19G/2.92G [01:07<00:27, 27.1MB/s]
75%|███████▌ | 2.19G/2.92G [01:07<00:22, 32.2MB/s]
75%|███████▌ | 2.20G/2.92G [01:07<00:19, 36.4MB/s]
75%|███████▌ | 2.20G/2.92G [01:07<00:19, 37.1MB/s]
76%|███████▌ | 2.20G/2.92G [01:07<00:18, 38.4MB/s]
76%|███████▌ | 2.21G/2.92G [01:07<00:17, 39.5MB/s]
76%|███████▌ | 2.21G/2.92G [01:07<00:17, 40.1MB/s]
76%|███████▌ | 2.22G/2.92G [01:08<00:17, 41.1MB/s]
76%|███████▌ | 2.22G/2.92G [01:08<00:17, 40.2MB/s]
76%|███████▋ | 2.23G/2.92G [01:08<00:16, 41.1MB/s]
76%|███████▋ | 2.23G/2.92G [01:08<00:35, 19.6MB/s]
77%|███████▋ | 2.23G/2.92G [01:08<00:31, 21.6MB/s]
77%|███████▋ | 2.24G/2.92G [01:08<00:29, 23.5MB/s]
77%|███████▋ | 2.24G/2.92G [01:09<00:27, 24.5MB/s]
77%|███████▋ | 2.24G/2.92G [01:09<00:26, 25.3MB/s]
77%|███████▋ | 2.25G/2.92G [01:09<00:26, 25.6MB/s]
77%|███████▋ | 2.25G/2.92G [01:09<00:23, 28.7MB/s]
77%|███████▋ | 2.25G/2.92G [01:09<00:20, 32.6MB/s]
77%|███████▋ | 2.26G/2.92G [01:09<00:18, 35.9MB/s]
78%|███████▊ | 2.26G/2.92G [01:09<00:16, 39.0MB/s]
78%|███████▊ | 2.27G/2.92G [01:09<00:15, 40.8MB/s]
78%|███████▊ | 2.27G/2.92G [01:09<00:16, 38.8MB/s]
78%|███████▊ | 2.28G/2.92G [01:09<00:16, 38.3MB/s]
78%|███████▊ | 2.28G/2.92G [01:10<00:16, 39.4MB/s]
78%|███████▊ | 2.28G/2.92G [01:10<00:15, 41.8MB/s]
78%|███████▊ | 2.29G/2.92G [01:10<00:29, 21.4MB/s]
79%|███████▊ | 2.29G/2.92G [01:10<00:27, 22.9MB/s]
79%|███████▊ | 2.30G/2.92G [01:10<00:25, 24.5MB/s]
79%|███████▉ | 2.30G/2.92G [01:10<00:25, 24.4MB/s]
79%|███████▉ | 2.30G/2.92G [01:11<00:24, 24.8MB/s]
79%|███████▉ | 2.30G/2.92G [01:11<00:22, 27.6MB/s]
79%|███████▉ | 2.31G/2.92G [01:11<00:17, 34.5MB/s]
79%|███████▉ | 2.32G/2.92G [01:11<00:14, 41.4MB/s]
80%|███████▉ | 2.32G/2.92G [01:11<00:12, 46.2MB/s]
80%|███████▉ | 2.33G/2.92G [01:11<00:11, 49.8MB/s]
80%|███████▉ | 2.33G/2.92G [01:11<00:11, 52.4MB/s]
80%|████████ | 2.34G/2.92G [01:11<00:10, 54.2MB/s]
80%|████████ | 2.35G/2.92G [01:11<00:10, 55.4MB/s]
81%|████████ | 2.35G/2.92G [01:11<00:10, 54.6MB/s]
81%|████████ | 2.36G/2.92G [01:12<00:11, 50.1MB/s]
81%|████████ | 2.36G/2.92G [01:12<00:11, 47.7MB/s]
81%|████████ | 2.37G/2.92G [01:12<00:11, 46.3MB/s]
81%|████████▏ | 2.37G/2.92G [01:12<00:25, 21.6MB/s]
81%|████████▏ | 2.38G/2.92G [01:13<00:23, 22.8MB/s]
82%|████████▏ | 2.38G/2.92G [01:13<00:22, 23.5MB/s]
82%|████████▏ | 2.38G/2.92G [01:13<00:23, 22.4MB/s]
82%|████████▏ | 2.39G/2.92G [01:13<00:22, 24.1MB/s]
82%|████████▏ | 2.39G/2.92G [01:13<00:18, 28.6MB/s]
82%|████████▏ | 2.39G/2.92G [01:13<00:15, 33.0MB/s]
82%|████████▏ | 2.40G/2.92G [01:13<00:14, 36.5MB/s]
82%|████████▏ | 2.40G/2.92G [01:13<00:13, 38.5MB/s]
83%|████████▎ | 2.41G/2.92G [01:13<00:13, 38.7MB/s]
83%|████████▎ | 2.41G/2.92G [01:14<00:12, 39.5MB/s]
83%|████████▎ | 2.42G/2.92G [01:14<00:12, 41.6MB/s]
83%|████████▎ | 2.42G/2.92G [01:14<00:11, 43.1MB/s]
83%|████████▎ | 2.43G/2.92G [01:14<00:10, 45.0MB/s]
83%|████████▎ | 2.43G/2.92G [01:14<00:10, 45.7MB/s]
83%|████████▎ | 2.44G/2.92G [01:14<00:15, 30.7MB/s]
84%|████████▎ | 2.44G/2.92G [01:14<00:15, 30.1MB/s]
84%|████████▎ | 2.44G/2.92G [01:14<00:16, 29.3MB/s]
84%|████████▍ | 2.45G/2.92G [01:15<00:17, 26.9MB/s]
84%|████████▍ | 2.45G/2.92G [01:15<00:17, 27.5MB/s]
84%|████████▍ | 2.45G/2.92G [01:15<00:16, 27.9MB/s]
84%|████████▍ | 2.46G/2.92G [01:15<00:17, 25.9MB/s]
84%|████████▍ | 2.46G/2.92G [01:15<00:17, 25.8MB/s]
84%|████████▍ | 2.46G/2.92G [01:15<00:16, 27.0MB/s]
85%|████████▍ | 2.47G/2.92G [01:15<00:13, 32.9MB/s]
85%|████████▍ | 2.47G/2.92G [01:15<00:11, 38.3MB/s]
85%|████████▍ | 2.48G/2.92G [01:15<00:10, 41.6MB/s]
85%|████████▌ | 2.48G/2.92G [01:16<00:10, 40.8MB/s]
85%|████████▌ | 2.49G/2.92G [01:16<00:10, 42.3MB/s]
85%|████████▌ | 2.49G/2.92G [01:16<00:09, 43.8MB/s]
85%|████████▌ | 2.49G/2.92G [01:16<00:09, 44.2MB/s]
86%|████████▌ | 2.50G/2.92G [01:16<00:08, 46.9MB/s]
86%|████████▌ | 2.51G/2.92G [01:16<00:08, 49.0MB/s]
86%|████████▌ | 2.51G/2.92G [01:17<00:16, 24.3MB/s]
86%|████████▌ | 2.51G/2.92G [01:17<00:15, 25.3MB/s]
86%|████████▋ | 2.52G/2.92G [01:17<00:17, 23.5MB/s]
86%|████████▋ | 2.52G/2.92G [01:17<00:17, 23.3MB/s]
86%|████████▋ | 2.52G/2.92G [01:17<00:16, 23.3MB/s]
87%|████████▋ | 2.53G/2.92G [01:17<00:16, 24.2MB/s]
87%|████████▋ | 2.53G/2.92G [01:17<00:12, 31.4MB/s]
87%|████████▋ | 2.54G/2.92G [01:17<00:09, 38.2MB/s]
87%|████████▋ | 2.54G/2.92G [01:17<00:08, 43.8MB/s]
87%|████████▋ | 2.55G/2.92G [01:18<00:07, 47.8MB/s]
88%|████████▊ | 2.56G/2.92G [01:18<00:07, 50.8MB/s]
88%|████████▊ | 2.56G/2.92G [01:18<00:06, 53.1MB/s]
88%|████████▊ | 2.57G/2.92G [01:18<00:06, 55.5MB/s]
88%|████████▊ | 2.57G/2.92G [01:18<00:06, 57.0MB/s]
88%|████████▊ | 2.58G/2.92G [01:18<00:05, 59.5MB/s]
89%|████████▊ | 2.59G/2.92G [01:19<00:11, 28.8MB/s]
89%|████████▉ | 2.59G/2.92G [01:19<00:11, 28.0MB/s]
89%|████████▉ | 2.59G/2.92G [01:19<00:11, 27.6MB/s]
89%|████████▉ | 2.60G/2.92G [01:19<00:11, 28.0MB/s]
89%|████████▉ | 2.60G/2.92G [01:19<00:10, 31.0MB/s]
89%|████████▉ | 2.61G/2.92G [01:19<00:07, 38.7MB/s]
90%|████████▉ | 2.62G/2.92G [01:19<00:06, 45.2MB/s]
90%|████████▉ | 2.62G/2.92G [01:19<00:05, 50.4MB/s]
90%|█████████ | 2.63G/2.92G [01:20<00:05, 51.6MB/s]
90%|█████████ | 2.63G/2.92G [01:20<00:05, 55.9MB/s]
91%|█████████ | 2.64G/2.92G [01:20<00:04, 59.3MB/s]
91%|█████████ | 2.65G/2.92G [01:20<00:04, 62.4MB/s]
91%|█████████ | 2.66G/2.92G [01:20<00:04, 64.6MB/s]
91%|█████████ | 2.66G/2.92G [01:20<00:03, 66.6MB/s]
91%|█████████▏| 2.67G/2.92G [01:20<00:05, 43.6MB/s]
92%|█████████▏| 2.68G/2.92G [01:20<00:06, 39.4MB/s]
92%|█████████▏| 2.68G/2.92G [01:21<00:06, 36.8MB/s]
92%|█████████▏| 2.68G/2.92G [01:21<00:07, 33.5MB/s]
92%|█████████▏| 2.69G/2.92G [01:21<00:06, 33.9MB/s]
92%|█████████▏| 2.69G/2.92G [01:21<00:07, 28.8MB/s]
92%|█████████▏| 2.70G/2.92G [01:21<00:07, 30.6MB/s]
93%|█████████▎| 2.70G/2.92G [01:21<00:05, 40.1MB/s]
93%|█████████▎| 2.71G/2.92G [01:21<00:04, 48.4MB/s]
93%|█████████▎| 2.72G/2.92G [01:21<00:03, 54.5MB/s]
93%|█████████▎| 2.72G/2.92G [01:22<00:03, 58.9MB/s]
94%|█████████▎| 2.73G/2.92G [01:22<00:02, 62.4MB/s]
94%|█████████▍| 2.74G/2.92G [01:22<00:02, 66.4MB/s]
94%|█████████▍| 2.75G/2.92G [01:22<00:02, 69.3MB/s]
94%|█████████▍| 2.75G/2.92G [01:22<00:02, 71.7MB/s]
95%|█████████▍| 2.76G/2.92G [01:22<00:02, 71.6MB/s]
95%|█████████▍| 2.77G/2.92G [01:23<00:04, 36.8MB/s]
95%|█████████▌| 2.77G/2.92G [01:23<00:04, 32.8MB/s]
95%|█████████▌| 2.78G/2.92G [01:23<00:04, 33.0MB/s]
95%|█████████▌| 2.78G/2.92G [01:23<00:04, 30.4MB/s]
96%|█████████▌| 2.79G/2.92G [01:23<00:03, 33.1MB/s]
96%|█████████▌| 2.80G/2.92G [01:23<00:02, 42.8MB/s]
96%|█████████▌| 2.80G/2.92G [01:23<00:02, 50.9MB/s]
96%|█████████▋| 2.81G/2.92G [01:23<00:01, 57.8MB/s]
97%|█████████▋| 2.82G/2.92G [01:24<00:01, 63.3MB/s]
97%|█████████▋| 2.83G/2.92G [01:24<00:01, 67.5MB/s]
97%|█████████▋| 2.84G/2.92G [01:24<00:01, 72.3MB/s]
97%|█████████▋| 2.84G/2.92G [01:24<00:00, 75.6MB/s]
98%|█████████▊| 2.85G/2.92G [01:24<00:00, 77.6MB/s]
98%|█████████▊| 2.86G/2.92G [01:24<00:00, 75.8MB/s]
98%|█████████▊| 2.87G/2.92G [01:25<00:01, 34.9MB/s]
98%|█████████▊| 2.87G/2.92G [01:25<00:01, 34.9MB/s]
99%|█████████▊| 2.88G/2.92G [01:25<00:01, 34.4MB/s]
99%|█████████▉| 2.88G/2.92G [01:25<00:01, 33.6MB/s]
99%|█████████▉| 2.89G/2.92G [01:25<00:00, 35.3MB/s]
99%|█████████▉| 2.90G/2.92G [01:25<00:00, 46.1MB/s]
100%|█████████▉| 2.90G/2.92G [01:25<00:00, 54.2MB/s]
100%|█████████▉| 2.91G/2.92G [01:25<00:00, 62.3MB/s]
100%|██████████| 2.92G/2.92G [01:26<00:00, 33.9MB/s]
Dataset CIFAR10C
Number of datapoints: 190000
Root location: data/CIFAR-10-C
StandardTransform
Transform: Compose(
ToImage()
ToDtype(scale=True)
Normalize(mean=[0.4914, 0.4822, 0.4465], std=[0.2023, 0.1994, 0.201], inplace=False)
)
Model Initialization#
We will use the ResNet18 architecture, a widely adopted convolutional neural network known for its deep residual learning capabilities. The model is initialized with 10 output classes corresponding to the CIFAR-10 dataset categories.
# Initialize the ResNet18 model with 10 output classes
model = resnet(arch=18, in_channels=3, num_classes=10)
Define the Classification Routine#
The ClassificationRoutine is one of the most crucial building blocks in TorchUncertainty. It streamlines the training and evaluation processes. It integrates the model, loss function, and optimizer into a cohesive routine compatible with PyTorch Lightning’s Trainer. This abstraction simplifies the implementation of standard training loops and evaluation protocols. To come back to what matters in this tutorial, the routine also handles the evaluation of the performance of the model under distribution shift detection. To enable it, activate the eval_shift flag. Note that you can also evaluate the Out-of-distribution detection at the same time by also setting eval_ood to True.
# Define the loss function: Cross-Entropy Loss for multi-class classification
criterion = nn.CrossEntropyLoss()
# Define the optimizer: Adam optimizer with a learning rate of 0.001
optimizer = optim.Adam(model.parameters(), lr=0.001)
# Initialize the ClassificationRoutine with the model, number of classes, loss function, and optimizer
routine = ClassificationRoutine(
model=model, num_classes=10, loss=criterion, optim_recipe=optimizer, eval_shift=True
)
Training the Model#
With the routine defined, we can now set up the TUTrainer and commence training. The TUTrainer handles the training loop, including epoch management, logging, and checkpointing. We specify the maximum number of epochs, the precision and the device to be used.
# Initialize the TUTrainer with a maximum of 10 epochs and the specified device
trainer = TUTrainer(
max_epochs=10, precision="16-mixed", accelerator="cuda", devices=1, enable_progress_bar=False
)
# Begin training the model using the CIFAR-10 DataModule
trainer.fit(routine, datamodule=datamodule)
Evaluating on In-Distribution and Distribution-shifted Data#
Now that the model is trained, we can evaluate its performance on the original in-distribution test set, as well as the distribution-shifted set. Typing the next line will automatically compute the in-distribution metrics as well as their values on the distribution-shifted set.
# Evaluate the trained model on the original CIFAR-10 test set and on CIFAR-10C
results = trainer.test(routine, datamodule=datamodule)
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ Classification ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Acc │ 76.040% │
│ Brier │ 0.33169 │
│ Entropy │ 0.56686 │
│ NLL │ 0.68303 │
└──────────────┴───────────────────────────┘
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ Calibration ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ ECE │ 4.260% │
│ aECE │ 4.178% │
└──────────────┴───────────────────────────┘
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ Selective Classification ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ AUGRC │ 5.781% │
│ AURC │ 7.552% │
│ Cov@5Risk │ nan% │
│ Risk@80Cov │ 15.238% │
└──────────────┴───────────────────────────┘
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ Distribution Shift lvl5 ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ AUGRC │ 7.084% │
│ AURC │ 9.525% │
│ Acc │ 73.101% │
│ Brier │ 0.37237 │
│ Cov@5Risk │ nan% │
│ ECE │ 5.066% │
│ NLL │ 0.77899 │
│ Risk@80Cov │ 18.422% │
│ aECE │ 5.064% │
└──────────────┴───────────────────────────┘
Distribution-shift metrics#
The distribution shift metrics are computed only when the eval_shift flag of the routine is True. In this case, the values of the metrics are shown last. They correspond to the in-distribution metrics but computed on the distribution-shifted datasets, hence the worse results.
Total running time of the script: (2 minutes 34.911 seconds)