.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_tutorials/Classification/tutorial_distribution_shift.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_tutorials_Classification_tutorial_distribution_shift.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_tutorials_Classification_tutorial_distribution_shift.py:


Evaluating Model Performance Under Distribution Shift with TorchUncertainty
===========================================================================

In this tutorial, we explore how to assess a model's robustness when faced with distribution shifts.
Specifically, we will:

- Shortly train a **ResNet18** model on the standard **CIFAR-10** dataset.
- Evaluate its performance on both the original CIFAR-10 test set and a corrupted version of CIFAR-10 to simulate distribution shift.
- Analyze the model's performance and robustness under these conditions.

By the end of this tutorial, you will understand how to use TorchUncertainty to evaluate and interpret model behavior under distribution shifts.

.. GENERATED FROM PYTHON SOURCE LINES 17-22

Imports and Setup
-----------------

First, we need to import the necessary libraries and set up our environment.
This includes importing PyTorch, TorchUncertainty components, and TorchUncertainty's Trainer (built on top of Lightning's).

.. GENERATED FROM PYTHON SOURCE LINES 22-30

.. code-block:: Python


    from torch import nn, optim

    from torch_uncertainty import TUTrainer
    from torch_uncertainty.datamodules import CIFAR10DataModule
    from torch_uncertainty.models.classification.resnet import resnet
    from torch_uncertainty.routines.classification import ClassificationRoutine








.. GENERATED FROM PYTHON SOURCE LINES 31-39

DataModule Setup
----------------

TorchUncertainty provides convenient DataModules for standard datasets like CIFAR-10.
DataModules handle data loading, preprocessing, and batching, simplifying the data pipeline. Each datamodule
also include the corresponding out-of-distribution and distribution shift datasets, which are then used by the routine.
For CIFAR-10, the corresponding distribution-shift dataset is CIFAR-10C as used in the community.
To enable Distribution Shift evaluation, activate the `eval_shift` flag as done below.

.. GENERATED FROM PYTHON SOURCE LINES 39-49

.. code-block:: Python


    # Initialize the CIFAR-10 DataModule
    datamodule = CIFAR10DataModule(
        root="./data",
        batch_size=512,
        num_workers=8,
        eval_shift=True,
        shift_severity=5,  # Set severity level of the corruption (1 to 5): max-strength!
    )








.. GENERATED FROM PYTHON SOURCE LINES 50-59

CIFAR-10C
---------

CIFAR-10C is a transformed version of CIFAR-10 test set. Dan Hendrycks and Thomas Dietterich applied computer vision
transforms, known as corruptions to degrade the quality of the image and test deep learning models in adverse conditions.
There are 15 (+4 optional) corruptions in total, including noise, blur, weather effects, etc. Each corruption has 5 different
levels of severity ranging from small corruptions to very strong effects on the image. You can set the desired corruption level with
the shift-severity argument. We refer to [1] for more details.
You can get a more detailed overview and examples of the corruptions on the corresponding tutorial.

.. GENERATED FROM PYTHON SOURCE LINES 59-68

.. code-block:: Python


    # These lines are usually not necessary (they are called by the Trainer),
    # but we want to get access to the dataset before training
    datamodule.prepare_data()
    datamodule.setup("test")

    # Let's check the CIFAR-10C, it should contain (15+4)*10000 images for the selected severity level.
    print(datamodule.shift)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


      0%|          | 0.00/170M [00:00<?, ?B/s]
      0%|          | 32.8k/170M [00:00<10:27, 272kB/s]
      0%|          | 98.3k/170M [00:00<06:48, 417kB/s]
      0%|          | 393k/170M [00:00<02:30, 1.13MB/s]
      1%|          | 983k/170M [00:00<01:18, 2.16MB/s]
      1%|          | 2.13M/170M [00:00<00:41, 4.05MB/s]
      2%|▏         | 3.67M/170M [00:00<00:24, 6.88MB/s]
      4%|▎         | 6.39M/170M [00:00<00:14, 11.7MB/s]
      5%|▌         | 9.11M/170M [00:01<00:10, 15.1MB/s]
      7%|▋         | 12.5M/170M [00:01<00:07, 20.0MB/s]
     10%|▉         | 16.8M/170M [00:01<00:05, 26.3MB/s]
     12%|█▏        | 21.1M/170M [00:01<00:04, 31.0MB/s]
     14%|█▍        | 24.5M/170M [00:01<00:05, 28.5MB/s]
     17%|█▋        | 28.9M/170M [00:01<00:04, 32.5MB/s]
     19%|█▉        | 33.1M/170M [00:01<00:03, 35.0MB/s]
     22%|██▏       | 36.7M/170M [00:01<00:03, 35.4MB/s]
     24%|██▎       | 40.3M/170M [00:01<00:04, 32.3MB/s]
     26%|██▌       | 44.2M/170M [00:02<00:03, 33.9MB/s]
     28%|██▊       | 47.7M/170M [00:02<00:03, 32.7MB/s]
     30%|██▉       | 51.1M/170M [00:02<00:03, 32.8MB/s]
     32%|███▏      | 54.4M/170M [00:02<00:03, 33.1MB/s]
     34%|███▍      | 57.9M/170M [00:02<00:03, 33.5MB/s]
     36%|███▌      | 61.3M/170M [00:02<00:03, 31.9MB/s]
     38%|███▊      | 64.8M/170M [00:02<00:03, 32.6MB/s]
     40%|███▉      | 68.1M/170M [00:02<00:03, 32.2MB/s]
     42%|████▏     | 71.4M/170M [00:02<00:03, 32.4MB/s]
     44%|████▍     | 74.7M/170M [00:03<00:03, 30.9MB/s]
     46%|████▌     | 77.9M/170M [00:03<00:03, 29.4MB/s]
     47%|████▋     | 80.8M/170M [00:03<00:03, 28.0MB/s]
     49%|████▉     | 83.7M/170M [00:03<00:03, 27.1MB/s]
     51%|█████     | 86.4M/170M [00:03<00:03, 26.6MB/s]
     52%|█████▏    | 89.1M/170M [00:03<00:03, 25.3MB/s]
     54%|█████▍    | 91.7M/170M [00:03<00:03, 25.2MB/s]
     55%|█████▌    | 94.2M/170M [00:03<00:03, 23.4MB/s]
     57%|█████▋    | 96.8M/170M [00:03<00:03, 23.7MB/s]
     58%|█████▊    | 99.2M/170M [00:04<00:03, 22.3MB/s]
     60%|█████▉    | 101M/170M [00:04<00:03, 22.1MB/s] 
     61%|██████    | 104M/170M [00:04<00:03, 20.3MB/s]
     62%|██████▏   | 106M/170M [00:04<00:03, 20.7MB/s]
     63%|██████▎   | 108M/170M [00:04<00:03, 19.9MB/s]
     65%|██████▍   | 110M/170M [00:04<00:03, 20.1MB/s]
     66%|██████▌   | 112M/170M [00:04<00:02, 19.9MB/s]
     67%|██████▋   | 114M/170M [00:04<00:02, 19.8MB/s]
     68%|██████▊   | 116M/170M [00:04<00:02, 20.1MB/s]
     69%|██████▉   | 118M/170M [00:05<00:02, 20.5MB/s]
     71%|███████   | 121M/170M [00:05<00:02, 22.1MB/s]
     72%|███████▏  | 123M/170M [00:05<00:02, 22.3MB/s]
     74%|███████▎  | 126M/170M [00:05<00:02, 22.4MB/s]
     75%|███████▍  | 128M/170M [00:05<00:01, 21.5MB/s]
     76%|███████▌  | 130M/170M [00:05<00:01, 21.3MB/s]
     78%|███████▊  | 132M/170M [00:05<00:01, 21.4MB/s]
     79%|███████▉  | 134M/170M [00:05<00:01, 20.7MB/s]
     80%|████████  | 137M/170M [00:05<00:01, 21.7MB/s]
     82%|████████▏ | 139M/170M [00:05<00:01, 20.8MB/s]
     83%|████████▎ | 141M/170M [00:06<00:01, 21.9MB/s]
     84%|████████▍ | 144M/170M [00:06<00:01, 20.9MB/s]
     86%|████████▌ | 146M/170M [00:06<00:01, 21.9MB/s]
     87%|████████▋ | 148M/170M [00:06<00:01, 21.9MB/s]
     88%|████████▊ | 151M/170M [00:06<00:00, 22.0MB/s]
     90%|█████████ | 154M/170M [00:06<00:00, 23.9MB/s]
     92%|█████████▏| 156M/170M [00:06<00:00, 22.6MB/s]
     93%|█████████▎| 159M/170M [00:06<00:00, 23.7MB/s]
     94%|█████████▍| 161M/170M [00:06<00:00, 21.7MB/s]
     96%|█████████▌| 163M/170M [00:07<00:00, 21.8MB/s]
     97%|█████████▋| 166M/170M [00:07<00:00, 22.0MB/s]
     98%|█████████▊| 168M/170M [00:07<00:00, 22.2MB/s]
    100%|█████████▉| 170M/170M [00:07<00:00, 21.4MB/s]
    100%|██████████| 170M/170M [00:07<00:00, 23.0MB/s]

      0%|          | 0.00/2.92G [00:00<?, ?B/s]
      0%|          | 459k/2.92G [00:00<11:23, 4.27MB/s]
      0%|          | 1.54M/2.92G [00:00<06:22, 7.62MB/s]
      0%|          | 2.49M/2.92G [00:00<05:45, 8.44MB/s]
      0%|          | 3.60M/2.92G [00:00<05:15, 9.23MB/s]
      0%|          | 4.55M/2.92G [00:00<05:33, 8.73MB/s]
      0%|          | 5.47M/2.92G [00:00<05:37, 8.64MB/s]
      0%|          | 6.52M/2.92G [00:00<05:17, 9.16MB/s]
      0%|          | 7.57M/2.92G [00:00<05:05, 9.54MB/s]
      0%|          | 8.59M/2.92G [00:00<05:00, 9.68MB/s]
      0%|          | 9.63M/2.92G [00:01<04:54, 9.89MB/s]
      0%|          | 10.7M/2.92G [00:01<04:46, 10.1MB/s]
      0%|          | 11.8M/2.92G [00:01<04:54, 9.87MB/s]
      0%|          | 12.8M/2.92G [00:01<07:02, 6.87MB/s]
      0%|          | 13.7M/2.92G [00:01<06:30, 7.44MB/s]
      0%|          | 14.6M/2.92G [00:02<11:00, 4.39MB/s]
      1%|          | 15.5M/2.92G [00:02<09:19, 5.19MB/s]
      1%|          | 16.3M/2.92G [00:02<08:35, 5.63MB/s]
      1%|          | 17.4M/2.92G [00:02<07:21, 6.57MB/s]
      1%|          | 18.2M/2.92G [00:02<06:58, 6.93MB/s]
      1%|          | 19.1M/2.92G [00:02<06:36, 7.32MB/s]
      1%|          | 20.3M/2.92G [00:02<05:50, 8.26MB/s]
      1%|          | 21.3M/2.92G [00:02<05:25, 8.90MB/s]
      1%|          | 22.3M/2.92G [00:02<05:13, 9.23MB/s]
      1%|          | 23.4M/2.92G [00:02<05:04, 9.51MB/s]
      1%|          | 24.4M/2.92G [00:03<04:59, 9.67MB/s]
      1%|          | 25.4M/2.92G [00:03<09:42, 4.96MB/s]
      1%|          | 26.5M/2.92G [00:03<10:20, 4.66MB/s]
      1%|          | 27.4M/2.92G [00:03<09:03, 5.32MB/s]
      1%|          | 28.4M/2.92G [00:03<07:48, 6.17MB/s]
      1%|          | 29.4M/2.92G [00:04<06:55, 6.96MB/s]
      1%|          | 30.4M/2.92G [00:04<06:29, 7.42MB/s]
      1%|          | 31.5M/2.92G [00:04<05:52, 8.19MB/s]
      1%|          | 32.4M/2.92G [00:04<05:49, 8.25MB/s]
      1%|          | 33.3M/2.92G [00:04<05:40, 8.48MB/s]
      1%|          | 34.4M/2.92G [00:04<05:16, 9.10MB/s]
      1%|          | 35.5M/2.92G [00:04<05:04, 9.46MB/s]
      1%|▏         | 36.5M/2.92G [00:04<04:59, 9.63MB/s]
      1%|▏         | 37.6M/2.92G [00:04<04:46, 10.0MB/s]
      1%|▏         | 38.7M/2.92G [00:05<04:44, 10.1MB/s]
      1%|▏         | 39.7M/2.92G [00:05<07:10, 6.69MB/s]
      1%|▏         | 40.6M/2.92G [00:05<08:35, 5.59MB/s]
      1%|▏         | 41.6M/2.92G [00:05<07:20, 6.53MB/s]
      1%|▏         | 42.4M/2.92G [00:05<08:24, 5.70MB/s]
      1%|▏         | 43.2M/2.92G [00:05<07:54, 6.06MB/s]
      2%|▏         | 44.3M/2.92G [00:06<06:41, 7.16MB/s]
      2%|▏         | 45.4M/2.92G [00:06<06:02, 7.93MB/s]
      2%|▏         | 46.4M/2.92G [00:06<05:41, 8.40MB/s]
      2%|▏         | 47.4M/2.92G [00:06<05:29, 8.72MB/s]
      2%|▏         | 48.5M/2.92G [00:06<05:13, 9.16MB/s]
      2%|▏         | 49.5M/2.92G [00:06<05:03, 9.44MB/s]
      2%|▏         | 50.6M/2.92G [00:06<05:04, 9.43MB/s]
      2%|▏         | 51.5M/2.92G [00:06<05:01, 9.52MB/s]
      2%|▏         | 52.6M/2.92G [00:06<04:59, 9.58MB/s]
      2%|▏         | 53.6M/2.92G [00:06<04:52, 9.80MB/s]
      2%|▏         | 54.6M/2.92G [00:07<04:50, 9.87MB/s]
      2%|▏         | 55.6M/2.92G [00:07<07:17, 6.54MB/s]
      2%|▏         | 56.5M/2.92G [00:07<11:25, 4.17MB/s]
      2%|▏         | 65.8M/2.92G [00:07<02:36, 18.2MB/s]
      3%|▎         | 77.6M/2.92G [00:07<01:17, 36.4MB/s]
      3%|▎         | 89.4M/2.92G [00:08<00:53, 53.3MB/s]
      3%|▎         | 101M/2.92G [00:08<00:41, 68.0MB/s] 
      4%|▍         | 113M/2.92G [00:08<00:35, 79.2MB/s]
      4%|▍         | 124M/2.92G [00:08<00:31, 89.2MB/s]
      5%|▍         | 135M/2.92G [00:08<00:29, 92.8MB/s]
      5%|▍         | 145M/2.92G [00:08<00:28, 95.9MB/s]
      5%|▌         | 155M/2.92G [00:08<00:30, 90.1MB/s]
      6%|▌         | 165M/2.92G [00:09<00:51, 53.5MB/s]
      6%|▌         | 172M/2.92G [00:09<01:02, 44.1MB/s]
      6%|▌         | 179M/2.92G [00:09<01:09, 39.5MB/s]
      6%|▋         | 184M/2.92G [00:09<01:14, 36.5MB/s]
      6%|▋         | 188M/2.92G [00:09<01:18, 34.8MB/s]
      7%|▋         | 192M/2.92G [00:09<01:16, 35.7MB/s]
      7%|▋         | 196M/2.92G [00:10<01:14, 36.7MB/s]
      7%|▋         | 201M/2.92G [00:10<01:10, 38.5MB/s]
      7%|▋         | 205M/2.92G [00:10<01:11, 37.8MB/s]
      7%|▋         | 209M/2.92G [00:10<01:10, 38.2MB/s]
      7%|▋         | 213M/2.92G [00:10<01:08, 39.3MB/s]
      7%|▋         | 217M/2.92G [00:10<01:13, 36.8MB/s]
      8%|▊         | 221M/2.92G [00:10<01:12, 37.3MB/s]
      8%|▊         | 225M/2.92G [00:10<01:35, 28.1MB/s]
      8%|▊         | 228M/2.92G [00:11<01:37, 27.6MB/s]
      8%|▊         | 231M/2.92G [00:11<01:34, 28.5MB/s]
      8%|▊         | 235M/2.92G [00:11<01:30, 29.6MB/s]
      8%|▊         | 238M/2.92G [00:11<01:30, 29.5MB/s]
      8%|▊         | 241M/2.92G [00:11<01:29, 29.8MB/s]
      8%|▊         | 244M/2.92G [00:11<01:33, 28.7MB/s]
      8%|▊         | 247M/2.92G [00:11<01:30, 29.4MB/s]
      9%|▊         | 250M/2.92G [00:11<01:28, 30.3MB/s]
      9%|▉         | 256M/2.92G [00:11<01:09, 38.4MB/s]
      9%|▉         | 268M/2.92G [00:11<00:43, 61.4MB/s]
     10%|▉         | 280M/2.92G [00:12<00:33, 77.8MB/s]
     10%|▉         | 291M/2.92G [00:12<00:29, 88.2MB/s]
     10%|█         | 302M/2.92G [00:12<00:27, 94.5MB/s]
     11%|█         | 312M/2.92G [00:12<00:27, 93.5MB/s]
     11%|█         | 321M/2.92G [00:12<00:28, 90.5MB/s]
     11%|█▏        | 330M/2.92G [00:12<00:29, 88.8MB/s]
     12%|█▏        | 339M/2.92G [00:12<00:29, 87.3MB/s]
     12%|█▏        | 348M/2.92G [00:12<00:32, 78.6MB/s]
     12%|█▏        | 356M/2.92G [00:13<00:52, 48.9MB/s]
     12%|█▏        | 362M/2.92G [00:13<00:58, 43.4MB/s]
     13%|█▎        | 368M/2.92G [00:13<01:06, 38.6MB/s]
     13%|█▎        | 372M/2.92G [00:13<01:13, 34.8MB/s]
     13%|█▎        | 376M/2.92G [00:13<01:16, 33.3MB/s]
     13%|█▎        | 381M/2.92G [00:13<01:12, 34.8MB/s]
     13%|█▎        | 389M/2.92G [00:14<00:55, 45.4MB/s]
     14%|█▎        | 399M/2.92G [00:14<00:43, 57.8MB/s]
     14%|█▍        | 406M/2.92G [00:14<00:40, 61.6MB/s]
     14%|█▍        | 413M/2.92G [00:14<00:41, 59.9MB/s]
     14%|█▍        | 419M/2.92G [00:14<00:40, 61.5MB/s]
     15%|█▍        | 426M/2.92G [00:14<00:39, 63.5MB/s]
     15%|█▍        | 433M/2.92G [00:14<00:38, 65.3MB/s]
     15%|█▌        | 440M/2.92G [00:14<00:37, 66.8MB/s]
     15%|█▌        | 447M/2.92G [00:14<00:38, 64.0MB/s]
     16%|█▌        | 453M/2.92G [00:15<00:59, 41.4MB/s]
     16%|█▌        | 459M/2.92G [00:15<01:09, 35.4MB/s]
     16%|█▌        | 463M/2.92G [00:15<01:10, 34.8MB/s]
     16%|█▌        | 467M/2.92G [00:15<01:14, 33.1MB/s]
     16%|█▌        | 471M/2.92G [00:15<01:12, 33.6MB/s]
     16%|█▋        | 475M/2.92G [00:15<01:16, 32.1MB/s]
     16%|█▋        | 478M/2.92G [00:16<01:25, 28.7MB/s]
     17%|█▋        | 485M/2.92G [00:16<01:04, 37.7MB/s]
     17%|█▋        | 493M/2.92G [00:16<00:51, 47.5MB/s]
     17%|█▋        | 500M/2.92G [00:16<00:43, 55.3MB/s]
     17%|█▋        | 508M/2.92G [00:16<00:39, 61.4MB/s]
     18%|█▊        | 516M/2.92G [00:16<00:36, 65.9MB/s]
     18%|█▊        | 524M/2.92G [00:16<00:34, 69.2MB/s]
     18%|█▊        | 532M/2.92G [00:16<00:33, 71.9MB/s]
     18%|█▊        | 539M/2.92G [00:16<00:34, 68.9MB/s]
     19%|█▊        | 546M/2.92G [00:17<01:06, 35.9MB/s]
     19%|█▉        | 551M/2.92G [00:17<01:11, 33.3MB/s]
     19%|█▉        | 556M/2.92G [00:17<01:40, 23.5MB/s]
     19%|█▉        | 560M/2.92G [00:18<01:36, 24.5MB/s]
     19%|█▉        | 563M/2.92G [00:18<01:32, 25.5MB/s]
     19%|█▉        | 566M/2.92G [00:18<01:28, 26.5MB/s]
     20%|█▉        | 570M/2.92G [00:18<01:24, 27.7MB/s]
     20%|█▉        | 573M/2.92G [00:18<01:21, 28.8MB/s]
     20%|█▉        | 576M/2.92G [00:18<01:18, 29.8MB/s]
     20%|█▉        | 580M/2.92G [00:18<01:16, 30.7MB/s]
     20%|█▉        | 583M/2.92G [00:18<01:14, 31.5MB/s]
     20%|██        | 586M/2.92G [00:18<01:11, 32.7MB/s]
     20%|██        | 590M/2.92G [00:19<01:18, 29.8MB/s]
     20%|██        | 593M/2.92G [00:19<01:15, 30.7MB/s]
     20%|██        | 596M/2.92G [00:19<01:16, 30.4MB/s]
     21%|██        | 600M/2.92G [00:19<01:13, 31.8MB/s]
     21%|██        | 604M/2.92G [00:19<01:09, 33.2MB/s]
     21%|██        | 607M/2.92G [00:19<01:11, 32.3MB/s]
     21%|██        | 610M/2.92G [00:19<01:11, 32.2MB/s]
     21%|██        | 614M/2.92G [00:19<01:15, 30.5MB/s]
     21%|██        | 617M/2.92G [00:19<01:16, 30.0MB/s]
     21%|██▏       | 620M/2.92G [00:19<01:12, 31.5MB/s]
     21%|██▏       | 624M/2.92G [00:20<01:07, 33.8MB/s]
     22%|██▏       | 628M/2.92G [00:20<01:04, 35.6MB/s]
     22%|██▏       | 632M/2.92G [00:20<01:02, 36.8MB/s]
     22%|██▏       | 636M/2.92G [00:20<01:00, 37.7MB/s]
     22%|██▏       | 641M/2.92G [00:20<00:57, 39.5MB/s]
     22%|██▏       | 645M/2.92G [00:20<00:57, 39.3MB/s]
     22%|██▏       | 649M/2.92G [00:20<01:00, 37.8MB/s]
     22%|██▏       | 653M/2.92G [00:20<00:56, 39.9MB/s]
     23%|██▎       | 657M/2.92G [00:21<01:47, 21.0MB/s]
     23%|██▎       | 661M/2.92G [00:21<01:36, 23.4MB/s]
     23%|██▎       | 664M/2.92G [00:21<01:28, 25.4MB/s]
     23%|██▎       | 667M/2.92G [00:21<01:28, 25.4MB/s]
     23%|██▎       | 670M/2.92G [00:21<01:22, 27.1MB/s]
     23%|██▎       | 674M/2.92G [00:21<01:24, 26.5MB/s]
     23%|██▎       | 677M/2.92G [00:21<01:20, 27.8MB/s]
     23%|██▎       | 680M/2.92G [00:21<01:14, 29.8MB/s]
     23%|██▎       | 685M/2.92G [00:22<01:05, 34.2MB/s]
     24%|██▎       | 690M/2.92G [00:22<00:58, 37.8MB/s]
     24%|██▍       | 694M/2.92G [00:22<00:55, 40.3MB/s]
     24%|██▍       | 698M/2.92G [00:22<00:59, 37.5MB/s]
     24%|██▍       | 702M/2.92G [00:22<00:59, 37.1MB/s]
     24%|██▍       | 707M/2.92G [00:22<00:57, 38.7MB/s]
     24%|██▍       | 711M/2.92G [00:22<00:55, 39.8MB/s]
     25%|██▍       | 716M/2.92G [00:22<00:50, 43.3MB/s]
     25%|██▍       | 721M/2.92G [00:23<01:09, 31.4MB/s]
     25%|██▍       | 724M/2.92G [00:23<01:10, 31.1MB/s]
     25%|██▍       | 728M/2.92G [00:23<01:12, 30.1MB/s]
     25%|██▌       | 731M/2.92G [00:23<01:11, 30.6MB/s]
     25%|██▌       | 734M/2.92G [00:23<01:12, 30.3MB/s]
     25%|██▌       | 737M/2.92G [00:23<01:11, 30.5MB/s]
     25%|██▌       | 740M/2.92G [00:23<01:12, 30.1MB/s]
     25%|██▌       | 744M/2.92G [00:23<01:09, 31.1MB/s]
     26%|██▌       | 749M/2.92G [00:23<01:01, 35.5MB/s]
     26%|██▌       | 754M/2.92G [00:24<00:54, 39.5MB/s]
     26%|██▌       | 759M/2.92G [00:24<00:50, 43.1MB/s]
     26%|██▌       | 764M/2.92G [00:24<00:47, 45.4MB/s]
     26%|██▋       | 769M/2.92G [00:24<00:45, 46.8MB/s]
     27%|██▋       | 774M/2.92G [00:24<00:45, 47.6MB/s]
     27%|██▋       | 779M/2.92G [00:24<00:44, 48.3MB/s]
     27%|██▋       | 784M/2.92G [00:24<00:47, 44.6MB/s]
     27%|██▋       | 789M/2.92G [00:24<00:45, 46.8MB/s]
     27%|██▋       | 795M/2.92G [00:24<00:58, 36.3MB/s]
     27%|██▋       | 799M/2.92G [00:25<01:08, 30.7MB/s]
     27%|██▋       | 802M/2.92G [00:25<01:08, 30.9MB/s]
     28%|██▊       | 806M/2.92G [00:25<01:09, 30.6MB/s]
     28%|██▊       | 809M/2.92G [00:25<01:11, 29.4MB/s]
     28%|██▊       | 812M/2.92G [00:25<01:08, 30.5MB/s]
     28%|██▊       | 815M/2.92G [00:25<01:11, 29.5MB/s]
     28%|██▊       | 819M/2.92G [00:25<01:11, 29.2MB/s]
     28%|██▊       | 823M/2.92G [00:25<01:04, 32.5MB/s]
     28%|██▊       | 829M/2.92G [00:26<00:52, 40.1MB/s]
     29%|██▊       | 835M/2.92G [00:26<00:45, 45.8MB/s]
     29%|██▉       | 841M/2.92G [00:26<00:41, 49.9MB/s]
     29%|██▉       | 847M/2.92G [00:26<00:39, 52.4MB/s]
     29%|██▉       | 853M/2.92G [00:26<00:38, 54.2MB/s]
     29%|██▉       | 858M/2.92G [00:26<00:44, 46.1MB/s]
     30%|██▉       | 864M/2.92G [00:26<00:41, 48.9MB/s]
     30%|██▉       | 869M/2.92G [00:26<00:40, 51.0MB/s]
     30%|██▉       | 875M/2.92G [00:27<01:23, 24.4MB/s]
     30%|███       | 879M/2.92G [00:27<01:18, 25.9MB/s]
     30%|███       | 883M/2.92G [00:27<01:14, 27.3MB/s]
     30%|███       | 886M/2.92G [00:27<01:12, 28.0MB/s]
     30%|███       | 890M/2.92G [00:27<01:13, 27.7MB/s]
     31%|███       | 893M/2.92G [00:27<01:18, 25.9MB/s]
     31%|███       | 898M/2.92G [00:28<01:04, 31.2MB/s]
     31%|███       | 903M/2.92G [00:28<00:55, 36.1MB/s]
     31%|███       | 907M/2.92G [00:28<00:55, 36.2MB/s]
     31%|███       | 912M/2.92G [00:28<00:50, 39.5MB/s]
     31%|███▏      | 917M/2.92G [00:28<00:46, 42.9MB/s]
     32%|███▏      | 922M/2.92G [00:28<00:44, 44.5MB/s]
     32%|███▏      | 927M/2.92G [00:28<00:41, 47.5MB/s]
     32%|███▏      | 933M/2.92G [00:28<00:39, 49.6MB/s]
     32%|███▏      | 938M/2.92G [00:28<00:39, 50.4MB/s]
     32%|███▏      | 943M/2.92G [00:29<01:28, 22.4MB/s]
     32%|███▏      | 947M/2.92G [00:29<01:20, 24.3MB/s]
     33%|███▎      | 951M/2.92G [00:29<01:15, 26.0MB/s]
     33%|███▎      | 954M/2.92G [00:29<01:10, 27.8MB/s]
     33%|███▎      | 958M/2.92G [00:29<01:08, 28.6MB/s]
     33%|███▎      | 962M/2.92G [00:29<01:04, 30.3MB/s]
     33%|███▎      | 967M/2.92G [00:30<00:53, 36.2MB/s]
     33%|███▎      | 973M/2.92G [00:30<00:46, 41.8MB/s]
     34%|███▎      | 979M/2.92G [00:30<00:41, 46.4MB/s]
     34%|███▎      | 985M/2.92G [00:30<00:39, 49.5MB/s]
     34%|███▍      | 990M/2.92G [00:30<00:38, 50.5MB/s]
     34%|███▍      | 995M/2.92G [00:30<00:37, 51.0MB/s]
     34%|███▍      | 1.00G/2.92G [00:30<00:37, 51.7MB/s]
     34%|███▍      | 1.01G/2.92G [00:30<00:35, 53.3MB/s]
     35%|███▍      | 1.01G/2.92G [00:30<00:34, 54.6MB/s]
     35%|███▍      | 1.02G/2.92G [00:30<00:33, 56.6MB/s]
     35%|███▌      | 1.02G/2.92G [00:31<00:54, 34.6MB/s]
     35%|███▌      | 1.03G/2.92G [00:31<00:57, 33.0MB/s]
     35%|███▌      | 1.03G/2.92G [00:31<01:01, 30.7MB/s]
     36%|███▌      | 1.04G/2.92G [00:31<01:01, 30.7MB/s]
     36%|███▌      | 1.04G/2.92G [00:31<01:01, 30.6MB/s]
     36%|███▌      | 1.04G/2.92G [00:31<01:00, 30.8MB/s]
     36%|███▌      | 1.05G/2.92G [00:32<01:03, 29.3MB/s]
     36%|███▌      | 1.05G/2.92G [00:32<00:57, 32.5MB/s]
     36%|███▌      | 1.06G/2.92G [00:32<00:49, 37.4MB/s]
     36%|███▋      | 1.06G/2.92G [00:32<00:46, 39.5MB/s]
     36%|███▋      | 1.06G/2.92G [00:32<00:45, 41.2MB/s]
     37%|███▋      | 1.07G/2.92G [00:32<00:50, 36.5MB/s]
     37%|███▋      | 1.07G/2.92G [00:32<00:52, 35.2MB/s]
     37%|███▋      | 1.08G/2.92G [00:32<00:53, 34.6MB/s]
     37%|███▋      | 1.08G/2.92G [00:32<00:53, 34.1MB/s]
     37%|███▋      | 1.08G/2.92G [00:33<01:35, 19.3MB/s]
     37%|███▋      | 1.09G/2.92G [00:33<01:28, 20.8MB/s]
     37%|███▋      | 1.09G/2.92G [00:33<01:21, 22.5MB/s]
     37%|███▋      | 1.09G/2.92G [00:33<01:15, 24.1MB/s]
     38%|███▊      | 1.10G/2.92G [00:33<01:10, 25.7MB/s]
     38%|███▊      | 1.10G/2.92G [00:33<01:11, 25.4MB/s]
     38%|███▊      | 1.10G/2.92G [00:33<01:12, 25.1MB/s]
     38%|███▊      | 1.10G/2.92G [00:34<01:06, 27.1MB/s]
     38%|███▊      | 1.11G/2.92G [00:34<01:00, 29.7MB/s]
     38%|███▊      | 1.11G/2.92G [00:34<00:55, 32.4MB/s]
     38%|███▊      | 1.12G/2.92G [00:34<00:52, 34.3MB/s]
     38%|███▊      | 1.12G/2.92G [00:34<00:50, 35.9MB/s]
     39%|███▊      | 1.12G/2.92G [00:34<00:48, 36.7MB/s]
     39%|███▊      | 1.13G/2.92G [00:34<00:47, 37.6MB/s]
     39%|███▉      | 1.13G/2.92G [00:34<00:47, 37.8MB/s]
     39%|███▉      | 1.14G/2.92G [00:34<00:45, 39.0MB/s]
     39%|███▉      | 1.14G/2.92G [00:34<00:43, 40.5MB/s]
     39%|███▉      | 1.14G/2.92G [00:35<01:36, 18.5MB/s]
     39%|███▉      | 1.15G/2.92G [00:35<01:25, 20.6MB/s]
     39%|███▉      | 1.15G/2.92G [00:35<01:18, 22.4MB/s]
     40%|███▉      | 1.15G/2.92G [00:35<01:11, 24.8MB/s]
     40%|███▉      | 1.16G/2.92G [00:35<01:10, 24.8MB/s]
     40%|███▉      | 1.16G/2.92G [00:35<01:05, 26.9MB/s]
     40%|███▉      | 1.16G/2.92G [00:36<01:02, 27.9MB/s]
     40%|████      | 1.17G/2.92G [00:36<00:55, 31.6MB/s]
     40%|████      | 1.17G/2.92G [00:36<00:49, 35.4MB/s]
     40%|████      | 1.18G/2.92G [00:36<00:45, 37.9MB/s]
     40%|████      | 1.18G/2.92G [00:36<00:43, 40.2MB/s]
     41%|████      | 1.19G/2.92G [00:36<00:41, 41.9MB/s]
     41%|████      | 1.19G/2.92G [00:36<00:40, 42.7MB/s]
     41%|████      | 1.20G/2.92G [00:36<00:39, 43.3MB/s]
     41%|████      | 1.20G/2.92G [00:36<00:41, 41.3MB/s]
     41%|████▏     | 1.20G/2.92G [00:37<00:39, 43.7MB/s]
     41%|████▏     | 1.21G/2.92G [00:37<01:16, 22.4MB/s]
     42%|████▏     | 1.21G/2.92G [00:37<01:14, 23.0MB/s]
     42%|████▏     | 1.22G/2.92G [00:37<01:09, 24.4MB/s]
     42%|████▏     | 1.22G/2.92G [00:37<01:04, 26.5MB/s]
     42%|████▏     | 1.22G/2.92G [00:37<00:59, 28.3MB/s]
     42%|████▏     | 1.23G/2.92G [00:37<00:58, 29.1MB/s]
     42%|████▏     | 1.23G/2.92G [00:38<00:57, 29.3MB/s]
     42%|████▏     | 1.23G/2.92G [00:38<01:02, 27.0MB/s]
     42%|████▏     | 1.24G/2.92G [00:38<01:03, 26.3MB/s]
     42%|████▏     | 1.24G/2.92G [00:38<00:51, 32.7MB/s]
     43%|████▎     | 1.24G/2.92G [00:38<00:45, 36.6MB/s]
     43%|████▎     | 1.25G/2.92G [00:38<00:43, 38.4MB/s]
     43%|████▎     | 1.25G/2.92G [00:38<00:40, 41.0MB/s]
     43%|████▎     | 1.26G/2.92G [00:38<00:40, 41.4MB/s]
     43%|████▎     | 1.26G/2.92G [00:38<00:38, 43.4MB/s]
     43%|████▎     | 1.27G/2.92G [00:39<00:35, 46.6MB/s]
     44%|████▎     | 1.27G/2.92G [00:39<00:54, 30.1MB/s]
     44%|████▍     | 1.28G/2.92G [00:39<00:55, 29.3MB/s]
     44%|████▍     | 1.28G/2.92G [00:39<00:57, 28.6MB/s]
     44%|████▍     | 1.28G/2.92G [00:39<01:00, 27.1MB/s]
     44%|████▍     | 1.29G/2.92G [00:39<01:02, 26.1MB/s]
     44%|████▍     | 1.29G/2.92G [00:40<01:04, 25.3MB/s]
     44%|████▍     | 1.29G/2.92G [00:40<01:03, 25.6MB/s]
     44%|████▍     | 1.30G/2.92G [00:40<01:05, 24.9MB/s]
     45%|████▍     | 1.30G/2.92G [00:40<00:51, 31.7MB/s]
     45%|████▍     | 1.31G/2.92G [00:40<00:43, 36.7MB/s]
     45%|████▍     | 1.31G/2.92G [00:40<00:38, 41.7MB/s]
     45%|████▌     | 1.32G/2.92G [00:40<00:35, 45.4MB/s]
     45%|████▌     | 1.32G/2.92G [00:40<00:37, 42.3MB/s]
     45%|████▌     | 1.33G/2.92G [00:40<00:35, 44.6MB/s]
     46%|████▌     | 1.33G/2.92G [00:40<00:34, 46.3MB/s]
     46%|████▌     | 1.34G/2.92G [00:41<01:07, 23.4MB/s]
     46%|████▌     | 1.34G/2.92G [00:41<01:08, 23.2MB/s]
     46%|████▌     | 1.34G/2.92G [00:41<01:07, 23.2MB/s]
     46%|████▌     | 1.35G/2.92G [00:41<01:04, 24.4MB/s]
     46%|████▋     | 1.35G/2.92G [00:41<01:03, 24.7MB/s]
     46%|████▋     | 1.35G/2.92G [00:42<00:57, 27.5MB/s]
     47%|████▋     | 1.36G/2.92G [00:42<00:44, 35.2MB/s]
     47%|████▋     | 1.37G/2.92G [00:42<00:37, 41.6MB/s]
     47%|████▋     | 1.37G/2.92G [00:42<00:33, 46.1MB/s]
     47%|████▋     | 1.38G/2.92G [00:42<00:30, 51.1MB/s]
     47%|████▋     | 1.38G/2.92G [00:42<00:28, 54.8MB/s]
     48%|████▊     | 1.39G/2.92G [00:42<00:28, 53.7MB/s]
     48%|████▊     | 1.40G/2.92G [00:42<00:29, 51.5MB/s]
     48%|████▊     | 1.40G/2.92G [00:43<00:40, 37.7MB/s]
     48%|████▊     | 1.40G/2.92G [00:43<00:44, 34.1MB/s]
     48%|████▊     | 1.41G/2.92G [00:43<00:45, 33.0MB/s]
     48%|████▊     | 1.41G/2.92G [00:43<00:49, 30.4MB/s]
     49%|████▊     | 1.42G/2.92G [00:43<00:49, 30.4MB/s]
     49%|████▊     | 1.42G/2.92G [00:43<00:49, 30.3MB/s]
     49%|████▊     | 1.42G/2.92G [00:43<00:57, 26.2MB/s]
     49%|████▉     | 1.43G/2.92G [00:43<00:46, 31.8MB/s]
     49%|████▉     | 1.43G/2.92G [00:44<00:39, 37.6MB/s]
     49%|████▉     | 1.44G/2.92G [00:44<00:35, 41.3MB/s]
     49%|████▉     | 1.44G/2.92G [00:44<00:36, 40.9MB/s]
     50%|████▉     | 1.45G/2.92G [00:44<00:37, 38.9MB/s]
     50%|████▉     | 1.45G/2.92G [00:44<00:37, 39.1MB/s]
     50%|████▉     | 1.45G/2.92G [00:44<00:38, 38.0MB/s]
     50%|████▉     | 1.46G/2.92G [00:44<00:37, 38.7MB/s]
     50%|█████     | 1.46G/2.92G [00:44<00:37, 38.7MB/s]
     50%|█████     | 1.47G/2.92G [00:45<00:51, 28.5MB/s]
     50%|█████     | 1.47G/2.92G [00:45<00:52, 27.7MB/s]
     50%|█████     | 1.47G/2.92G [00:45<00:49, 29.3MB/s]
     51%|█████     | 1.48G/2.92G [00:45<00:48, 29.6MB/s]
     51%|█████     | 1.48G/2.92G [00:45<00:48, 30.0MB/s]
     51%|█████     | 1.48G/2.92G [00:45<00:53, 26.8MB/s]
     51%|█████     | 1.49G/2.92G [00:45<00:51, 27.6MB/s]
     51%|█████     | 1.49G/2.92G [00:45<00:45, 31.6MB/s]
     51%|█████     | 1.49G/2.92G [00:45<00:40, 34.9MB/s]
     51%|█████▏    | 1.50G/2.92G [00:46<00:37, 37.8MB/s]
     52%|█████▏    | 1.50G/2.92G [00:46<00:35, 40.1MB/s]
     52%|█████▏    | 1.51G/2.92G [00:46<00:34, 40.9MB/s]
     52%|█████▏    | 1.51G/2.92G [00:46<00:39, 35.3MB/s]
     52%|█████▏    | 1.52G/2.92G [00:46<00:37, 37.5MB/s]
     52%|█████▏    | 1.52G/2.92G [00:46<00:35, 39.3MB/s]
     52%|█████▏    | 1.53G/2.92G [00:46<00:49, 27.9MB/s]
     52%|█████▏    | 1.53G/2.92G [00:46<00:48, 28.7MB/s]
     52%|█████▏    | 1.53G/2.92G [00:47<00:48, 28.7MB/s]
     53%|█████▎    | 1.53G/2.92G [00:47<00:49, 27.7MB/s]
     53%|█████▎    | 1.54G/2.92G [00:47<00:49, 28.0MB/s]
     53%|█████▎    | 1.54G/2.92G [00:47<00:53, 26.0MB/s]
     53%|█████▎    | 1.54G/2.92G [00:47<00:54, 25.1MB/s]
     53%|█████▎    | 1.55G/2.92G [00:47<00:52, 26.0MB/s]
     53%|█████▎    | 1.55G/2.92G [00:47<00:53, 25.7MB/s]
     53%|█████▎    | 1.55G/2.92G [00:47<00:46, 29.7MB/s]
     53%|█████▎    | 1.56G/2.92G [00:47<00:38, 35.4MB/s]
     54%|█████▎    | 1.56G/2.92G [00:48<00:34, 39.7MB/s]
     54%|█████▎    | 1.57G/2.92G [00:48<00:32, 42.1MB/s]
     54%|█████▍    | 1.57G/2.92G [00:48<00:32, 40.9MB/s]
     54%|█████▍    | 1.58G/2.92G [00:48<00:30, 43.6MB/s]
     54%|█████▍    | 1.58G/2.92G [00:48<00:31, 43.1MB/s]
     54%|█████▍    | 1.59G/2.92G [00:48<00:33, 39.4MB/s]
     54%|█████▍    | 1.59G/2.92G [00:48<00:46, 28.5MB/s]
     55%|█████▍    | 1.59G/2.92G [00:48<00:49, 27.0MB/s]
     55%|█████▍    | 1.60G/2.92G [00:49<00:49, 26.5MB/s]
     55%|█████▍    | 1.60G/2.92G [00:49<00:50, 26.2MB/s]
     55%|█████▍    | 1.60G/2.92G [00:49<00:50, 26.2MB/s]
     55%|█████▌    | 1.61G/2.92G [00:49<00:50, 26.2MB/s]
     55%|█████▌    | 1.61G/2.92G [00:49<00:48, 26.9MB/s]
     55%|█████▌    | 1.61G/2.92G [00:49<00:41, 31.4MB/s]
     55%|█████▌    | 1.62G/2.92G [00:49<00:37, 34.7MB/s]
     56%|█████▌    | 1.62G/2.92G [00:49<00:34, 37.9MB/s]
     56%|█████▌    | 1.63G/2.92G [00:49<00:32, 39.9MB/s]
     56%|█████▌    | 1.63G/2.92G [00:50<00:30, 41.8MB/s]
     56%|█████▌    | 1.64G/2.92G [00:50<00:30, 42.4MB/s]
     56%|█████▌    | 1.64G/2.92G [00:50<00:29, 43.1MB/s]
     56%|█████▋    | 1.64G/2.92G [00:50<00:29, 43.8MB/s]
     57%|█████▋    | 1.65G/2.92G [00:50<00:28, 44.7MB/s]
     57%|█████▋    | 1.65G/2.92G [00:50<00:27, 45.6MB/s]
     57%|█████▋    | 1.66G/2.92G [00:50<00:34, 36.2MB/s]
     57%|█████▋    | 1.66G/2.92G [00:50<00:38, 32.9MB/s]
     57%|█████▋    | 1.67G/2.92G [00:51<00:39, 31.8MB/s]
     57%|█████▋    | 1.67G/2.92G [00:51<00:40, 30.9MB/s]
     57%|█████▋    | 1.67G/2.92G [00:51<00:46, 27.1MB/s]
     57%|█████▋    | 1.68G/2.92G [00:51<00:49, 25.0MB/s]
     58%|█████▊    | 1.68G/2.92G [00:51<00:49, 25.0MB/s]
     58%|█████▊    | 1.68G/2.92G [00:51<00:50, 24.7MB/s]
     58%|█████▊    | 1.69G/2.92G [00:51<00:38, 31.8MB/s]
     58%|█████▊    | 1.69G/2.92G [00:51<00:32, 37.4MB/s]
     58%|█████▊    | 1.70G/2.92G [00:51<00:29, 41.4MB/s]
     58%|█████▊    | 1.70G/2.92G [00:52<00:27, 44.2MB/s]
     58%|█████▊    | 1.71G/2.92G [00:52<00:26, 45.9MB/s]
     59%|█████▊    | 1.71G/2.92G [00:52<00:25, 47.3MB/s]
     59%|█████▉    | 1.72G/2.92G [00:52<00:24, 48.7MB/s]
     59%|█████▉    | 1.72G/2.92G [00:52<00:24, 49.7MB/s]
     59%|█████▉    | 1.73G/2.92G [00:52<00:24, 48.3MB/s]
     59%|█████▉    | 1.73G/2.92G [00:53<00:51, 22.9MB/s]
     59%|█████▉    | 1.74G/2.92G [00:53<00:50, 23.6MB/s]
     60%|█████▉    | 1.74G/2.92G [00:53<00:47, 25.0MB/s]
     60%|█████▉    | 1.74G/2.92G [00:53<00:46, 25.3MB/s]
     60%|█████▉    | 1.75G/2.92G [00:53<00:43, 26.7MB/s]
     60%|█████▉    | 1.75G/2.92G [00:53<00:38, 30.2MB/s]
     60%|██████    | 1.75G/2.92G [00:53<00:34, 33.6MB/s]
     60%|██████    | 1.76G/2.92G [00:53<00:32, 36.0MB/s]
     60%|██████    | 1.76G/2.92G [00:53<00:31, 36.3MB/s]
     61%|██████    | 1.77G/2.92G [00:54<00:33, 34.4MB/s]
     61%|██████    | 1.77G/2.92G [00:54<00:34, 33.5MB/s]
     61%|██████    | 1.77G/2.92G [00:54<00:33, 34.5MB/s]
     61%|██████    | 1.78G/2.92G [00:54<00:31, 35.9MB/s]
     61%|██████    | 1.78G/2.92G [00:54<00:30, 37.9MB/s]
     61%|██████    | 1.79G/2.92G [00:54<00:57, 19.6MB/s]
     61%|██████▏   | 1.79G/2.92G [00:55<00:51, 21.9MB/s]
     61%|██████▏   | 1.79G/2.92G [00:55<00:47, 23.5MB/s]
     62%|██████▏   | 1.80G/2.92G [00:55<00:46, 24.3MB/s]
     62%|██████▏   | 1.80G/2.92G [00:55<00:44, 25.1MB/s]
     62%|██████▏   | 1.80G/2.92G [00:55<00:40, 27.3MB/s]
     62%|██████▏   | 1.81G/2.92G [00:55<00:32, 34.5MB/s]
     62%|██████▏   | 1.81G/2.92G [00:55<00:28, 39.4MB/s]
     62%|██████▏   | 1.82G/2.92G [00:55<00:25, 43.0MB/s]
     62%|██████▏   | 1.82G/2.92G [00:55<00:23, 45.7MB/s]
     63%|██████▎   | 1.83G/2.92G [00:55<00:22, 48.4MB/s]
     63%|██████▎   | 1.83G/2.92G [00:56<00:21, 51.3MB/s]
     63%|██████▎   | 1.84G/2.92G [00:56<00:20, 52.9MB/s]
     63%|██████▎   | 1.85G/2.92G [00:56<00:20, 52.5MB/s]
     63%|██████▎   | 1.85G/2.92G [00:56<00:20, 52.3MB/s]
     64%|██████▎   | 1.86G/2.92G [00:56<00:47, 22.3MB/s]
     64%|██████▎   | 1.86G/2.92G [00:57<00:45, 23.3MB/s]
     64%|██████▍   | 1.86G/2.92G [00:57<00:42, 24.6MB/s]
     64%|██████▍   | 1.87G/2.92G [00:57<00:42, 24.9MB/s]
     64%|██████▍   | 1.87G/2.92G [00:57<00:37, 28.1MB/s]
     64%|██████▍   | 1.88G/2.92G [00:57<00:32, 32.1MB/s]
     64%|██████▍   | 1.88G/2.92G [00:57<00:29, 35.5MB/s]
     65%|██████▍   | 1.89G/2.92G [00:57<00:27, 37.7MB/s]
     65%|██████▍   | 1.89G/2.92G [00:57<00:25, 40.2MB/s]
     65%|██████▍   | 1.89G/2.92G [00:57<00:24, 41.3MB/s]
     65%|██████▌   | 1.90G/2.92G [00:58<00:23, 43.4MB/s]
     65%|██████▌   | 1.90G/2.92G [00:58<00:22, 44.9MB/s]
     65%|██████▌   | 1.91G/2.92G [00:58<00:21, 46.8MB/s]
     66%|██████▌   | 1.91G/2.92G [00:58<00:43, 23.0MB/s]
     66%|██████▌   | 1.92G/2.92G [00:58<00:42, 23.8MB/s]
     66%|██████▌   | 1.92G/2.92G [00:58<00:41, 23.9MB/s]
     66%|██████▌   | 1.92G/2.92G [00:59<00:40, 24.7MB/s]
     66%|██████▌   | 1.93G/2.92G [00:59<00:40, 24.4MB/s]
     66%|██████▌   | 1.93G/2.92G [00:59<00:37, 26.3MB/s]
     66%|██████▋   | 1.93G/2.92G [00:59<00:33, 29.4MB/s]
     66%|██████▋   | 1.94G/2.92G [00:59<00:30, 31.8MB/s]
     67%|██████▋   | 1.94G/2.92G [00:59<00:28, 33.9MB/s]
     67%|██████▋   | 1.95G/2.92G [00:59<00:27, 35.1MB/s]
     67%|██████▋   | 1.95G/2.92G [00:59<00:26, 36.5MB/s]
     67%|██████▋   | 1.95G/2.92G [00:59<00:27, 35.3MB/s]
     67%|██████▋   | 1.96G/2.92G [01:00<00:28, 33.9MB/s]
     67%|██████▋   | 1.96G/2.92G [01:00<00:26, 35.8MB/s]
     67%|██████▋   | 1.97G/2.92G [01:00<00:25, 37.2MB/s]
     68%|██████▊   | 1.97G/2.92G [01:00<00:24, 38.7MB/s]
     68%|██████▊   | 1.97G/2.92G [01:00<00:32, 29.3MB/s]
     68%|██████▊   | 1.98G/2.92G [01:00<00:32, 29.2MB/s]
     68%|██████▊   | 1.98G/2.92G [01:00<00:32, 28.5MB/s]
     68%|██████▊   | 1.98G/2.92G [01:00<00:34, 27.2MB/s]
     68%|██████▊   | 1.99G/2.92G [01:01<00:36, 25.4MB/s]
     68%|██████▊   | 1.99G/2.92G [01:01<00:37, 24.7MB/s]
     68%|██████▊   | 1.99G/2.92G [01:01<00:39, 23.7MB/s]
     68%|██████▊   | 2.00G/2.92G [01:01<00:32, 28.3MB/s]
     69%|██████▊   | 2.00G/2.92G [01:01<00:27, 33.2MB/s]
     69%|██████▊   | 2.01G/2.92G [01:01<00:24, 36.7MB/s]
     69%|██████▉   | 2.01G/2.92G [01:01<00:29, 31.0MB/s]
     69%|██████▉   | 2.01G/2.92G [01:01<00:28, 32.0MB/s]
     69%|██████▉   | 2.02G/2.92G [01:01<00:25, 35.2MB/s]
     69%|██████▉   | 2.02G/2.92G [01:02<00:24, 37.3MB/s]
     69%|██████▉   | 2.03G/2.92G [01:02<00:22, 40.1MB/s]
     70%|██████▉   | 2.03G/2.92G [01:02<00:21, 41.6MB/s]
     70%|██████▉   | 2.04G/2.92G [01:02<00:20, 44.1MB/s]
     70%|██████▉   | 2.04G/2.92G [01:02<00:29, 29.6MB/s]
     70%|███████   | 2.04G/2.92G [01:02<00:30, 28.6MB/s]
     70%|███████   | 2.05G/2.92G [01:02<00:32, 27.1MB/s]
     70%|███████   | 2.05G/2.92G [01:02<00:31, 27.6MB/s]
     70%|███████   | 2.05G/2.92G [01:03<00:30, 28.2MB/s]
     70%|███████   | 2.06G/2.92G [01:03<00:30, 28.3MB/s]
     71%|███████   | 2.06G/2.92G [01:03<00:33, 26.0MB/s]
     71%|███████   | 2.06G/2.92G [01:03<00:31, 26.9MB/s]
     71%|███████   | 2.07G/2.92G [01:03<00:29, 29.1MB/s]
     71%|███████   | 2.07G/2.92G [01:03<00:23, 35.6MB/s]
     71%|███████   | 2.08G/2.92G [01:03<00:21, 39.8MB/s]
     71%|███████▏  | 2.08G/2.92G [01:03<00:20, 40.0MB/s]
     71%|███████▏  | 2.08G/2.92G [01:04<00:27, 30.8MB/s]
     72%|███████▏  | 2.09G/2.92G [01:04<00:26, 31.0MB/s]
     72%|███████▏  | 2.09G/2.92G [01:04<00:27, 29.9MB/s]
     72%|███████▏  | 2.10G/2.92G [01:04<00:29, 27.5MB/s]
     72%|███████▏  | 2.10G/2.92G [01:04<00:42, 19.2MB/s]
     72%|███████▏  | 2.10G/2.92G [01:04<00:40, 20.0MB/s]
     72%|███████▏  | 2.10G/2.92G [01:05<00:38, 21.2MB/s]
     72%|███████▏  | 2.11G/2.92G [01:05<00:35, 23.0MB/s]
     72%|███████▏  | 2.11G/2.92G [01:05<00:33, 24.1MB/s]
     72%|███████▏  | 2.11G/2.92G [01:05<00:27, 29.5MB/s]
     73%|███████▎  | 2.12G/2.92G [01:05<00:22, 34.9MB/s]
     73%|███████▎  | 2.12G/2.92G [01:05<00:21, 37.3MB/s]
     73%|███████▎  | 2.13G/2.92G [01:05<00:19, 41.1MB/s]
     73%|███████▎  | 2.13G/2.92G [01:05<00:18, 43.2MB/s]
     73%|███████▎  | 2.14G/2.92G [01:05<00:18, 42.2MB/s]
     73%|███████▎  | 2.14G/2.92G [01:05<00:17, 44.4MB/s]
     74%|███████▎  | 2.15G/2.92G [01:06<00:18, 42.6MB/s]
     74%|███████▍  | 2.15G/2.92G [01:06<00:16, 45.9MB/s]
     74%|███████▍  | 2.16G/2.92G [01:06<00:16, 46.4MB/s]
     74%|███████▍  | 2.16G/2.92G [01:06<00:22, 33.6MB/s]
     74%|███████▍  | 2.17G/2.92G [01:06<00:22, 33.9MB/s]
     74%|███████▍  | 2.17G/2.92G [01:06<00:22, 33.6MB/s]
     74%|███████▍  | 2.17G/2.92G [01:06<00:23, 32.1MB/s]
     75%|███████▍  | 2.18G/2.92G [01:06<00:24, 30.7MB/s]
     75%|███████▍  | 2.18G/2.92G [01:07<00:25, 29.2MB/s]
     75%|███████▍  | 2.18G/2.92G [01:07<00:25, 28.5MB/s]
     75%|███████▍  | 2.19G/2.92G [01:07<00:27, 27.1MB/s]
     75%|███████▌  | 2.19G/2.92G [01:07<00:22, 32.2MB/s]
     75%|███████▌  | 2.20G/2.92G [01:07<00:19, 36.4MB/s]
     75%|███████▌  | 2.20G/2.92G [01:07<00:19, 37.1MB/s]
     76%|███████▌  | 2.20G/2.92G [01:07<00:18, 38.4MB/s]
     76%|███████▌  | 2.21G/2.92G [01:07<00:17, 39.5MB/s]
     76%|███████▌  | 2.21G/2.92G [01:07<00:17, 40.1MB/s]
     76%|███████▌  | 2.22G/2.92G [01:08<00:17, 41.1MB/s]
     76%|███████▌  | 2.22G/2.92G [01:08<00:17, 40.2MB/s]
     76%|███████▋  | 2.23G/2.92G [01:08<00:16, 41.1MB/s]
     76%|███████▋  | 2.23G/2.92G [01:08<00:35, 19.6MB/s]
     77%|███████▋  | 2.23G/2.92G [01:08<00:31, 21.6MB/s]
     77%|███████▋  | 2.24G/2.92G [01:08<00:29, 23.5MB/s]
     77%|███████▋  | 2.24G/2.92G [01:09<00:27, 24.5MB/s]
     77%|███████▋  | 2.24G/2.92G [01:09<00:26, 25.3MB/s]
     77%|███████▋  | 2.25G/2.92G [01:09<00:26, 25.6MB/s]
     77%|███████▋  | 2.25G/2.92G [01:09<00:23, 28.7MB/s]
     77%|███████▋  | 2.25G/2.92G [01:09<00:20, 32.6MB/s]
     77%|███████▋  | 2.26G/2.92G [01:09<00:18, 35.9MB/s]
     78%|███████▊  | 2.26G/2.92G [01:09<00:16, 39.0MB/s]
     78%|███████▊  | 2.27G/2.92G [01:09<00:15, 40.8MB/s]
     78%|███████▊  | 2.27G/2.92G [01:09<00:16, 38.8MB/s]
     78%|███████▊  | 2.28G/2.92G [01:09<00:16, 38.3MB/s]
     78%|███████▊  | 2.28G/2.92G [01:10<00:16, 39.4MB/s]
     78%|███████▊  | 2.28G/2.92G [01:10<00:15, 41.8MB/s]
     78%|███████▊  | 2.29G/2.92G [01:10<00:29, 21.4MB/s]
     79%|███████▊  | 2.29G/2.92G [01:10<00:27, 22.9MB/s]
     79%|███████▊  | 2.30G/2.92G [01:10<00:25, 24.5MB/s]
     79%|███████▉  | 2.30G/2.92G [01:10<00:25, 24.4MB/s]
     79%|███████▉  | 2.30G/2.92G [01:11<00:24, 24.8MB/s]
     79%|███████▉  | 2.30G/2.92G [01:11<00:22, 27.6MB/s]
     79%|███████▉  | 2.31G/2.92G [01:11<00:17, 34.5MB/s]
     79%|███████▉  | 2.32G/2.92G [01:11<00:14, 41.4MB/s]
     80%|███████▉  | 2.32G/2.92G [01:11<00:12, 46.2MB/s]
     80%|███████▉  | 2.33G/2.92G [01:11<00:11, 49.8MB/s]
     80%|███████▉  | 2.33G/2.92G [01:11<00:11, 52.4MB/s]
     80%|████████  | 2.34G/2.92G [01:11<00:10, 54.2MB/s]
     80%|████████  | 2.35G/2.92G [01:11<00:10, 55.4MB/s]
     81%|████████  | 2.35G/2.92G [01:11<00:10, 54.6MB/s]
     81%|████████  | 2.36G/2.92G [01:12<00:11, 50.1MB/s]
     81%|████████  | 2.36G/2.92G [01:12<00:11, 47.7MB/s]
     81%|████████  | 2.37G/2.92G [01:12<00:11, 46.3MB/s]
     81%|████████▏ | 2.37G/2.92G [01:12<00:25, 21.6MB/s]
     81%|████████▏ | 2.38G/2.92G [01:13<00:23, 22.8MB/s]
     82%|████████▏ | 2.38G/2.92G [01:13<00:22, 23.5MB/s]
     82%|████████▏ | 2.38G/2.92G [01:13<00:23, 22.4MB/s]
     82%|████████▏ | 2.39G/2.92G [01:13<00:22, 24.1MB/s]
     82%|████████▏ | 2.39G/2.92G [01:13<00:18, 28.6MB/s]
     82%|████████▏ | 2.39G/2.92G [01:13<00:15, 33.0MB/s]
     82%|████████▏ | 2.40G/2.92G [01:13<00:14, 36.5MB/s]
     82%|████████▏ | 2.40G/2.92G [01:13<00:13, 38.5MB/s]
     83%|████████▎ | 2.41G/2.92G [01:13<00:13, 38.7MB/s]
     83%|████████▎ | 2.41G/2.92G [01:14<00:12, 39.5MB/s]
     83%|████████▎ | 2.42G/2.92G [01:14<00:12, 41.6MB/s]
     83%|████████▎ | 2.42G/2.92G [01:14<00:11, 43.1MB/s]
     83%|████████▎ | 2.43G/2.92G [01:14<00:10, 45.0MB/s]
     83%|████████▎ | 2.43G/2.92G [01:14<00:10, 45.7MB/s]
     83%|████████▎ | 2.44G/2.92G [01:14<00:15, 30.7MB/s]
     84%|████████▎ | 2.44G/2.92G [01:14<00:15, 30.1MB/s]
     84%|████████▎ | 2.44G/2.92G [01:14<00:16, 29.3MB/s]
     84%|████████▍ | 2.45G/2.92G [01:15<00:17, 26.9MB/s]
     84%|████████▍ | 2.45G/2.92G [01:15<00:17, 27.5MB/s]
     84%|████████▍ | 2.45G/2.92G [01:15<00:16, 27.9MB/s]
     84%|████████▍ | 2.46G/2.92G [01:15<00:17, 25.9MB/s]
     84%|████████▍ | 2.46G/2.92G [01:15<00:17, 25.8MB/s]
     84%|████████▍ | 2.46G/2.92G [01:15<00:16, 27.0MB/s]
     85%|████████▍ | 2.47G/2.92G [01:15<00:13, 32.9MB/s]
     85%|████████▍ | 2.47G/2.92G [01:15<00:11, 38.3MB/s]
     85%|████████▍ | 2.48G/2.92G [01:15<00:10, 41.6MB/s]
     85%|████████▌ | 2.48G/2.92G [01:16<00:10, 40.8MB/s]
     85%|████████▌ | 2.49G/2.92G [01:16<00:10, 42.3MB/s]
     85%|████████▌ | 2.49G/2.92G [01:16<00:09, 43.8MB/s]
     85%|████████▌ | 2.49G/2.92G [01:16<00:09, 44.2MB/s]
     86%|████████▌ | 2.50G/2.92G [01:16<00:08, 46.9MB/s]
     86%|████████▌ | 2.51G/2.92G [01:16<00:08, 49.0MB/s]
     86%|████████▌ | 2.51G/2.92G [01:17<00:16, 24.3MB/s]
     86%|████████▌ | 2.51G/2.92G [01:17<00:15, 25.3MB/s]
     86%|████████▋ | 2.52G/2.92G [01:17<00:17, 23.5MB/s]
     86%|████████▋ | 2.52G/2.92G [01:17<00:17, 23.3MB/s]
     86%|████████▋ | 2.52G/2.92G [01:17<00:16, 23.3MB/s]
     87%|████████▋ | 2.53G/2.92G [01:17<00:16, 24.2MB/s]
     87%|████████▋ | 2.53G/2.92G [01:17<00:12, 31.4MB/s]
     87%|████████▋ | 2.54G/2.92G [01:17<00:09, 38.2MB/s]
     87%|████████▋ | 2.54G/2.92G [01:17<00:08, 43.8MB/s]
     87%|████████▋ | 2.55G/2.92G [01:18<00:07, 47.8MB/s]
     88%|████████▊ | 2.56G/2.92G [01:18<00:07, 50.8MB/s]
     88%|████████▊ | 2.56G/2.92G [01:18<00:06, 53.1MB/s]
     88%|████████▊ | 2.57G/2.92G [01:18<00:06, 55.5MB/s]
     88%|████████▊ | 2.57G/2.92G [01:18<00:06, 57.0MB/s]
     88%|████████▊ | 2.58G/2.92G [01:18<00:05, 59.5MB/s]
     89%|████████▊ | 2.59G/2.92G [01:19<00:11, 28.8MB/s]
     89%|████████▉ | 2.59G/2.92G [01:19<00:11, 28.0MB/s]
     89%|████████▉ | 2.59G/2.92G [01:19<00:11, 27.6MB/s]
     89%|████████▉ | 2.60G/2.92G [01:19<00:11, 28.0MB/s]
     89%|████████▉ | 2.60G/2.92G [01:19<00:10, 31.0MB/s]
     89%|████████▉ | 2.61G/2.92G [01:19<00:07, 38.7MB/s]
     90%|████████▉ | 2.62G/2.92G [01:19<00:06, 45.2MB/s]
     90%|████████▉ | 2.62G/2.92G [01:19<00:05, 50.4MB/s]
     90%|█████████ | 2.63G/2.92G [01:20<00:05, 51.6MB/s]
     90%|█████████ | 2.63G/2.92G [01:20<00:05, 55.9MB/s]
     91%|█████████ | 2.64G/2.92G [01:20<00:04, 59.3MB/s]
     91%|█████████ | 2.65G/2.92G [01:20<00:04, 62.4MB/s]
     91%|█████████ | 2.66G/2.92G [01:20<00:04, 64.6MB/s]
     91%|█████████ | 2.66G/2.92G [01:20<00:03, 66.6MB/s]
     91%|█████████▏| 2.67G/2.92G [01:20<00:05, 43.6MB/s]
     92%|█████████▏| 2.68G/2.92G [01:20<00:06, 39.4MB/s]
     92%|█████████▏| 2.68G/2.92G [01:21<00:06, 36.8MB/s]
     92%|█████████▏| 2.68G/2.92G [01:21<00:07, 33.5MB/s]
     92%|█████████▏| 2.69G/2.92G [01:21<00:06, 33.9MB/s]
     92%|█████████▏| 2.69G/2.92G [01:21<00:07, 28.8MB/s]
     92%|█████████▏| 2.70G/2.92G [01:21<00:07, 30.6MB/s]
     93%|█████████▎| 2.70G/2.92G [01:21<00:05, 40.1MB/s]
     93%|█████████▎| 2.71G/2.92G [01:21<00:04, 48.4MB/s]
     93%|█████████▎| 2.72G/2.92G [01:21<00:03, 54.5MB/s]
     93%|█████████▎| 2.72G/2.92G [01:22<00:03, 58.9MB/s]
     94%|█████████▎| 2.73G/2.92G [01:22<00:02, 62.4MB/s]
     94%|█████████▍| 2.74G/2.92G [01:22<00:02, 66.4MB/s]
     94%|█████████▍| 2.75G/2.92G [01:22<00:02, 69.3MB/s]
     94%|█████████▍| 2.75G/2.92G [01:22<00:02, 71.7MB/s]
     95%|█████████▍| 2.76G/2.92G [01:22<00:02, 71.6MB/s]
     95%|█████████▍| 2.77G/2.92G [01:23<00:04, 36.8MB/s]
     95%|█████████▌| 2.77G/2.92G [01:23<00:04, 32.8MB/s]
     95%|█████████▌| 2.78G/2.92G [01:23<00:04, 33.0MB/s]
     95%|█████████▌| 2.78G/2.92G [01:23<00:04, 30.4MB/s]
     96%|█████████▌| 2.79G/2.92G [01:23<00:03, 33.1MB/s]
     96%|█████████▌| 2.80G/2.92G [01:23<00:02, 42.8MB/s]
     96%|█████████▌| 2.80G/2.92G [01:23<00:02, 50.9MB/s]
     96%|█████████▋| 2.81G/2.92G [01:23<00:01, 57.8MB/s]
     97%|█████████▋| 2.82G/2.92G [01:24<00:01, 63.3MB/s]
     97%|█████████▋| 2.83G/2.92G [01:24<00:01, 67.5MB/s]
     97%|█████████▋| 2.84G/2.92G [01:24<00:01, 72.3MB/s]
     97%|█████████▋| 2.84G/2.92G [01:24<00:00, 75.6MB/s]
     98%|█████████▊| 2.85G/2.92G [01:24<00:00, 77.6MB/s]
     98%|█████████▊| 2.86G/2.92G [01:24<00:00, 75.8MB/s]
     98%|█████████▊| 2.87G/2.92G [01:25<00:01, 34.9MB/s]
     98%|█████████▊| 2.87G/2.92G [01:25<00:01, 34.9MB/s]
     99%|█████████▊| 2.88G/2.92G [01:25<00:01, 34.4MB/s]
     99%|█████████▉| 2.88G/2.92G [01:25<00:01, 33.6MB/s]
     99%|█████████▉| 2.89G/2.92G [01:25<00:00, 35.3MB/s]
     99%|█████████▉| 2.90G/2.92G [01:25<00:00, 46.1MB/s]
    100%|█████████▉| 2.90G/2.92G [01:25<00:00, 54.2MB/s]
    100%|█████████▉| 2.91G/2.92G [01:25<00:00, 62.3MB/s]
    100%|██████████| 2.92G/2.92G [01:26<00:00, 33.9MB/s]
    Dataset CIFAR10C
        Number of datapoints: 190000
        Root location: data/CIFAR-10-C
        StandardTransform
    Transform: Compose(
                     ToImage()
                     ToDtype(scale=True)
                     Normalize(mean=[0.4914, 0.4822, 0.4465], std=[0.2023, 0.1994, 0.201], inplace=False)
               )




.. GENERATED FROM PYTHON SOURCE LINES 69-74

Model Initialization
--------------------

We will use the ResNet18 architecture, a widely adopted convolutional neural network known for its deep residual learning capabilities.
The model is initialized with 10 output classes corresponding to the CIFAR-10 dataset categories.

.. GENERATED FROM PYTHON SOURCE LINES 74-78

.. code-block:: Python


    # Initialize the ResNet18 model with 10 output classes
    model = resnet(arch=18, in_channels=3, num_classes=10)








.. GENERATED FROM PYTHON SOURCE LINES 79-89

Define the Classification Routine
---------------------------------

The `ClassificationRoutine` is one of the most crucial building blocks in TorchUncertainty.
It streamlines the training and evaluation processes.
It integrates the model, loss function, and optimizer into a cohesive routine compatible with PyTorch Lightning's Trainer.
This abstraction simplifies the implementation of standard training loops and evaluation protocols.
To come back to what matters in this tutorial, the routine also handles the evaluation of the performance
of the model under distribution shift detection. To enable it, activate the `eval_shift` flag. Note that you can also evaluate
the Out-of-distribution detection at the same time by also setting `eval_ood` to True.

.. GENERATED FROM PYTHON SOURCE LINES 89-101

.. code-block:: Python


    # Define the loss function: Cross-Entropy Loss for multi-class classification
    criterion = nn.CrossEntropyLoss()

    # Define the optimizer: Adam optimizer with a learning rate of 0.001
    optimizer = optim.Adam(model.parameters(), lr=0.001)

    # Initialize the ClassificationRoutine with the model, number of classes, loss function, and optimizer
    routine = ClassificationRoutine(
        model=model, num_classes=10, loss=criterion, optim_recipe=optimizer, eval_shift=True
    )








.. GENERATED FROM PYTHON SOURCE LINES 102-108

Training the Model
------------------

With the routine defined, we can now set up the TUTrainer and commence training.
The TUTrainer handles the training loop, including epoch management, logging, and checkpointing.
We specify the maximum number of epochs, the precision and the device to be used.

.. GENERATED FROM PYTHON SOURCE LINES 108-117

.. code-block:: Python


    # Initialize the TUTrainer with a maximum of 10 epochs and the specified device
    trainer = TUTrainer(
        max_epochs=10, precision="16-mixed", accelerator="cuda", devices=1, enable_progress_bar=False
    )

    # Begin training the model using the CIFAR-10 DataModule
    trainer.fit(routine, datamodule=datamodule)








.. GENERATED FROM PYTHON SOURCE LINES 118-124

Evaluating on In-Distribution and Distribution-shifted Data
-----------------------------------------------------------

Now that the model is trained, we can evaluate its performance on the original in-distribution test set,
as well as the distribution-shifted set. Typing the next line will automatically compute the in-distribution
metrics as well as their values on the distribution-shifted set.

.. GENERATED FROM PYTHON SOURCE LINES 124-128

.. code-block:: Python


    # Evaluate the trained model on the original CIFAR-10 test set and on CIFAR-10C
    results = trainer.test(routine, datamodule=datamodule)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃ Test metric  ┃      Classification       ┃
    ┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │     Acc      │          76.040%          │
    │    Brier     │          0.33169          │
    │   Entropy    │          0.56686          │
    │     NLL      │          0.68303          │
    └──────────────┴───────────────────────────┘
    ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃ Test metric  ┃        Calibration        ┃
    ┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │     ECE      │          4.260%           │
    │     aECE     │          4.178%           │
    └──────────────┴───────────────────────────┘
    ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃ Test metric  ┃ Selective Classification  ┃
    ┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │    AUGRC     │          5.781%           │
    │     AURC     │          7.552%           │
    │  Cov@5Risk   │           nan%            │
    │  Risk@80Cov  │          15.238%          │
    └──────────────┴───────────────────────────┘
    ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃ Test metric  ┃  Distribution Shift lvl5  ┃
    ┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │    AUGRC     │          7.084%           │
    │     AURC     │          9.525%           │
    │     Acc      │          73.101%          │
    │    Brier     │          0.37237          │
    │  Cov@5Risk   │           nan%            │
    │     ECE      │          5.066%           │
    │     NLL      │          0.77899          │
    │  Risk@80Cov  │          18.422%          │
    │     aECE     │          5.064%           │
    └──────────────┴───────────────────────────┘




.. GENERATED FROM PYTHON SOURCE LINES 129-135

Distribution-shift metrics
--------------------------

The distribution shift metrics are computed only when the `eval_shift` flag of the routine is True.
In this case, the values of the metrics are shown last. They correspond to the in-distribution metrics but
computed on the distribution-shifted datasets, hence the worse results.


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (2 minutes 34.911 seconds)


.. _sphx_glr_download_auto_tutorials_Classification_tutorial_distribution_shift.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: tutorial_distribution_shift.ipynb <tutorial_distribution_shift.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: tutorial_distribution_shift.py <tutorial_distribution_shift.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: tutorial_distribution_shift.zip <tutorial_distribution_shift.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_