---
title: "pytorch - GPU"
subtitle: "Lecture 19"
author: "Dr. Colin Rundel"
footer: "Sta 663 - Spring 2025"
format:
  revealjs:
    theme: slides.scss
    transition: fade
    slide-number: true
    self-contained: true
execute: 
  echo: true
---

```{python setup}
#| include: false
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
import scipy

import torch

import os

plt.rcParams['figure.dpi'] = 200

torch.set_printoptions(
  edgeitems=30, linewidth=46,
  precision = 4
)

np.set_printoptions(
  edgeitems=30, linewidth=48,
  precision = 5, suppress=True
)

pd.set_option("display.width", 130)
pd.set_option("display.max_columns", 10)
pd.set_option("display.precision", 6)
```

```{r r_setup}
#| include: false

flexiblas::flexiblas_load_backend("ATLAS")
flexiblas::flexiblas_switch(3)

knitr::opts_chunk$set(
  fig.align="center",
  cache=FALSE
)

local({
  hook_err_old <- knitr::knit_hooks$get("error")  # save the old hook
  knitr::knit_hooks$set(error = function(x, options) {
    # now do whatever you want to do with x, and pass
    # the new x to the old hook
    x = sub("## \n## Detailed traceback:\n.*$", "", x)
    x = sub("Error in py_call_impl\\(.*?\\)\\: ", "", x)
    #x = stringr::str_wrap(x, width = 100)
    hook_err_old(x, options)
  })
  
  hook_warn_old <- knitr::knit_hooks$get("warning")  # save the old hook
  knitr::knit_hooks$set(warning = function(x, options) {
    x = sub("<string>:1: ", "", x)
    #x = stringr::str_wrap(x, width = 100)
    hook_warn_old(x, options)
  })
  
  hook_msg_old <- knitr::knit_hooks$get("output")  # save the old hook
  knitr::knit_hooks$set(output = function(x, options) {
    if (is.null(options$wrap))
      options$wrap = TRUE
    
    x = stringr::str_replace(x, "(## ).* ([A-Za-z]+Warning:)", "\\1\\2")
    x = stringr::str_split(x, "\n")[[1]]
    #x = stringr::str_wrap(x, width = 120, exdent = 3)
    x = stringr::str_remove_all(x, "\r")
    if (options$wrap) {
        x = stringi::stri_wrap(x, width=120, exdent = 3, normalize=FALSE)
    } else {
        # Trim from of the profiling tables
        n = x |> stringr::str_match("^\\s+") |> nchar() |> min(na.rm=TRUE)
        x = stringr::str_remove(x, paste0("^ {",n,"}|^-{",n,"}"))
    }
    x = paste(x, collapse="\n")
    
    #x = stringr::str_wrap(x, width = 100)
    hook_msg_old(x, options)
  })
})
```

## CUDA

> CUDA (or Compute Unified Device Architecture) is a parallel computing platform and application programming interface (API) that allows software to use certain types of graphics processing unit (GPU) for general purpose processing, an approach called general-purpose computing on GPUs (GPGPU). CUDA is a software layer that gives direct access to the GPU's virtual instruction set and parallel computational elements, for the execution of compute kernels.

<br/>

Core libraries:

:::: {.columns}
::: {.column width='33%'}
* cuBLAS

* cuSOLVER

* cuSPARSE
:::

::: {.column width='33%'}
* cuTENSOR

* cuFFT

* cuRAND
:::

::: {.column width='33%'}
* Thrust

* cuDNN
:::
::::



## CUDA Kernels

::: {.xsmall}
```c
// Kernel - Adding two matrices MatA and MatB
__global__ void MatAdd(float MatA[N][N], float MatB[N][N], float MatC[N][N])
{
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    int j = blockIdx.y * blockDim.y + threadIdx.y;
    if (i < N && j < N)
        MatC[i][j] = MatA[i][j] + MatB[i][j];
}
 
int main()
{
    ...
    // Matrix addition kernel launch from host code
    dim3 threadsPerBlock(16, 16);
    dim3 numBlocks(
        (N + threadsPerBlock.x -1) / threadsPerBlock.x, 
        (N+threadsPerBlock.y -1) / threadsPerBlock.y
    );
    
    MatAdd<<<numBlocks, threadsPerBlock>>>(MatA, MatB, MatC);
    ...
}
```
:::


##

![](imgs/gpu_bench1.png){fig-align="center" width="100%"}


##

![](imgs/gpu_bench2.png){fig-align="center" width="100%"}

## GPU Status

::: {.xsmall}
```{bash smi}
nvidia-smi
```
:::

## Torch GPU Information

::: {.xsmall}
```{python has_cuda}
torch.cuda.is_available()

torch.cuda.device_count()

torch.cuda.get_device_name("cuda:0")
torch.cuda.get_device_name("cuda:1")

torch.cuda.get_device_properties(0)
torch.cuda.get_device_properties(1)
```
:::


## GPU Tensors

Usage of the GPU is governed by the location of the Tensors - to use the GPU we allocate them on the GPU device.

:::: {.columns .xsmall}
::: {.column width='50%'}
```{python tensor_devices}
cpu = torch.device('cpu')
cuda0 = torch.device('cuda:0')
cuda1 = torch.device('cuda:1')

x = torch.linspace(0,1,5, device=cuda0); x
y = torch.randn(5,2, device=cuda0); y
z = torch.rand(2,3, device=cpu); z
```
:::

::: {.column width='50%' .fragment}
```{python tensor_mult}
#| error: true
x @ y
y @ z
y @ z.to(cuda0)
```
:::
::::


## NN Layers + GPU

NN layers (parameters) also need to be assigned to the GPU to be used with GPU tensors,

::: {.xsmall}
```{python nn1}
#| error: true
nn = torch.nn.Linear(5,5)
X = torch.randn(10,5).cuda()
```
:::

. . .

::: {.xsmall}
```{python nn2}
#| error: true
nn(X)
```
:::

. . .

:::: {.columns .xsmall}
::: {.column width='50%'}
```{python nn3}
nn.cuda()(X)
```
:::

::: {.column width='50%'}
```{python nn4}
nn.to(device="cuda")(X)
```
:::
::::


## Back to MNIST

Same MNIST data from last time (1x8x8 images),

::: {.xsmall}
```{python mnist}
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

digits = load_digits()
X, y = digits.data, digits.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, shuffle=True, random_state=1234
)

X_train = torch.from_numpy(X_train).float()
y_train = torch.from_numpy(y_train)
X_test = torch.from_numpy(X_test).float()
y_test = torch.from_numpy(y_test)
```
:::

. . .

To use the GPU for computation we need to copy these tensors to the GPU,

::: {.xsmall}
```{python}
X_train_cuda = X_train.to(device=cuda0)
y_train_cuda = y_train.to(device=cuda0)
X_test_cuda = X_test.to(device=cuda0)
y_test_cuda = y_test.to(device=cuda0)
```
:::


## Convolutional NN

::: {.xsmall}
```{python}
class mnist_conv_model(torch.nn.Module):
    def __init__(self, device):
        super().__init__()
        self.device = torch.device(device)
        
        self.model = torch.nn.Sequential(
          torch.nn.Unflatten(1, (1,8,8)),
          torch.nn.Conv2d(
            in_channels=1, out_channels=8,
            kernel_size=3, stride=1, padding=1
          ),
          torch.nn.ReLU(),
          torch.nn.MaxPool2d(kernel_size=2),
          torch.nn.Flatten(),
          torch.nn.Linear(8 * 4 * 4, 10)
        ).to(device=self.device)
        
    def forward(self, X):
        return self.model(X)
    
    def fit(self, X, y, lr=0.001, n=1000, acc_step=10):
      opt = torch.optim.SGD(self.parameters(), lr=lr, momentum=0.9) 
      losses = []
      for i in range(n):
          opt.zero_grad()
          loss = torch.nn.CrossEntropyLoss()(self(X), y)
          loss.backward()
          opt.step()
          losses.append(loss.item())
      
      return losses
    
    def accuracy(self, X, y):
      val, pred = torch.max(self(X), dim=1)
      return( (pred == y).sum() / len(y) )
```
:::


## CPU vs Cuda

:::: {.columns .xsmall}
::: {.column width='50%'}
```{python}
m = mnist_conv_model(device="cpu")
loss = m.fit(X_train, y_train, n=1000)
loss[-1]
m.accuracy(X_test, y_test)
```
:::

::: {.column width='50%'}
```{python}
m_cuda = mnist_conv_model(device="cuda")
loss = m_cuda.fit(X_train_cuda, y_train_cuda, n=1000)
loss[-1]
m_cuda.accuracy(X_test_cuda, y_test_cuda)
```
:::
::::

. . .

::: {.center .large}
Why are the answers here different?
:::

. . .

::: {.columns .xsmall}
::: {.column}
```{python}
X_train.dtype
```
:::
::: {.column}
```{python}
X_train_cuda.dtype
```
:::
:::


## Performance

:::: {.columns .xsmall}
::: {.column width='50%'}
CPU performance:
```{python}
m = mnist_conv_model(device="cpu")

start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)

start.record()
loss = m.fit(X_train, y_train, n=1000)
end.record()

torch.cuda.synchronize()
print(start.elapsed_time(end) / 1000) 
```
:::

::: {.column width='50%'}
GPU performance:
```{python}
m_cuda = mnist_conv_model(device="cuda")

start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)

start.record()
loss = m_cuda.fit(X_train_cuda, y_train_cuda, n=1000)
end.record()

torch.cuda.synchronize()
print(start.elapsed_time(end) / 1000) 
```
:::
::::


## Profiling CPU - 1 forward step

::: {.tiny}
```{python}
m = mnist_conv_model(device="cpu")
with torch.autograd.profiler.profile(with_stack=True, profile_memory=True) as prof_cpu:
    tmp = m(X_train)
```

```{python}
#| wrap: false
print(prof_cpu.key_averages().table(sort_by='self_cpu_time_total', row_limit=10))
```
:::

## Profiling GPU - 1 forward step

::: {.tiny}
```{python}
m_cuda = mnist_conv_model(device="cuda")
with torch.autograd.profiler.profile(with_stack=True) as prof_cuda:
    tmp = m_cuda(X_train_cuda)
```

```{python}
#| wrap: false
print(prof_cuda.key_averages().table(sort_by='self_cpu_time_total', row_limit=10))
```
:::


## Profiling CPU - fit

::: {.tiny}
```{python}
m = mnist_conv_model(device="cpu")
with torch.autograd.profiler.profile(with_stack=True, profile_memory=True) as prof_cpu:
    losses = m.fit(X_train, y_train, n=1000)
```

```{python}
#| wrap: false
print(prof_cpu.key_averages().table(sort_by='self_cpu_time_total', row_limit=10))
```
:::

## Profiling GPU - fit

::: {.tiny}
```{python}
m_cuda = mnist_conv_model(device="cuda")
with torch.autograd.profiler.profile(with_stack=True) as prof_cuda:
    losses = m_cuda.fit(X_train_cuda, y_train_cuda, n=1000)
```

```{python}
#| wrap: false
print(prof_cuda.key_averages().table(sort_by='self_cpu_time_total', row_limit=10))
```
:::



# CIFAR10

<br/>

::: {.xsmall}
[homepage](https://www.cs.toronto.edu/~kriz/cifar.html)
:::

## Loading the data

::: {.xsmall}
```{python}
#| results: hide
import torchvision

training_data = torchvision.datasets.CIFAR10(
    root="/data",
    train=True,
    download=True,
    transform=torchvision.transforms.ToTensor()
)

test_data = torchvision.datasets.CIFAR10(
    root="/data",
    train=False,
    download=True,
    transform=torchvision.transforms.ToTensor()
)
```
:::

Downloads data to "/data/cifar-10-batches-py" which is ~178M on disk.

## CIFAR10 data

::: {.xsmall}
```{python}
training_data.classes
training_data.data.shape
test_data.data.shape
```
:::

. . .

::: {.xsmall}
```{python}
training_data[0]
```
:::

## Example data

```{python echo=FALSE, out.width="85%"}
fig, axes = plt.subplots(nrows=4, ncols=6, figsize=(10, 6), layout="constrained")

for i, ax in enumerate([ax for row in axes for ax in row]):
    ax.set_axis_off()
    img, cls = training_data[i]
    
    p = ax.imshow(img.numpy().transpose((1,2,0)))
    t = ax.set_title(f"{training_data.classes[cls]}")
    
plt.show()
```

## Data Loaders

Torch handles large datasets and minibatches through the use of the `DataLoader` class, 

::: {.xsmall}
```{python}
training_loader = torch.utils.data.DataLoader(
    training_data, 
    batch_size=100,
    shuffle=True,
    num_workers=4,
    pin_memory=True
)

test_loader = torch.utils.data.DataLoader(
    test_data, 
    batch_size=100,
    shuffle=True,
    num_workers=4,
    pin_memory=True
)
```
:::

## Loader as generator

The resulting `DataLoader` is iterable and yields the features and targets for each batch,

::: {.xsmall}
```{python}
training_loader
```
:::

. . .

::: {.xsmall}
```{python}
X, y = next(iter(training_loader))
X.shape
y.shape
```
:::

## Custom Datasets

::: {.small}
In this case we got our data (`training_data` and `test_data`) directly from torchvision which gave us a `dataset` object to use with our `DataLoader`. If we do not have a `Dataset` object then we need to create a custom class for our data telling torch how to load it. 

You class must define the methods: `__init__()`, `__len__()`, and `__get_item__()`.
:::

::: {.xsmall}
```{python}
class data(torch.utils.data.Dataset):
    def __init__(self, X, y):
        self.X = X
        self.y = y
    
    def __len__(self):
        return len(self.X)
    
    def __getitem__(self, idx):
        return self.X[idx], self.y[idx]

mnist_train = data(X_train, y_train)
```
:::

## Custom loader

::: {.xsmall}
```{python}
mnist_loader = torch.utils.data.DataLoader(
    mnist_train, 
    batch_size=1000,
    shuffle=True
)

it = iter(mnist_loader)
```
:::

::: {.columns .xsmall}
::: {.column}
```{python}
X, y = next(it)
X.shape
y.shape
```
:::
::: {.column}
```{python}
X, y = next(it)
X.shape
y.shape
```
:::
:::



## CIFAR CNN

::: {.xsmall}
```{python}
#| code-line-numbers: "|2-19|2,4,19|24-42|27-34|37-40"
class cifar_conv_model(torch.nn.Module):
    def __init__(self, device):
        super().__init__()
        self.device = torch.device(device)
        self.epoch = 0
        self.model = torch.nn.Sequential(
            torch.nn.Conv2d(3, 6, kernel_size=5),
            torch.nn.ReLU(),
            torch.nn.MaxPool2d(2, 2),
            torch.nn.Conv2d(6, 16, kernel_size=5),
            torch.nn.ReLU(),
            torch.nn.MaxPool2d(2, 2),
            torch.nn.Flatten(),
            torch.nn.Linear(16 * 5 * 5, 120),
            torch.nn.ReLU(),
            torch.nn.Linear(120, 84),
            torch.nn.ReLU(),
            torch.nn.Linear(84, 10)
        ).to(device=self.device)
        
    def forward(self, X):
        return self.model(X)
    
    def fit(self, loader, epochs=10, n_report=250, lr=0.001):
        opt = torch.optim.SGD(self.parameters(), lr=lr, momentum=0.9) 
      
        for j in range(epochs):
            running_loss = 0.0
            for i, (X, y) in enumerate(loader):
                X, y = X.to(self.device), y.to(self.device)
                opt.zero_grad()
                loss = torch.nn.CrossEntropyLoss()(self(X), y)
                loss.backward()
                opt.step()
    
                # print statistics
                running_loss += loss.item()
                if i % n_report == (n_report-1):    # print every 100 mini-batches
                    print(f'[Epoch {self.epoch + 1}, Minibatch {i + 1:4d}] loss: {running_loss / n_report:.3f}')
                    running_loss = 0.0
            
            self.epoch += 1
```
:::

::: {.aside}
Based on [PyTorch cifar10 tutorial](https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html)
:::


## CNN Performance - CPU (1 step)

::: {.tiny}
```{python}
X, y = next(iter(training_loader))

m_cpu = cifar_conv_model(device="cpu")
tmp = m_cpu(X)

with torch.autograd.profiler.profile(with_stack=True) as prof_cpu:
    tmp = m_cpu(X)
```

```{python, wrap=FALSE}
#| code-overflow: scroll
print(prof_cpu.key_averages().table(sort_by='self_cpu_time_total', row_limit=10))
```
:::


## CNN Performance - GPU (1 step)

::: {.tiny}
```{python}
m_cuda = cifar_conv_model(device="cuda")
Xc, yc = X.to(device="cuda"), y.to(device="cuda")
tmp = m_cuda(Xc)
    
with torch.autograd.profiler.profile(with_stack=True) as prof_cuda:
    tmp = m_cuda(Xc)
```

```{python, wrap=FALSE}
print(prof_cuda.key_averages().table(sort_by='self_cpu_time_total', row_limit=10))
```
:::




## CNN Performance - CPU (1 epoch)

::: {.tiny}
```{python}
m_cpu = cifar_conv_model(device="cpu")

with torch.autograd.profiler.profile(with_stack=True) as prof_cpu:
    m_cpu.fit(loader=training_loader, epochs=1, n_report=501)
```

```{python, wrap=FALSE}
print(prof_cpu.key_averages().table(sort_by='self_cpu_time_total', row_limit=10))
```
:::

## CNN Performance - GPU (1 epoch)

::: {.tiny}
```{python}
m_cuda = cifar_conv_model(device="cuda")

with torch.autograd.profiler.profile(with_stack=True) as prof_cuda:
    m_cuda.fit(loader=training_loader, epochs=1, n_report=501)
```

```{python, wrap=FALSE}
print(prof_cuda.key_averages().table(sort_by='self_cpu_time_total', row_limit=10))
```
:::






## Loaders & Accuracy

::: {.small}
```{python}
def accuracy(model, loader, device):
    total, correct = 0, 0
    with torch.no_grad():
        for X, y in loader:
            X, y = X.to(device=device), y.to(device=device)
            pred = model(X)
            # the class with the highest energy is what we choose as prediction
            val, idx = torch.max(pred, 1)
            total += pred.size(0)
            correct += (idx == y).sum().item()
            
    return correct / total
```
:::


## Model fitting

::: {.xsmall}
```{python eval=FALSE}
m = cifar_conv_model("cuda")
m.fit(training_loader, epochs=10, n_report=500, lr=0.01)
```

```
## [Epoch 1, Minibatch  500] loss: 2.098
## [Epoch 2, Minibatch  500] loss: 1.692
## [Epoch 3, Minibatch  500] loss: 1.482
## [Epoch 4, Minibatch  500] loss: 1.374
## [Epoch 5, Minibatch  500] loss: 1.292
## [Epoch 6, Minibatch  500] loss: 1.226
## [Epoch 7, Minibatch  500] loss: 1.173
## [Epoch 8, Minibatch  500] loss: 1.117
## [Epoch 9, Minibatch  500] loss: 1.071
## [Epoch 10, Minibatch  500] loss: 1.035
```
:::

::: {.xsmall}
```{python eval=FALSE}
accuracy(m, training_loader, "cuda")
```
```
## 0.63444
```
```{python}
#| eval: false
accuracy(m, test_loader, "cuda")
```
```
## 0.572
```
:::


## More epochs

If we fit again, Torch continues with the existing model,

::: {.xsmall}
```{python eval=FALSE}
m.fit(training_loader, epochs=10, n_report=500)
```

```
## [Epoch 11, Minibatch  500] loss: 0.885
## [Epoch 12, Minibatch  500] loss: 0.853
## [Epoch 13, Minibatch  500] loss: 0.839
## [Epoch 14, Minibatch  500] loss: 0.828
## [Epoch 15, Minibatch  500] loss: 0.817
## [Epoch 16, Minibatch  500] loss: 0.806
## [Epoch 17, Minibatch  500] loss: 0.798
## [Epoch 18, Minibatch  500] loss: 0.787
## [Epoch 19, Minibatch  500] loss: 0.780
## [Epoch 20, Minibatch  500] loss: 0.773
```

```{python eval=FALSE}
accuracy(m, training_loader, "cuda")
```

```
## 0.73914
```

```{python}
#| eval: false
accuracy(m, test_loader, "cuda")
```

```
## 0.624
```
:::

## More epochs (again)

::: {.xsmall}
```{python eval=FALSE}
m.fit(training_loader, epochs=10, n_report=500)
```

```
## [Epoch 21, Minibatch  500] loss: 0.764
## [Epoch 22, Minibatch  500] loss: 0.756
## [Epoch 23, Minibatch  500] loss: 0.748
## [Epoch 24, Minibatch  500] loss: 0.739
## [Epoch 25, Minibatch  500] loss: 0.733
## [Epoch 26, Minibatch  500] loss: 0.726
## [Epoch 27, Minibatch  500] loss: 0.718
## [Epoch 28, Minibatch  500] loss: 0.710
## [Epoch 29, Minibatch  500] loss: 0.702
## [Epoch 30, Minibatch  500] loss: 0.698
```

```{python eval=FALSE}
accuracy(m, training_loader, "cuda")
```

```
## 0.76438
```

```{python}
#| eval: false
accuracy(m, test_loader, "cuda")
```

```
## 0.6217
```
:::

## The VGG16 model

::: {.xsmall}
```{python}
class VGG16(torch.nn.Module):
    def make_layers(self):
        cfg = [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 'M', 512, 512, 512, 'M', 512, 512, 512, 'M']
        layers = []
        in_channels = 3
        for x in cfg:
            if x == 'M':
                layers += [torch.nn.MaxPool2d(kernel_size=2, stride=2)]
            else:
                layers += [torch.nn.Conv2d(in_channels, x, kernel_size=3, padding=1),
                           torch.nn.BatchNorm2d(x),
                           torch.nn.ReLU(inplace=True)]
                in_channels = x
        layers += [
            torch.nn.AvgPool2d(kernel_size=1, stride=1),
            torch.nn.Flatten(),
            torch.nn.Linear(512,10)
        ]
        
        return torch.nn.Sequential(*layers).to(self.device)
    
    def __init__(self, device):
        super().__init__()
        self.device = torch.device(device)
        self.model = self.make_layers()
    
    def forward(self, X):
        return self.model(X)
```
:::

::: {.aside}
Based on code from [pytorch-cifar](https://github.com/kuangliu/pytorch-cifar), original [paper](https://arxiv.org/abs/1409.1556)
:::

## Model

::: {.xsmall}
```{python}
VGG16("cpu").model
```
:::


## VGG16 performance - CPU

::: {.xsmall}
```{python}
X, y = next(iter(training_loader))
m_cpu = VGG16(device="cpu")
tmp = m_cpu(X)

with torch.autograd.profiler.profile(with_stack=True) as prof_cpu:
    tmp = m_cpu(X)
```

```{python, wrap=FALSE}
print(prof_cpu.key_averages().table(sort_by='self_cpu_time_total', row_limit=10))
```
:::


## VGG16 performance - GPU

::: {.xsmall}
```{python}
m_cuda = VGG16(device="cuda")
Xc, yc = X.to(device="cuda"), y.to(device="cuda")
tmp = m_cuda(Xc)

with torch.autograd.profiler.profile(with_stack=True) as prof_cuda:
    tmp = m_cuda(Xc)
```

```{python, wrap=FALSE}
print(prof_cuda.key_averages().table(sort_by='self_cpu_time_total', row_limit=10))
```
:::


## VGG16 performance - Apple M1 GPU (mps)

::: {.xsmall}
```{python}
#| eval: false
m_mps = VGG16(device="mps")
Xm, ym = X.to(device="mps"), y.to(device="mps")

with torch.autograd.profiler.profile(with_stack=True) as prof_mps:
    tmp = m_mps(Xm)
```

```{python}
#| eval: false
print(prof_mps.key_averages().table(sort_by='self_cpu_time_total', row_limit=10))
```

```
--------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
                            Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls  
--------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
         aten::native_batch_norm        35.71%       3.045ms        35.71%       3.045ms     234.231us            13  
          aten::_mps_convolution        19.67%       1.677ms        19.88%       1.695ms     130.385us            13  
    aten::_batch_norm_impl_index        11.92%       1.016ms        36.02%       3.071ms     236.231us            13  
                     aten::relu_        11.29%     963.000us        11.29%     963.000us      74.077us            13  
                      aten::add_        10.40%     887.000us        10.44%     890.000us      68.462us            13  
--------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
Self CPU time total: 8.526ms
```
:::


## Fitting w/ `lr = 0.01`

::: {.xsmall}
```{python eval=FALSE}
m = VGG16(device="cuda")
fit(m, training_loader, epochs=10, n_report=500, lr=0.01)
```

```
## [Epoch 1, Minibatch  500] loss: 1.345
## [Epoch 2, Minibatch  500] loss: 0.790
## [Epoch 3, Minibatch  500] loss: 0.577
## [Epoch 4, Minibatch  500] loss: 0.445
## [Epoch 5, Minibatch  500] loss: 0.350
## [Epoch 6, Minibatch  500] loss: 0.274
## [Epoch 7, Minibatch  500] loss: 0.215
## [Epoch 8, Minibatch  500] loss: 0.167
## [Epoch 9, Minibatch  500] loss: 0.127
## [Epoch 10, Minibatch  500] loss: 0.103
```
:::

. . .

::: {.xsmall}
```{python eval=FALSE}
accuracy(model=m, loader=training_loader, device="cuda")
```

```
## 0.97008
```
```{python}
#| eval: false
accuracy(model=m, loader=test_loader, device="cuda")
```

```
## 0.8318
```
:::


## Fitting w/ `lr = 0.001`

::: {.xsmall}
```{python eval=FALSE}
m = VGG16(device="cuda")
fit(m, training_loader, epochs=10, n_report=500, lr=0.001)
```

```
## [Epoch 1, Minibatch  500] loss: 1.279
## [Epoch 2, Minibatch  500] loss: 0.827
## [Epoch 3, Minibatch  500] loss: 0.599
## [Epoch 4, Minibatch  500] loss: 0.428
## [Epoch 5, Minibatch  500] loss: 0.303
## [Epoch 6, Minibatch  500] loss: 0.210
## [Epoch 7, Minibatch  500] loss: 0.144
## [Epoch 8, Minibatch  500] loss: 0.108
## [Epoch 9, Minibatch  500] loss: 0.088
## [Epoch 10, Minibatch  500] loss: 0.063
```
:::

. . .

::: {.xsmall}
```{python eval=FALSE}
accuracy(model=m, loader=training_loader, device="cuda")
```

```
## 0.9815
```

```{python}
#| eval: false
accuracy(model=m, loader=test_loader, device="cuda")
```

```
## 0.7816
```
:::

## Report

::: {.xsmall}
```{python}
from sklearn.metrics import classification_report

def report(model, loader, device):
    y_true, y_pred = [], []
    with torch.no_grad():
        for X, y in loader:
            X = X.to(device=device)
            y_true.append( y.cpu().numpy() )
            y_pred.append( model(X).max(1)[1].cpu().numpy() )
    
    y_true = np.concatenate(y_true)
    y_pred = np.concatenate(y_pred)

    return classification_report(y_true, y_pred, target_names=loader.dataset.classes)
```
:::

##

::: {.xsmall}
```{python}
#| eval: false
print(report(model=m, loader=test_loader, device="cuda"))
```

```
##               precision    recall  f1-score   support
## 
##     airplane       0.82      0.88      0.85      1000
##   automobile       0.95      0.89      0.92      1000
##         bird       0.85      0.70      0.77      1000
##          cat       0.68      0.74      0.71      1000
##         deer       0.84      0.83      0.83      1000
##          dog       0.81      0.73      0.77      1000
##         frog       0.83      0.92      0.87      1000
##        horse       0.87      0.87      0.87      1000
##         ship       0.89      0.92      0.90      1000
##        truck       0.86      0.93      0.89      1000
## 
##     accuracy                           0.84     10000
##    macro avg       0.84      0.84      0.84     10000
## weighted avg       0.84      0.84      0.84     10000
```
:::


# Some "state-of-the-art"<br/>models

## Hugging Face

This is an online community and platform for sharing machine learning models (architectures and weights), data, and related artifacts. They also maintain a number of packages and related training materials that help with building, training, and deploying ML models.

Some notable resources,

* [`transformers`](https://huggingface.co/docs/transformers/index) - APIs and tools to easily download and train state-of-the-art (pretrained) transformer based models 

* [`diffusers`](https://huggingface.co/docs/diffusers/index) - provides pretrained vision and audio diffusion models, and serves as a modular toolbox for inference and training

* [`timm`](https://huggingface.co/docs/timm/index) - a library containing SOTA computer vision models, layers, utilities, optimizers, schedulers, data-loaders, augmentations, and training/evaluation scripts


## Stable Diffusion

::: {.xsmall}
```{python}
#| warning: false
#| message: false
#| results: hide
import torch
from diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler

model_id = "/data/stable-diffusion-2-1"

pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
pipe.enable_model_cpu_offload()
```
:::


. . .

::: {.xsmall}

```{python sd1}
#| results: hide
prompt = "a picture of thomas bayes with a cat on his lap"
```

```{python sd1-1}
#| results: hide
#| cache: true
prompt = "a picture of thomas bayes with a cat on his lap"
generator = [torch.Generator(device="cuda").manual_seed(i) for i in range(6)]
fit = pipe(prompt, generator=generator, num_inference_steps=20, num_images_per_prompt=6)
```
:::

. . .

::: {.xsmall}
```{python sd1-2}
#| cache: true
fit.images
```
:::

##

```{python sd1-plot}
#| echo: false
#| cache: true
fig, axes = plt.subplots(nrows=2, ncols=3, figsize=(10, 6), layout="constrained")

for i, ax in enumerate([ax for row in axes for ax in row]):
    ax.set_axis_off()
    p = ax.imshow(fit.images[i])
    
plt.show()
```

::: {.aside}
[Thomas Bayes GIS](https://www.google.com/search?q=thomas+bayes&source=lnms&tbm=isch&sa=X&biw=1280&bih=590&dpr=2)
:::


## Customizing prompts

::: {.xsmall}
```{python sd2-0}
#| results: hide
prompt = "a picture of thomas bayes with a cat on his lap"
prompts = [
  prompt + t for t in 
  ["in the style of a japanese wood block print",
   "as a hipster with facial hair and glasses",
   "as a simpsons character, cartoon, yellow",
   "in the style of a vincent van gogh painting",
   "in the style of a picasso painting",
   "with flowery wall paper"
  ]
]
```

```{python sd2}
#| results: hide
#| cache: true
generator = [torch.Generator(device="cuda").manual_seed(i) for i in range(6)]
fit = pipe(prompts, generator=generator, num_inference_steps=20, num_images_per_prompt=1)
```
:::

## 

```{python sd2-plot}
#| echo: false
#| cache: true
fig, axes = plt.subplots(nrows=2, ncols=3, figsize=(10, 6), layout="constrained")

for i, ax in enumerate([ax for row in axes for ax in row]):
    ax.set_axis_off()
    p = ax.imshow(fit.images[i])
    
plt.show()
```



## Increasing inference steps

::: {.xsmall}
```{python sd2-more}
#| results: hide
#| cache: true
generator = [torch.Generator(device="cuda").manual_seed(i) for i in range(6)]
fit = pipe(prompts, generator=generator, num_inference_steps=50, num_images_per_prompt=1)
```
:::

```{python sd2-more-plot}
#| echo: false
#| cache: true
fig, axes = plt.subplots(nrows=2, ncols=3, figsize=(10, 6), layout="constrained")

for i, ax in enumerate([ax for row in axes for ax in row]):
    ax.set_axis_off()
    p = ax.imshow(fit.images[i])
    
plt.show()
```

## A more current model

::: {.small}
This model is larger than the available GPU memory - so we adjust the weight types to make it fit.
:::

::: {.xsmall}
```{python}
from diffusers import BitsAndBytesConfig, SD3Transformer2DModel
from diffusers import StableDiffusion3Pipeline
import torch

model_id = "/data/stable-diffusion-3.5-medium"

nf4_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)
model_nf4 = SD3Transformer2DModel.from_pretrained(
    model_id,
    subfolder="transformer",
    quantization_config=nf4_config,
    torch_dtype=torch.bfloat16
)

pipe = StableDiffusion3Pipeline.from_pretrained(
    model_id, 
    transformer=model_nf4,
    torch_dtype=torch.bfloat16
)
pipe.enable_model_cpu_offload()
```
:::

## Images

::: {.xsmall}
```{python sd35}
#| results: hide
#| cache: true
generator = [torch.Generator(device="cuda").manual_seed(i) for i in range(6)]
fit = pipe(prompts, generator=generator, num_inference_steps=30, num_images_per_prompt=1)
```
:::

```{python sd35-plot}
#| echo: false
#| cache: true
fig, axes = plt.subplots(nrows=2, ncols=3, figsize=(10, 6), layout="constrained")

for i, ax in enumerate([ax for row in axes for ax in row]):
    ax.set_axis_off()
    p = ax.imshow(fit.images[i])
    
plt.show()
```


## LLM - Qwen2.5-3B

::: {.xsmall}
```{python}
#| eval: false
from transformers import pipeline

generator = pipeline('text-generation', model='Qwen/Qwen2.5-3B-Instruct')
prompt = "Can you write me a short bed time story about Thomas Bayes and his pet cat? Limit the story to no more than three paragraphs.\n\n"

result = generator(
    prompt, max_length=500, num_return_sequences=1,
    truncation=True
)

print( result[0]['generated_text'] )
```

```
Can you write me a short bed time story about Thomas Bayes and his pet cat? Limit the story to no more than three paragraphs.

In a quiet corner of a small, cozy house lived Thomas Bayes, a man known for his clever mind but often found lost in thought. His favorite place was a little study where he spent most of his days pondering the mysteries of probability. One day, as he sat by the window watching the clouds drift by, he noticed a curious feline peeking through the curtains. Intrigued, he let the cat inside, naming it Bayes after himself, much to the cat's delight. The cat, with its sleek black fur and piercing green eyes, became Bayes' constant companion. They would sit together on Bayes' lap, discussing theories over soft purrs, until the night grew dark and Bayes retired to his study to continue his work, always with Bayes curled up at his feet, a silent witness to his intellectual journey. As the moonlight filtered through the windows, Bayes would drift off to sleep, dreaming of the future of statistics, with Bayes, his loyal and curious friend, by his side.
```
:::

::: {.aside}
Qwen is an opensource LLM built by Alibaba
:::