<jupyter_start><jupyter_text>Import git repo and base network<jupyter_code>%cd /content/
!git clone https://github.com/osmandemirel/knowledge-distillation-pytorch
%cd /content/
# base_resnet-18
# https://stanford.app.box.com/s/5lwrieh9g1upju0iz9ru93m9d7uo3sox
# This link should refresh from above link because its a dynamic download link so this one will be expire.
!wget https://public.boxcloud.com/d/1/b1!kzbOvtVd0rs9h_-9m8ppzhTLw_GWSSJHkHVVsbdVn0jljDNv1Lb2v4YXC51eD9c77gS13aDgBDApGv95ZICe5nRXE-OzwVybYxXD3CxZw3zb-au0ew9qi9Mr1iPjPdm4ED4ColPM4Z0H6lBf3uOI5aEgxc-Axm0F0G_O0kCM6SJnmfuHuN6Rrocu7vYIUDNkmp9Xmvs2_ognyWki26TpU4PPPLmjTIGqRI0sjQeZOknzAstuSW1VUPnCC_s7x4EvprQTrgT5dFefT2deRH2cyzbpEbHhAXi7wpVM8W8jvbMMTZYNIWSzyA9jOocOOoBIdUpugLzkQDQD2GUjAVu6dzZKvYqCTPO3a6sQLuPRgsmRlBX2oW2E7zbecgNSX8seoavGIZN03gOsjQL1UJEaaBtorVMd_hbvGSRoDRBKDlf5sWA8EwudgJSALfcgOyumRNUFFwDqXGRTBcXJnpIS5CPUI9HfMBU37eBEsFHDWOuhtSmAPPQzI39itiFMtIL07dMfYx53OTU4CS5VHYxo4whlLrCX9pk-5Pel8q1B9sc6n7DiEUS<KEY>ZF9LPkmnM-yuKrcqcNxGF02O3-iTrQ../download
# unzip downloaded model
!unzip /content/download 
# clear base_resnet18 experiments
!rm -rf /content/knowledge-distillation-pytorch/experiments/base_resnet18/*

# copy download model to repo
!cp -r /content/pretrained_teacher_models/base_resnet18 /content/knowledge-distillation-pytorch/experiments/<jupyter_output><empty_output><jupyter_text>Train base cnn **With params 1**<jupyter_code>import json
base_config_path = "/content/knowledge-distillation-pytorch/experiments/base_cnn/params.json"

base_cnn_params = {
    "params_1":{
    "model_version": "cnn",
    "subset_percent": 1.0,
    "augmentation": "no",
    "teacher": "none",
    "alpha": 0,
    "temperature": 1,
    "learning_rate": 1e-3,
    "batch_size": 128,
    "num_epochs": 30,
    "dropout_rate": 0.5, 
    "num_channels": 32,
    "save_summary_steps": 100,
    "num_workers": 8},

    "params_2":{
      "model_version": "cnn",
      "subset_percent": 1.0,
      "augmentation": "yes",
      "teacher": "none",
      "alpha": 0,
      "temperature": 1,
      "learning_rate": 3e-4,
      "batch_size": 64,
      "num_epochs": 30,
      "dropout_rate": 0.35, 
      "num_channels": 32,
      "save_summary_steps": 100,
      "num_workers": 8}
}
with open(base_config_path, "w")as f:
    json.dump(base_cnn_params["params_1"], f)
%cd /content/knowledge-distillation-pytorch
!python train.py --model_dir /content/knowledge-distillation-pytorch/experiments/base_cnn<jupyter_output>/content/knowledge-distillation-pytorch
2021-06-08 11:14:06.811236: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
Loading the datasets...
Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ./data-cifar10/cifar-10-python.tar.gz
170499072it [00:02, 78147122.12it/s]                   
Extracting ./data-cifar10/cifar-10-python.tar.gz to ./data-cifar10
Files already downloaded and verified
/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py:477: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  cpuset_checked))
Files already downloaded and verified
Files already downloaded a[...]<jupyter_text>**With params 2**<jupyter_code>with open(base_config_path, "w")as f:
    json.dump(base_cnn_params["params_2"], f)
%cd /content/knowledge-distillation-pytorch
!python train.py --model_dir /content/knowledge-distillation-pytorch/experiments/base_cnn<jupyter_output>/content/knowledge-distillation-pytorch
2021-06-07 18:01:25.107537: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
Loading the datasets...
Files already downloaded and verified
Files already downloaded and verified
/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py:477: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  cpuset_checked))
Files already downloaded and verified
Files already downloaded and verified
- done.
Starting training for 30 epoch(s)
Training Fold 1 model..
/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of `lr_scheduler.[...]<jupyter_text>Train cnn distil **With params 1**<jupyter_code>import json
base_config_path = "/content/knowledge-distillation-pytorch/experiments/cnn_distill/params.json"

cnn_distil_params= {
    "params_1":{
      "model_version": "cnn_distill",
      "subset_percent": 1.0,
      "augmentation": "no",
      "teacher": "resnet18",
      "alpha": 0.9,
      "temperature": 20,
      "learning_rate": 1e-3,
      "batch_size": 128,
      "num_epochs": 30,
      "dropout_rate": 0.5, 
      "num_channels": 32,
      "save_summary_steps": 100,
      "num_workers": 8},
    "params_2":{
      "model_version": "cnn_distill",
      "subset_percent": 1.0,
      "augmentation": "yes",
      "teacher": "resnet18",
      "alpha": 0.95,
      "temperature": 10,
      "learning_rate": 3e-4,
      "batch_size": 64,
      "num_epochs": 30,
      "dropout_rate": 0.35, 
      "num_channels": 32,
      "save_summary_steps": 100,
      "num_workers": 8}
}
with open(base_config_path, "w")as f:
    json.dump(cnn_distil_params["params_1"], f)
%cd /content/knowledge-distillation-pytorch
!python train.py --model_dir /content/knowledge-distillation-pytorch/experiments/cnn_distill<jupyter_output>/content/knowledge-distillation-pytorch
2021-06-08 11:39:57.596564: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
Loading the datasets...
Files already downloaded and verified
Files already downloaded and verified
/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py:477: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  cpuset_checked))
Files already downloaded and verified
Files already downloaded and verified
- done.
Experiment - model version: cnn_distill
Starting training for 30 epoch(s)
First, loading the teacher model and computing its outputs...
Training Fold 1 model..
/usr/local/lib[...]<jupyter_text>**With params 2**<jupyter_code>import json
base_config_path = "/content/knowledge-distillation-pytorch/experiments/cnn_distill/params.json"

with open(base_config_path, "w")as f:
    json.dump(cnn_distil_params["params_2"], f)
%cd /content/knowledge-distillation-pytorch
!python train.py --model_dir /content/knowledge-distillation-pytorch/experiments/cnn_distill<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup