<jupyter_start><jupyter_text>Transform Experimentscustom transforms<jupyter_code>%reload_ext autoreload
%autoreload 2
%matplotlib inline
from PIL import Image as pil_image
#pip3 install nvidia-ml-py3
import tracemalloc, threading, torch, time, pynvml
from fastai.utils.mem import *
from fastai.vision import *
import fastai
print(fastai.__version__)
torch.cuda.set_device(0)
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
#see https://forums.fast.ai/t/gpu-optimizations-central/33944/15
#"memory profiler that taps into each epoch, and can be fine-tuned to each separate stage"
if not torch.cuda.is_available(): raise Exception("pytorch is required")

def preload_pytorch():
    torch.ones((1, 1)).cuda()
    
def gpu_mem_get_used_no_cache():
    torch.cuda.empty_cache()
    return gpu_mem_get().used

def gpu_mem_used_get_fast(gpu_handle):
    info = pynvml.nvmlDeviceGetMemoryInfo(gpu_handle)
    return int(info.used/2**20)

preload_pytorch()
pynvml.nvmlInit()

class PeakMemMetric(LearnerCallback):
    _order=-20 # Needs to run before the recorder

    def peak_monitor_start(self):
        self.peak_monitoring = True
path = Path('../data/mnist/mnist_png')
np.random.seed(42)<jupyter_output><empty_output><jupyter_text>No transforms<jupyter_code>#no transforms
tfms = get_transforms(do_flip=False, 
                      flip_vert=False, 
                      max_rotate=0., 
                      max_zoom=0., 
                      max_lighting=0., 
                      max_warp=0., 
                      p_affine=0., 
                      p_lighting=0.)

data = ImageDataBunch.from_folder(path, valid_pct = 0.2,ds_tfms=tfms, size=28)
data.show_batch(rows=3, figsize=(5,5))<jupyter_output><empty_output><jupyter_text>fit<jupyter_code>learn = create_cnn(data, models.resnet18, metrics=error_rate)<jupyter_output><empty_output><jupyter_text>Defaults for lr_find:    lr_find(learn:Learner, start_lr:Floats=1e-7, end_lr:Floats=10, num_it:int=100, stop_div:bool=True, **kwargs:Any):NB in lr_find we call Learner.fit()<jupyter_code>learn.lr_find()
learn.recorder.plot(skip_start=10, skip_end=5)<jupyter_output>LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.<jupyter_text>fit_one_cycle()*You can put a single number (e.g. 1e-3): Every layer gets the same learning rate. So you're not using discriminative learning rates.**You can write a slice. So you can write slice with a single number (e.g. slice(1e-3)): The final layers get a learning rate of whatever you wrote down (1e-3), and then all the other layers get the same learning rate which is that divided by 3. So all of the other layers will be 1e-3/3. The last layers will be 1e-3.**You can write slice with two numbers (e.g. slice(1e-5, 1e-3)). The final layers (these randomly added layers) will still be again 1e-3. The first layers will get 1e-5, and the other layers will get learning rates that are equally spread between those two - so multiplicatively equal. If there were three layers, there would be 1e-5, 1e-4, 1e-3, so equal multiples each time.* (<NAME>, 2019, lesson5)  DefaultsIn defaults the default lr in fastai is:        slice(None, 3e-3, None)    This is the lr that is set *if lr_find() has not been called*:    lr: slice(None, 0.003, None)which is then set to (in this case there are 3 layer goups):        lr = np.array([lr.stop/10]*(len(self.layer_groups)-1) + [lr.stop])    and set for this callback        OptimWrapper._lr: [0.0003, 0.0003, 0.003]     Default weight decay:    wd = 0.01<jupyter_code>learn.fit(2)<jupyter_output><empty_output><jupyter_text>fit one cycleIn some parts of the course JH mentions - on initial fit - to use steepest part of curve eg for curve above lr = 3e-3learn.fit_one_cycle(5, slice(lr))See Lesson3 Python3 partial [39:17] https://github.com/hiromis/notes/blob/master/Lesson3.md JH: As a rule of thumb, for top learning, use 1e-4 or 3e-4eg in Lesson2 Step 4: Training a model [26:49] (see hiromi's excellent notes here: https://github.com/hiromis/notes/blob/master/Lesson2.md)In example figure below, Jeremy picked 3e-5 as bottom lr and 3e-4 at top learning rate in the slicelearn.fit_one_cycle(2, max_lr=slice(3e-5,3e-4)) ![title](figures/lesson2_step4_13.png)<jupyter_code>learn = create_cnn(data, models.resnet18, metrics=error_rate)
learn.lr_find()
learn.recorder.plot(skip_start=10, skip_end=5)<jupyter_output>LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.<jupyter_text>Default parameters for fit_one_cycle:fit_one_cycle(learn:Learner, cyc_len:int, max_lr:Union[Floats,slice]=defaults.lr,                  moms:Tuple[float,float]=(0.95,0.85), div_factor:float=25., pct_start:float=0.3,                  wd:float=None, callbacks:Optional[CallbackList]=None, **kwargs)                  In code below where we have start_lr of 3e-5 and stop_lr of 3e-4,the lr_range is distributed accoss the n layer_grougs.Here we have 3 layer goups, and the lr_range is set to:[3e-05 9.49e-05 3e-04]<jupyter_code>learn.fit_one_cycle(2, max_lr=slice(3e-5,3e-4))
learn.recorder.plot_losses()<jupyter_output><empty_output><jupyter_text>In above we see validation loss is lower than training loss:    Either:        learning rate is too small         number of epochs is too low        model is too simple<jupyter_code>learn = create_cnn(data, models.resnet18, metrics=error_rate)
learn.fit_one_cycle(2, max_lr=slice(3e-3,3e-2))
learn.recorder.plot_losses()<jupyter_output><empty_output><jupyter_text>Better - using higher lr  Unfreeze JH: take whatever did last time and divide it by 5 or 10    So if used 3e-3 for initial fit, then useslice(steepest slope or 10x smaller than bottom , lr/10)<jupyter_code>learn.unfreeze()

learn.lr_find()
learn.recorder.plot(skip_start=0, skip_end=5)
learn.fit_one_cycle(4, slice(2e-6, 6e-5))
learn.recorder.plot_lr(show_moms=True)
learn.recorder.plot_losses()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup