<jupyter_start><jupyter_text>Cloud and sea tile classifierIt has been identified that ESRGAN in both pretrain PSNR mode and GAN mode struggles with super-resoluting satellite image tiles completely covered by either sea surface or opaque clouds. In addition, both areas of interest, the harbors of Toulon and La Spezia, has lots of sea surface (approaching 50%). Left without intervention around 50% of tiles will only consist of sea and opaque clouds. It is assumed that this leads to an unwanted imbalance in what we want the model to be optimized to perform on.There are several ways to mitigate this imbalance. One way would be to manually draw a sea surface polygon in a GIS software and undersample tiles extracted from within this polygon. A downside of this approach is that interesting features (ships) within the sea surface polygon would also be undersampled.Another approach would be to train a cloud and sea tile classifier to detect the the unwanted tiles and discard all or a significant proportion of these tiles before training. This approach has the benefit of addressing the problem head on. The main downside of the approach is that it might be time-consuming to label tiles. However it is hypothesized that relatively little training data is needed to train a modern neural net classifier on such a *simple* classification task.<jupyter_code>import pickle
import geopandas
import pandas as pd
import pathlib
import datetime
import rasterio
import rasterio.plot
import tensorflow as tf

from modules.tile_generator import *
from modules.helpers import *
from modules.image_utils import *
from modules.tile_input_pipeline import *
from modules.cloudsea_classifier import *

# Check GPUs:",
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
    try:
        for gpu in gpus:
            # Prevent TensorFlow from allocating all memory of all GPUs:
            tf.config.experimental.set_memory_growth(gpu, True)
        logical_gpus = tf.config.experimental.list_logical_devices('GPU')
        print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
    except RuntimeError as e:
        print(e)
tf.sysconfig.get_build_info()
# Toggles whether to actually do the generation on this run
# Be careful with setting these to True if tiles are already labelled!
# New tiles will overwrite old tiles and labels are trash
GENERATE_NEW_TILES = False
CONVERT_TO_PNG = False
CREATE_LABEL_CSV = False

with open('metadata_df.pickle', 'rb') as file:
    meta = pickle.load(file)
# Path to location where individual satellite images are located
DATA_PATH = 'data/toulon-laspezia/'
DATA_PATH_TILES = 'data/toulon-laspezia-cloud-sea-classifier/'

SENSORS = ['WV02', 'GE01', 'WV03_VNIR']
AREAS = ['La_Spezia', 'Toulon']
meta = meta.loc[meta['sensorVehicle'].isin(SENSORS)]
meta = meta.loc[meta['area_name'].isin(AREAS)]

N_IMAGES = len(meta.index)

#96x96, 128x128, 196x196, 384x384 -- All tiles are squares
TILE_SIZES = [96, 128, 196, 384]
# number of tiles to generate at each tile size
N_TILES = {96: 500, 128: 1000, 196: 500, 384: 500}
N_TILES_TOTAL = sum(N_TILES.values())

print(N_IMAGES)
print(N_TILES)
print(N_TILES_TOTAL)

TRAIN_TILE_SIZE = 224

PAN_OR_MS_OR_BOTH = 'pan'
if PAN_OR_MS_OR_BOTH == 'pan':
    TRAIN_TILE_BANDS = 1
elif PAN_OR_MS_OR_BOTH == 'ms':
    TRAIN_TILE_BANDS = 4
elif PAN_OR_MS_OR_BOTH == 'both':
    TRAIN_TILE_BANDS = 5

RESIZE_METHOD = 'bilinear'

BATCH_SIZE = 16

VAL_SPLIT = 0.3<jupyter_output>73
{96: 500, 128: 1000, 196: 500, 384: 500}
2500<jupyter_text>Allocate n_tiles to every image (weighted by size of image)<jupyter_code>if GENERATE_NEW_TILES:
    meta = allocate_tiles(meta, by_partition=False, n_tiles_total=N_TILES[96], new_column_name='n_tiles_96')
    meta = allocate_tiles(meta, by_partition=False, n_tiles_total=N_TILES[128], new_column_name='n_tiles_128')
    meta = allocate_tiles(meta, by_partition=False, n_tiles_total=N_TILES[196], new_column_name='n_tiles_196')
    meta = allocate_tiles(meta, by_partition=False, n_tiles_total=N_TILES[384], new_column_name='n_tiles_384')
    meta<jupyter_output><empty_output><jupyter_text>Generate tiles to disk<jupyter_code>if GENERATE_NEW_TILES:
    meta['n_tiles'] = 0
    for tile_size in TILE_SIZES:
        pathlib.Path(DATA_PATH_TILES).joinpath(str(tile_size)).mkdir()
        tile_size_ms = int(tile_size/4)
        meta['n_tiles'] = meta[str('n_tiles_'+str(tile_size))]
        generate_all_tiles(meta, save_dir = str(DATA_PATH_TILES+'/'+str(tile_size)), 
                           ms_height_width=(tile_size_ms,tile_size_ms), sr_factor=4, 
                           cloud_sea_removal=False)<jupyter_output><empty_output><jupyter_text>Flatten the directory structure after generationLots of foor loops in order to do one change at a time.<jupyter_code>if GENERATE_NEW_TILES:
    # Remove train/val/test directories
    for tilesize_dir in pathlib.Path(DATA_PATH_TILES).iterdir():
        for partition_dir in tilesize_dir.iterdir():
            for image_dir in partition_dir.iterdir():
                dest = tilesize_dir.joinpath(image_dir.stem)
                source = image_dir
                source.rename(dest)
            partition_dir.rmdir()

    # Add tile size to filenames
    for tilesize_dir in pathlib.Path(DATA_PATH_TILES).iterdir():
        for image_dir in tilesize_dir.iterdir():
            for ms_pan_dir in image_dir.iterdir():
                for tile in ms_pan_dir.iterdir():
                    new_tile_name = str(tilesize_dir.stem+'-'+tile.name)
                    new_path = ms_pan_dir.joinpath(new_tile_name)
                    tile.rename(new_path)

    # Completely flatten file structure, remove tile size directories
    for tilesize_dir in pathlib.Path(DATA_PATH_TILES).iterdir():
        for image_dir in tilesize_dir.iterdir():
            for ms_pan_dir in image_dir.iterdir():
                for tile in ms_pan_dir.iterdir():
                    new_dir = pathlib.Path(DATA_PATH_TILES).joinpath(image_dir.stem, ms_pan_dir.name)
                    new_dir.mkdir(parents=True, exist_ok=True)
                    new_path = new_dir.joinpath(tile.name)
                    tile.rename(new_path)
                ms_pan_dir.rmdir()
            image_dir.rmdir()
        tilesize_dir.rmdir()

    # Add image_int_uid to filenames and flatten structure completely
    for image_dir in pathlib.Path(DATA_PATH_TILES).iterdir():
        if image_dir.stem == 'ms' or image_dir.stem == 'pan':
            continue
        for ms_pan_dir in image_dir.iterdir():
            for tile in ms_pan_dir.iterdir():
                int_uid = get_int_uid(meta, image_dir.stem)
                new_tile_name = str(str(int_uid).zfill(2)+'-'+tile.name)
                new_dir = pathlib.Path(DATA_PATH_TILES).joinpath(ms_pan_dir.stem)
                new_dir.mkdir(parents=True, exist_ok=True)
                new_path = new_dir.joinpath(new_tile_name)
                tile.rename(new_path)
            ms_pan_dir.rmdir()
        image_dir.rmdir()

# List all tif files
tif_paths = [file for file in pathlib.Path(DATA_PATH_TILES).glob('**/*.tif')]
tif_paths_ms = tif_paths[:2500]
tif_paths_pan = tif_paths[2500:]

# Divide by 2 because each tile consists of 1 MS + 1 PAN
print('Number of tiles generated and present in flat file structure:', str(int(len(tif_paths)/2)))<jupyter_output>Number of tiles generated and present in flat file structure: 2500<jupyter_text>Convert to pngWhile the input to the actual cloud/sea classifier is tif files it is practical to also convert the image tiles to png. This makes labelling easier.<jupyter_code>if CONVERT_TO_PNG:
    for tif_path in tif_paths:
        ms_or_pan = tif_path.parent.stem
        
        # sensor type is needed for conversion of ms to rgb png::
        int_uid = int(tif_path.stem[:2])
        string_uid = get_string_uid(meta, int_uid)
        sensor = get_sensor(meta, string_uid)
        
        # saves png to disk
        geotiff_to_png(tif_path, ms_or_pan=ms_or_pan, scale=True, stretch_img=True, sensor=sensor)

# List all png files
png_paths = [file for file in pathlib.Path(DATA_PATH_TILES).glob('**/*.png')]

# Divide by 2 because each tile consists of 1 MS + 1 PAN
print('Number of tiles generated and present in flat file structure:', str(int(len(png_paths)/2)))<jupyter_output>Number of tiles generated and present in flat file structure: 2500<jupyter_text>Create label csv fileLabels are `None` before manual labelling<jupyter_code>if CREATE_LABEL_CSV:
    label_df = pd.DataFrame([tif_path.stem for tif_path in tif_paths[:N_TILES_TOTAL]], columns=['tile_uid'])
    label_df['cloud-sea'] = None
    label_df.to_csv(pathlib.Path(DATA_PATH_TILES).joinpath('labels-to-be.csv'), index=False)<jupyter_output><empty_output><jupyter_text>Labelling*... 4 tedious labelling hours later...*  Loading the labeled csvOur `load_and_populate_label_df` function also extracts `sensor`, `tile_size` and `img_uid` from the `tile_uid` column. This is used later when preparing data for training.<jupyter_code>label_df = load_and_populate_label_df(DATA_PATH_TILES + '/labels.csv', meta)
label_df<jupyter_output><empty_output><jupyter_text>Preprocessing of images and labelsImages in have different resolution. Classifier model architecture `EfficientNet` requires fixed image sizes as input so images are resized to `TRAIN_TILE_SIZE_PAN = 224`. [Native EfficientNet image sizes](https://keras.io/examples/vision/image_classification_efficientnet_fine_tuning/)Note that `EfficientNet` has rescaling and normalizing layers as their first layers so such preprocessing is not required in advance. In fact it seems like such preprocessing hurts performance of the model. We can therefore keep image tiles with `dtype=uint16`.<jupyter_code>X, y = prepare_for_training(label_df, tif_paths_pan, tif_paths_ms,
                            pan_or_ms_or_both=PAN_OR_MS_OR_BOTH,
                            pan_tile_size=TRAIN_TILE_SIZE, ms_tile_size=TRAIN_TILE_SIZE, 
                            resize_method=RESIZE_METHOD)<jupyter_output>Verification OK. All 2500 image tile UIDs match.
Loaded 0 images
Loaded 100 images
Loaded 200 images
Loaded 300 images
Loaded 400 images
Loaded 500 images
Loaded 600 images
Loaded 700 images
Loaded 800 images
Loaded 900 images
Loaded 1000 images
Loaded 1100 images
Loaded 1200 images
Loaded 1300 images
Loaded 1400 images
Loaded 1500 images
Loaded 1600 images
Loaded 1700 images
Loaded 1800 images
Loaded 1900 images
Loaded 2000 images
Loaded 2100 images
Loaded 2200 images
Loaded 2300 images
Loaded 2400 images
Finished preparing 2500 images and labels for training!
X.shape: (2500, 224, 224, 1)
y.shape: (2500,)<jupyter_text>Model<jupyter_code>model = build_model(augment=True, input_shape=(TRAIN_TILE_SIZE, TRAIN_TILE_SIZE, TRAIN_TILE_BANDS))
model.summary()
pretrain_model_name = str('cloud-sea-classifier-effnetb0-pan-augm')
log_dir = pathlib.Path(str('logs/cloud-sea-classifier/fit/' + pretrain_model_name 
                           + datetime.datetime.now().strftime('%Y%m%d-%H%M%S')))
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, 
                                                      histogram_freq=10, 
                                                      write_graph=False, 
                                                      write_images=False,
                                                      update_freq='epoch')

checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(
    filepath = str('models/cloud-sea-classifier/' + pretrain_model_name + '-{epoch:02d}-{val_loss:.6f}.h5'), 
    monitor = "val_acc",
    save_best_only = False,
    save_weights_only = True,
    )
EPOCHS = 100
model.fit(X, y, batch_size=BATCH_SIZE, epochs=EPOCHS,
          validation_split=VAL_SPLIT, initial_epoch=0,
          callbacks=[checkpoint_callback, tensorboard_callback])<jupyter_output><empty_output><jupyter_text>Hyperparameter tuning<jupyter_code>RESIZE_METHODS = ['bilinear', 'nearest', 'bicubic']
PAN_MS_BOTH = ['pan', 'ms', 'both']
LEARNING_RATES = [0.001, 0.0005, 0.0001]

EPOCHS = 200

for resize_method in RESIZE_METHODS:
    for pan_ms_both in PAN_MS_BOTH:
        if pan_ms_both == 'pan':
            train_tile_bands = 1
        elif pan_ms_both == 'ms':
            train_tile_bands = 4
        elif pan_ms_both == 'both':
            train_tile_bands = 5
        for learning_rate in LEARNING_RATES:
                
            X, y = prepare_for_training(label_df, tif_paths_pan, tif_paths_ms,
                                        pan_or_ms_or_both=pan_ms_both,
                                        pan_tile_size=TRAIN_TILE_SIZE, ms_tile_size=TRAIN_TILE_SIZE, 
                                        resize_method=resize_method)

            model = build_model(augment=True, 
                                input_shape=(TRAIN_TILE_SIZE, TRAIN_TILE_SIZE, train_tile_bands), 
                                learning_rate=learning_rate)

            pretrain_model_name = str('cloudsea-effb0-augm-' + resize_method 
                                      + '-' + pan_ms_both + '-' + str(learning_rate) + '-')
            log_dir = pathlib.Path(str('logs/cloud-sea-classifier/fit/' + pretrain_model_name 
                                       + datetime.datetime.now().strftime('%Y%m%d-%H%M%S')))
            tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, 
                                                                  histogram_freq=10, 
                                                                  write_graph=False, 
                                                                  write_images=False,
                                                                  update_freq='epoch')

            checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(
                filepath = str('models/cloud-sea-classifier/' + pretrain_model_name + '-{epoch:02d}-{val_loss:.6f}.h5'), 
                monitor = "val_acc",
                save_best_only = False,
                save_weights_only = True,
                )

            model.fit(X, y, batch_size=BATCH_SIZE, epochs=EPOCHS,
                      validation_split=VAL_SPLIT, initial_epoch=0,
                      callbacks=[checkpoint_callback, tensorboard_callback])
for<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup