<jupyter_start><jupyter_text>Dataset Class for the Cub datasetThe data we will work on should be in the `preprocessing` directory, and should include the following:1. `./images` directory, with all bird images divided into the 200 categories2. `./text` directory, with all the bird images captions divided into the 200 categories3. `./images.csv` - csv file containing the path, image_id, class id, and train/test affiliation of each image4. `./classes.csv` - csv file containing a mapping between the class id and the class name<jupyter_code>import os

import pandas as pd
from skimage import io
from PIL import Image
import pickle
import torch
import torchvision
from torch.utils.data import Dataset, DataLoader
from torchvision import transforms
import sys
sys.path.append("../")
from dataset.birds_dataset import BirdsDataset
base_dataset_path = "/home/user_2/AttnGAN/datasets"

dataset_interim_path = os.path.join(base_dataset_path, "interim")
dataset_preprocess_path = os.path.join(base_dataset_path, "preprocessing")<jupyter_output><empty_output><jupyter_text>Organizing unzipped filesAll unzipped files are in the `interim` data folder and should be organized and moved to the `preprocessing` folder.Follow the following 4 steps to get the right order of the `preprocessing folder` 0 Unzip the `bird.zip` and `CUB_200_2011.tgz` to the `interim` directory and put everything in the base directory if you havn't already done so. 1Move manually the `./images` with all the images folder to `preprocessing` folder 2Move manually the `./text` folder with all the text captions for each image to the `preprocessing` folder  3Saving the images as .csv file<jupyter_code>images_df = pd.read_csv(os.path.join(dataset_interim_path, "images.txt"), sep=" ", names=["image_id", "path"])
images_df.head()
# Create another column to hold the class id
images_df["class_id"] = images_df.apply(lambda p : int(p.path[:3]), axis=1)

# remove the .jpg from the end of the path
images_df["path"] = images_df.apply(lambda p : p.path[:-4], axis=1)
# Create another column to hold the train/test affiliation of the image
# 1 - The image is in the training set.
# 0 - the image is in the test set.
train_test_split = pd.read_csv(os.path.join(dataset_interim_path, "train_test_split.txt"), sep=" ", names=["img_id", "is_train"])
images_df["is_train"] = train_test_split.is_train
images_df.head()
# Save the final unified csv file
images_df.to_csv(os.path.join(dataset_preprocess_path, "images.csv"))<jupyter_output><empty_output><jupyter_text>4Saving the class name to class index mapping as csv file<jupyter_code>classes_df = pd.read_csv(os.path.join(dataset_interim_path, "classes.txt"), sep=" ", names=["class_id", "class_name"])
classes_df.to_csv(os.path.join(dataset_preprocess_path, "classes.csv"))<jupyter_output><empty_output><jupyter_text>Some more metadata- [ ] What is it used for?<jupyter_code>with open (os.path.join(dataset_interim_path, "train", "filenames.pickle"), "rb") as f:
    train_filenames = pickle.load(f)

with open (os.path.join(dataset_interim_path, "train", "class_info.pickle"), "rb") as f1:
    train_class_info = pickle.load(f1, encoding="latin1")

with open (os.path.join(dataset_interim_path, "test", "filenames.pickle"), "rb") as f:
    test_filenames = pickle.load(f)

with open (os.path.join(dataset_interim_path, "test", "class_info.pickle"), "rb") as f:
    test_class_info = pickle.load(f)
# To use a pre-trained model:
# with open(os.path.join(dataset_interim_path, "captions.pickle"), "rb") as f:
#     captions = pickle.load(f)<jupyter_output><empty_output><jupyter_text>Dataset class to help load the dataBase path should be after the initial one-time ordering of the data folder.  Ordered data that should be taken by this class should be in the `preprocessing` folder.<jupyter_code>classes_df = pd.read_csv(os.path.join(dataset_preprocess_path, "classes.csv")).drop("Unnamed: 0", axis=1)
classes_df.head()
images_df = pd.read_csv(os.path.join(dataset_preprocess_path, "images.csv")).drop("Unnamed: 0", axis=1)
images_df.head()
total_sampels = len(images_df)
num_train = images_df.is_train.sum()
num_test = total_sampels - num_train

print("There are {:,} training samples, and {:,} test samples".format(num_train, num_test))
5994  + 5794
train_df = images_df[images_df.is_train == 1].drop("is_train", axis=1)
train_df.head()
image_transforms = transforms.Compose([transforms.Resize((224,224)), transforms.ToTensor()])
bird_dataset = BirdsDataset(dataset_preprocess_path, split="train", image_transform=image_transforms)
bird_dataset.vocabulary.itos[3]
dataloader = DataLoader(bird_dataset, shuffle=True, batch_size=4)
batch = next(iter(dataloader))
batch
print(batch["image"].shape)
print(batch["image_id"])
print(batch["class_id"])
print(batch["caption"][0])<jupyter_output>torch.Size([4, 3, 224, 224])
tensor([10350,  2535,   790,  8970])
tensor([176,  45,  15, 153])
tensor([  1,  10, 213,   6,  13,  83,   7,   9,   5,  18, 164,   5,   8,   4,
         30,  14,   2,   2,   2,   2])<jupyter_text>Clean the bad text sentencesRun all these to clean bad lines when done creatign the preprocessing directory.  similary, can just load the bad_files.pkl.gz and use it with the last cell<jupyter_code>import os
import re
import joblib 
import sys
sys.path.insert(0, "../../")
from dataset.preprocessing import clean_text_data
collapse_unknown_chars_regex = re.compile(r'[��]+')
# collapse_unknown_chars_regex.findall("this��bird��has��a��yellow-green��belly,��a��black��crown��and��a��short,��pointy��bill.")
base_path = "/home/user_2/AttnGAN/datasets/cub200-2011/preprocessing/text"
collapse_unknown_chars_regex = re.compile(r'[��]+')
bad_files = []
for dirname in os.listdir(base_path):
    for filename in os.listdir(os.path.join(base_path, dirname)):
        full_filepath = os.path.join(base_path, dirname, filename)
        with open(full_filepath, 'r') as f:
            if any([collapse_unknown_chars_regex.findall(line) for line in f.readlines()]):
                bad_files.append(full_filepath)
joblib.dump(bad_files, "./list_of_bad_filenames.pkl.gz")
len(bad_files)
collapse_unknown_chars_regex = re.compile(r'[��]+')
for bad_file in bad_files:
    new_lines = []
    with open(bad_file, "r") as f:
        bad_lines = f.readlines()
        for bad_line in bad_lines:
            new_lines.append(collapse_unknown_chars_regex.sub(' ', bad_line))
    os.remove(bad_file)
    with open(bad_file, "w") as f2:
        for good_line in new_lines:
            f2.write(good_line)
for file in bad_files:
    print(file)<jupyter_output>/home/user_2/AttnGAN/datasets/cub200-2011/preprocessing/text/177.Prothonotary_Warbler/Prothonotary_Warbler_0083_173929.txt
/home/user_2/AttnGAN/datasets/cub200-2011/preprocessing/text/177.Prothonotary_Warbler/Prothonotary_Warbler_0106_174221.txt
/home/user_2/AttnGAN/datasets/cub200-2011/preprocessing/text/055.Evening_Grosbeak/Evening_Grosbeak_0022_37761.txt
/home/user_2/AttnGAN/datasets/cub200-2011/preprocessing/text/067.Anna_Hummingbird/Anna_Hummingbird_0070_56085.txt
/home/user_2/AttnGAN/datasets/cub200-2011/preprocessing/text/038.Great_Crested_Flycatcher/Great_Crested_Flycatcher_0016_29406.txt
/home/user_2/AttnGAN/datasets/cub200-2011/preprocessing/text/187.American_Three_toed_Woodpecker/American_Three_Toed_Woodpecker_0014_179882.txt
/home/user_2/AttnGAN/datasets/cub200-2011/preprocessing/text/009.Brewer_Blackbird/Brewer_Blackbird_0079_2343.txt
/home/user_2/AttnGAN/datasets/cub200-2011/preprocessing/text/009.Brewer_Blackbird/Brewer_Blackbird_0115_2279.txt
/home/user_2/AttnGAN/datase[...]<jupyter_text>Unrelated<jupyter_code>t = torch.randn([2,10])
t.shape[0]
torch.cat?
t.numpy()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup