<jupyter_start><jupyter_text>Splitting dataIn the previous [notebook](https://github.com/hundredblocks/ml-powered-applications/blob/master/notebooks/dataset_exploration.ipynb), we explored a dataset. Next, we will separate it into a training and test split. Separating a dataset into splits is crucial to validate the performance of a model. By using only a subset of data to train a model, you can use unseen data to produce an estimate of how your model would perform in practice.In this notebook, I will demonstrate a few ways to do just that using the `writers` Stack Overflow dataset. First, we load and format data.<jupyter_code>import pandas as pd
import spacy
import umap
import numpy as np 
from pathlib import Path
import sys
sys.path.append("..")
import warnings
warnings.filterwarnings('ignore')
from ml_editor.data_processing import format_raw_df, get_random_train_test_split, get_vectorized_inputs_and_label, get_split_by_author

data_path = Path('../data/writers.csv')
df = pd.read_csv(data_path)
df = format_raw_df(df.copy())<jupyter_output><empty_output><jupyter_text>Random SplitThe simplest way to generate a test set is to randomly split data between a training set and a test set. This is what we do below.<jupyter_code>train_df_rand, test_df_rand = get_random_train_test_split(df[df["is_question"]], test_size=0.3, random_state=40)
print("%s questions in training, %s in test." % (len(train_df_rand),len(test_df_rand)))
train_owners = set(train_df_rand['OwnerUserId'].values)
test_owners = set(test_df_rand['OwnerUserId'].values)
print("%s different owners in the training set" % len(train_df_rand))
print("%s different owners in the testing set" % len(test_df_rand))
print("%s owners appear in both sets" % len(train_owners.intersection(test_owners)))<jupyter_output>5579 questions in training, 2392 in test.
5579 different owners in the training set
2392 different owners in the testing set
596 owners appear in both sets<jupyter_text>This approach comes with one drawback, can you guess what it is before moving on to the next section  Author SplitSome authors may be more skilled at asking questions then others. If an author appears in both the training and the test set, a model could successfully predict the performance of their questions simply by successfully identifying the author. Note that simply removing the `AuthorId` from the set of features does not fully solve this problem, as the formulation of a question may be author specific (especially if some authors include their signature). To make sure we are accurately judging question quality, we would want to make sure that a given author only appears in either the training set or the validation set. This guarantee that a model will not be able to leverage information to identify a given author and use it to predict more easily.To remove this potential source of bias, let's split data by author.<jupyter_code>train_author, test_author = get_split_by_author(df[df["is_question"]], test_size=0.3, random_state=40)

print("%s questions in training, %s in test." % (len(train_author),len(test_author)))
train_owners = set(train_author['OwnerUserId'].values)
test_owners = set(test_author['OwnerUserId'].values)
print("%s different owners in the training set" % len(train_owners))
print("%s different owners in the testing set" % len(test_owners))
print("%s owners appear in both sets" % len(train_owners.intersection(test_owners)))<jupyter_output>5676 questions in training, 2295 in test.
2723 different owners in the training set
1167 different owners in the testing set
0 owners appear in both sets
StarcoderdataJupyterStructuredCleanDedup