<jupyter_start><jupyter_text>Choix d'une décennie et du nombre de clusters<jupyter_code>DECADE = '1890'
N_CLUSTERS = 10<jupyter_output><empty_output><jupyter_text>Chargement des fichiers de la décennie<jupyter_code>files = [f for f in sorted(os.listdir(data_path)) if f"_{DECADE[:-1]}" in f]
# Exemple de fichiers
files
texts = [open(data_path + f).read() for f in files]
# Exemple de textes
texts[0][:400]<jupyter_output><empty_output><jupyter_text>Vectorisation du texte<jupyter_code>def process_text(text, stem=True):
    """ Tokenize text and remove punctuation """
    text = text.translate(string.punctuation)
    tokens = word_tokenize(text)
    return tokens
vectorizer = TfidfVectorizer(tokenizer=process_text,
                                stop_words=stopwords.words('french'),
                                max_df=0.5,
                                min_df=0.1,
                                lowercase=True)
%time tfidf_vectors = vectorizer.fit_transform(texts)
tfidf_vectors
# Exemple de vecteur TFIDF
pd.Series(
    tfidf_vectors[0].toarray()[0],
    index=vectorizer.get_feature_names()
    ).sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>Comprendre les vecteurs et leurs "distances"<jupyter_code>from scipy.spatial.distance import cosine
cosine([1, 2, 3], [1, 2, 3])
cosine([1, 2, 3], [1, 2, 2])
cosine([1, 2, 3], [2, 2, 2])
tfidf_array = tfidf_vectors.toarray()
tfidf_array[0]
texts[0][:200]
tfidf_array[0]
cosine(tfidf_array[0], tfidf_array[1])<jupyter_output><empty_output><jupyter_text>Clustering des vecteurs TFIDF Article intéressant sur le KMeans clustering :- https://medium.com/dataseries/k-means-clustering-explained-visually-in-5-minutes-b900cc69d175<jupyter_code>km_model = KMeans(n_clusters=N_CLUSTERS)
km_model.fit(tfidf_vectors)
clustering = collections.defaultdict(list)

for idx, label in enumerate(km_model.labels_):
    clustering[label].append(files[idx])
pprint(dict(clustering))<jupyter_output>{0: ['Bxl_1890_Tome_I2_Part_13.txt',
     'Bxl_1892_Tome_I1_Part_8.txt',
     'Bxl_1893_Tome_I1_Part_5.txt',
     'Bxl_1894_Tome_I1_Part_3.txt',
     'Bxl_1895_Tome_I1_Part_5.txt',
     'Bxl_1896_Tome_I1_2_Part_4.txt',
     'Bxl_1897_Tome_I2_Part_2.txt',
     'Bxl_1897_Tome_I2_Part_3.txt',
     'Bxl_1898_Tome_I2_Part_2.txt',
     'Bxl_1899_Tome_I1_Part_9.txt'],
 1: ['Bxl_1896_Tome_I1_1_Part_1.txt',
     'Bxl_1896_Tome_I1_1_Part_2.txt',
     'Bxl_1896_Tome_I1_1_Part_3.txt',
     'Bxl_1896_Tome_I1_1_Part_4.txt',
     'Bxl_1896_Tome_I1_1_Part_5.txt',
     'Bxl_1896_Tome_I1_2_Part_1.txt',
     'Bxl_1896_Tome_I1_2_Part_2.txt',
     'Bxl_1896_Tome_I1_2_Part_3.txt',
     'Bxl_1896_Tome_I1_2_Part_5.txt',
     'Bxl_1896_Tome_I1_2_Part_6.txt',
     'Bxl_1896_Tome_I2_1_Part_1.txt',
     'Bxl_1896_Tome_I2_1_Part_2.txt',
     'Bxl_1896_Tome_I2_1_Part_3.txt',
     'Bxl_1896_Tome_I2_1_Part_4.txt',
     'Bxl_1896_Tome_I2_1_Part_5.txt',
     'Bxl_1896_Tome_I2_1_Part_6.txt',
     'Bxl_1896_Tome_I2_1_Par[...]
StarcoderdataJupyterStructuredCleanDedup