<jupyter_start><jupyter_text>Topic modeling without Theta distributionBased on unpublished (yet) research by <NAME>, <NAME>, <NAME>One of the interpretations of topic modeling is this:\begin{equation}L(\Phi, \Theta) + R(\Phi, \Theta) \to \max_{\Phi, \Theta},\end{equation}Once the inference is directly formulated as an optimization problem, one can ask: is it possible to reduce the number f inferred parameters? In the practice, word-topic matrix $(\Phi)$ has a larger importance than document-topic matrix $(\Theta)$; so we can replace the original optimization problem with a different one:\begin{equation}L(\Phi, f(\Phi) ) + R(\Phi, f(\Phi) ) \to \max_{\Phi},\end{equation}Some math later, it turns out that the "formally correct" optimization algorithm for this problem is different from conventional EM algorithm, and has some surprising advantages. Technically speaking, these formulas require a different iteration process (which is implemented here: https://github.com/ilirhin/python_artm), but the formulas look similar enough to a custom $\Phi$ regularizer (if one sets `num_document_passes=1` in BigARTM)This notebook serves as a proof-of-concept for this idea.<jupyter_code>import sys
sys.path.insert(0, '../..')
import numpy as np
import pandas as pd
import requests
import scipy.sparse

from numba import jit
from numpy import random
from matplotlib import pyplot as plt

import artm

from topicnet.cooking_machine.dataset import Dataset
from topicnet.cooking_machine.models import (
    BaseScore as BaseTopicNetScore,
    TopicModel
)
from topicnet.cooking_machine.models.base_regularizer import BaseRegularizer
from topicnet.cooking_machine.models.topic_prior_regularizer import TopicPriorRegularizer
from topicnet.cooking_machine.models.thetaless_regularizer import (
    ThetalessRegularizer,
    dataset2sparse_matrix,
)

from topicnet.viewers.top_tokens_viewer import TopTokensViewer
from topicnet.viewers.top_documents_viewer import TopDocumentsViewer
%load_ext autoreload
%autoreload 2
dataset = Dataset(
    "/data/datasets/20_News_dataset/20NG_BOW.csv",
    internals_folder_path="./20NG_internals",
)

dataset.get_possible_modalities()
MAIN_MODALITY = "@word"<jupyter_output><empty_output><jupyter_text>Preparing the coherence score  Top-token coherence (co-occurrences are calculated using the same corpus)<jupyter_code>def calc_doc_occurrences(dataset, modality):
    """
    :param n_dw_matrix: sparse document-word matrix, shape is D x W
    :return: sparse matrix of co-occurrences

    doc_occurrences[w1, w2] = the number of the documents
    where there are w1 and w2
    """
    n_dw_matrix = dataset2sparse_matrix(dataset, modality, modalities_to_use=[modality])
    matrix = (scipy.sparse.csc_matrix(n_dw_matrix) > 0).astype(int)
    co_occurrences = matrix.T * matrix

    return co_occurrences.diagonal(), co_occurrences


def create_pmi_top_function(
    doc_occurrences, doc_co_occurrences,
    documents_number, top_sizes,
    co_occurrences_smooth=1.
):
    """
    :param doc_occurrences: array of doc occurrences of words
    :param doc_co_occurrences: sparse matrix of doc co-occurrences of words
    :param documents_number: number of the documents
    :param top_sizes: list of top values to calculate top-pmi for
    :param co_occurrences_smooth: constant to smooth co-occurrences in log
    :return: function which takes phi and theta and returns
    pair of two arrays: pmi-s of the tops and ppmi-s of the tops

    pmi[i] - pmi(top of size top_sizes[i])
    ppmi[i] - ppmi(top of size top_sizes[i])

    pmi(words) = sum_{u in words, v in words, u != v}
    log(
        (doc_co_occurrences[u, v] * documents_number + co_occurrences_smooth)
        / doc_occurrences[u] / doc_occurrences[v]
    )

    ppmi(words) = sum_{u in words, v in words, u != v}
    max(log(
        (doc_co_occurrences[u, v] * documents_number + co_occurrences_smooth)
        / doc_occurrences[u] / doc_occurrences[v]
    ), 0)

    """
    def func(phi):
        T, W = phi.shape
        max_top_size = max(top_sizes)
        pmi, ppmi = np.zeros(max_top_size), np.zeros(max_top_size)
        tops = np.argpartition(phi, -max_top_size, axis=1)[:, -max_top_size:]
        
        for t in range(T):
            top = sorted(tops[t], key=lambda w: - phi[t, w])
            co_occurrences = doc_co_occurrences[top, :][:, top].todense()
            occurrences = doc_occurrences[top]
            values = np.log(
                (co_occurrences * documents_number + co_occurrences_smooth)
                / (occurrences[:, np.newaxis] * occurrences[np.newaxis, :] + co_occurrences_smooth)
            )
            diag = np.diag_indices(len(values))
            values.cumsum(axis=0).cumsum(axis=1)[diag] - values[diag].cumsum()
            pmi += np.array(
               values.cumsum(axis=0).cumsum(axis=1)[diag] - values[diag].cumsum()
            ).ravel()

            values[values < 0.] = 0.
            ppmi += np.array(
               values.cumsum(axis=0).cumsum(axis=1)[diag] - values[diag].cumsum()
            ).ravel()
            
        sizes = np.arange(2, max_top_size + 1)
        pmi[1:] /= (T * sizes * (sizes - 1))
        ppmi[1:] /= (T * sizes * (sizes - 1))
        indices = np.array(top_sizes) - 1
        
        return pmi[indices], ppmi[indices]

    return func
%%time

occurences, co_occurences = calc_doc_occurrences(dataset, MAIN_MODALITY)

calc_pmi = create_pmi_top_function(
    occurences, co_occurences,
    dataset.get_dataset().shape[0], [20],
    co_occurrences_smooth=1e-2,
)
class TopTokenCoherence(BaseTopicNetScore):
    def __init__(self, func):
        super().__init__()

        self.calc_pmi = func

    def call(self, model: TopicModel):
        values = self.calc_pmi(model.get_phi_dense()[0].T)

        return values[1]<jupyter_output><empty_output><jupyter_text>Model training<jupyter_code>thetaless_reg = ThetalessRegularizer(
    name='thetaless', 
    tau=1,
    dataset=dataset, 
    modality=MAIN_MODALITY,
)

sparse_reg = artm.regularizers.SmoothSparsePhiRegularizer(
    name='sparse_reg', 
    tau=-0.5,
)
def create_model(dataset, modality: str):
    model = artm.ARTM(num_topics=30, num_document_passes=1, class_ids={modality: 1.0})
    model.initialize(dataset.get_dictionary())
    
    model.scores.add(
        artm.scores.PerplexityScore(name='perplexity', class_ids=[modality])
    )
    model.scores.add(
        artm.scores.SparsityPhiScore(name='sparsity', class_id=modality)
    )
    model.scores.add(
        artm.scores.TopicKernelScore(name='kernel', class_id=modality)
    )
    
    return model
NUM_FIT_ITERATIONS = 50
plsa_model = TopicModel(
    artm_model=create_model(dataset, MAIN_MODALITY), 
    model_id='plsa_model',
    custom_scores={
        "coherence_ppmi": TopTokenCoherence(calc_pmi)
    }
)

%time plsa_model._fit(dataset.get_batch_vectorizer(), NUM_FIT_ITERATIONS)
thetaless_model = TopicModel(
    artm_model=create_model(dataset, MAIN_MODALITY), 
    model_id='thetaless_model',
    custom_regularizers={
        thetaless_reg.name: thetaless_reg
    },
    custom_scores={
        "coherence_ppmi": TopTokenCoherence(calc_pmi)
    }
)

%time thetaless_model._fit(dataset.get_batch_vectorizer(), NUM_FIT_ITERATIONS)
artm_model = create_model(dataset, MAIN_MODALITY)
artm_model.regularizers.add(sparse_reg)

sparse_model = TopicModel(
    artm_model=artm_model, 
    model_id='sparse_model',
    custom_scores={"coherence_ppmi": TopTokenCoherence(calc_pmi)}
)

%time sparse_model._fit(dataset.get_batch_vectorizer(), NUM_FIT_ITERATIONS)
artm_model = create_model(dataset, MAIN_MODALITY)
artm_model.regularizers.add(sparse_reg)

sparse_thetaless_model = TopicModel(
    artm_model=artm_model, 
    model_id='sparse_thetaless_model',
    custom_regularizers={
        thetaless_reg.name: thetaless_reg
    },
    custom_scores={
        "coherence_ppmi": TopTokenCoherence(calc_pmi)
    }
)

%time sparse_thetaless_model._fit(dataset.get_batch_vectorizer(), NUM_FIT_ITERATIONS)<jupyter_output>CPU times: user 7min 1s, sys: 1min 20s, total: 8min 21s
Wall time: 3min 49s<jupyter_text>Quality Visualizing<jupyter_code>def plot_metric(view_name, value_name, title_name):
    plt.figure(figsize=(14,8))

    plt.subplot(2,2,1)
    values = np.array(getattr(thetaless_model.score_tracker[view_name], value_name))
    med = np.median(values)
    values[values > 5 * med] = np.nan
    last_thetaless = values[-1]

    plt.title(f'{title_name} with Thetaless regularizer')
    plt.plot(values[1:])

    plt.subplot(2,2,2)
    values = np.array(getattr(plsa_model.score_tracker[view_name], value_name))
    med = np.median(values)
    values[values > 5 * med] = np.nan
    last_plsa = values[-1]
    
    plt.title(f'{title_name} without any regularization')
    plt.plot(values[1:])
    
    plt.subplot(2,2,3)
    values = np.array(getattr(sparse_model.score_tracker[view_name], value_name))
    med = np.median(values)
    values[values > 5 * med] = np.nan
    last_sparse = values[-1]

    plt.title(f'{title_name} with sparsity regularizer')
    plt.plot(values[1:])
    
    plt.subplot(2,2,4)
    values = np.array(getattr(sparse_thetaless_model.score_tracker[view_name], value_name))
    med = np.median(values)
    values[values > 5 * med] = np.nan
    last_sparse_thetaless = values[-1]

    plt.title(f'{title_name} with both regularizers')
    plt.plot(values[1:])
    
    print(
        f'Last values:\n\t'
        f'thetaless – {last_thetaless}\n\t'
        f'plsa – {last_plsa}\n\t'
        f'sparse – {last_sparse}\n\t'
        f'sparse_thetaless – {last_sparse_thetaless}\n\t' 
    )

    plt.show()
plot_metric(view_name='perplexity', value_name='value', title_name='Perplexity')
plot_metric(view_name='sparsity', value_name='value', title_name='Sparsity')
plot_metric(view_name='kernel', value_name='average_contrast', title_name='Kernel Contrast')
plot_metric(view_name='kernel', value_name='average_purity', title_name='Kernel purity')
plot_metric(view_name='kernel', value_name='average_size', title_name='Kernel size')
plot_metric(view_name='coherence_ppmi', value_name='value', title_name='Top-20 Coherence')<jupyter_output>Last values:
	thetaless – 1.6930015502023317
	plsa – 1.5415081854659372
	sparse – 1.4994938821199244
	sparse_thetaless – 1.6034352180930274<jupyter_text>Coherence evaluation on an external corpus (with Palmetto)Unfortunately, the service isn't available right now<jupyter_code>def eval_top_words(name, model, num_top_tokens=10):
    ttv = TopTokensViewer(model, method='blei', num_top_tokens=num_top_tokens)
    output = ttv.view()
    S = "http://palmetto.aksw.org/palmetto-webapp/service/{}?words=".format("umass")
    topic_score = []

    for key, modalities in output.items():
        if 'topic' in key:
            words = list(output[key]['@default_class'].keys())

            print(S + "%20".join(words))

            result = requests.get(S + "%20".join(words)).text
            topic_score += [float(result)]

    print(round(np.mean(topic_score), 2), round(np.median(topic_score), 2))
# eval_top_words('thetaless', thetaless_model)
# eval_top_words('plsa', plsa_model)
# eval_top_words('sparse', sparse_model)
# eval_top_words('sparse_thetaless', sparse_thetaless_model)<jupyter_output><empty_output><jupyter_text>Let's look at topics<jupyter_code>def print_top_words(name, model, num_top_tokens=10):
    ttv = TopTokensViewer(model, method='blei', num_top_tokens=num_top_tokens)
    output = ttv.view()

    for key, modalities in output.items():
        if 'topic' in key:
            print(', '.join(output[key][MAIN_MODALITY].keys()))
            print()
print_top_words('thetaless', thetaless_model)
print_top_words('plsa', plsa_model)
print_top_words('sparse', sparse_model)
print_top_words('sparse_thetaless', sparse_thetaless_model)<jupyter_output>space, earth, science, design, center, launch, field, project, material, mission

get, good, back, power, buy, around, big, turn, stuff, light

year, game, last, play, team, win, player, lose, guy, season

system, mail, software, support, computer, machine, user, pc, standard, contact

question, post, read, write, follow, must, ask, different, bit, note

also, number, high, source, large, several, small, add, build, require

key, information, chip, provide, phone, use, service, order, encryption, technology

use, work, problem, need, run, please, help, thank, anyone, window

new, call, include, price, etc, sell, offer, check, pay, cost

drive, card, disk, driver, hard, speed, board, mb, fast, ms

effect, condition, medical, soon, disease, compare, patient, mile, average, ed

one, think, much, seem, really, still, put, hear, sure, probably

x, send, list, line, copy, 25, output, return, section, request

believe, people, true, fact, church, agree, argument, matter, israel, jews

kill, m[...]<jupyter_text>Last coherence values (using different number of top tokens) The plots above show how the coherence (PPMI) of top 20 tokens evolves. In addition, we can look at the last PMI and PPMI values using different sizes of top-token list (5, 10, 20, 50, 100 words)<jupyter_code>occurences, co_occurences = calc_doc_occurrences(dataset, MAIN_MODALITY)

calc_pmi = create_pmi_top_function(
    occurences, co_occurences,
    dataset.get_dataset().shape[0], [5, 10, 20, 50, 100],
    co_occurrences_smooth=1e-2
)
calc_pmi(plsa_model.get_phi_dense()[0].T)
calc_pmi(thetaless_model.get_phi_dense()[0].T)
calc_pmi(sparse_model.get_phi_dense()[0].T)
calc_pmi(sparse_thetaless_model.get_phi_dense()[0].T)
# TODO: Also, let's take a look at train + test together, instead of relying on train only:<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup