<jupyter_start><jupyter_text>export ARTM_SHARED_LIBRARY=/Users/salik/Projects/mobod/bigartm/build/src/artm/libartm.dylib<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer
import pandas as pd
import os
from sklearn.datasets import dump_svmlight_file
import artm
os.environ['ARTM_SHARED_LIBRARY']='/Users/salik/Projects/MIPT-MOBOD/bigartm/build/src/artm/libartm.dylib'<jupyter_output><empty_output><jupyter_text>Скачать данные hillary-clinton-emails с сайта kaggle.com (потребуется регистра- ция). Взять из данных полные тексты писем и создать из них файл в формате Vowpal Wabbit.<jupyter_code>data_directory = 'hillary-clinton-emails'
emails = pd.read_csv(os.path.join(data_directory, 'Emails.csv'))
vec = CountVectorizer(decode_error='replace')
X = vec.fit_transform(emails.RawText).todense()
df = pd.DataFrame(X)
df['subject'] = emails['ExtractedSubject']
df['id'] = emails['Id']
X1 = df.copy()
Y = X1['id']
X1 = X1.drop('id', axis=1)
X1 = X1.drop('subject', axis=1)
dump_svmlight_file(X1, Y, './dump_svmlight_file.csv')
tokens = {val: key for key, val in zip(vec.vocabulary_.keys(), vec.vocabulary_.values())}
with open('./vowpal_wabbit.dat', 'w+') as vowpal_wabbit_file:
    for line in open('./dump_svmlight_file.csv'):
        try:
            y, x = line.split( " ", 1 )
        except ValueError:
            pass
        features_artm = []
        for s in x.split(' '):
            try:
                features_artm.append(':'.join([tokens[int(s.split(':')[0])], s.split(':')[1]]))
            except:
                pass


        new_line = 'doc' + y + ' ' + ' '.join(features_artm).encode('utf-8')
        vowpal_wabbit_file.write( new_line )<jupyter_output><empty_output><jupyter_text>Создать батчи и словарь коллекции в соответствии с инструкциями из учебного ноутбука.<jupyter_code>batch = artm.BatchVectorizer(data_path='vowpal_wabbit.dat',
                                        data_format='vowpal_wabbit',
                                        target_folder='batches')
my_dictionary = artm.Dictionary()
my_dictionary.gather(data_path='batches')
my_dictionary.filter(min_tf=10, max_tf=4000)<jupyter_output><empty_output><jupyter_text>Написать скрипт для обучения обучения модели с регуляризаторами. Рекомен- дуется обучать модель с 25-30 темами. Рекомендуется использовать регуляри- затор разреживания и декорреляции тем. После обучения каждой модели извлекать из неё топ-20 самых вероятных слов вместе с их вероятностями и оценивать степень интерпретируемости тем.<jupyter_code>model = artm.ARTM(num_topics=25, dictionary=my_dictionary)
model.scores.add(artm.PerplexityScore(name='my_fisrt_perplexity_score',
                                      use_unigram_document_model=False,
                                      dictionary=my_dictionary))
model.fit_offline(batch_vectorizer=batch, num_collection_passes=10)
model.score_tracker['my_fisrt_perplexity_score'].last_value<jupyter_output><empty_output><jupyter_text>---<jupyter_code>model = artm.ARTM(num_topics=25, dictionary=my_dictionary, cache_theta=False)
model.scores.add(artm.PerplexityScore(name='PerplexityScore',
                                      use_unigram_document_model=False,
                                      dictionary=my_dictionary))
model.scores.add(artm.SparsityThetaScore(name='SparseTheta'))
model.scores.add(artm.TopTokensScore(name='TopTokensScore'))
model.fit_offline(batch_vectorizer=batch, num_collection_passes=10)
print model.score_tracker['PerplexityScore'].last_value
print model.score_tracker['SparseTheta'].last_value
for topic_name in model.topic_names:
    print topic_name + ': ',
    print model.score_tracker['TopTokensScore'].last_tokens[topic_name]<jupyter_output>379.274454588
0.00492950654582
topic_0:  [u'party', u'were', u'cameron', u'which', u'government', u'labour', u'election', u'out', u'tories', u'uk']
topic_1:  [u'china', u'pakistan', u'countries', u'world', u'people', u'rights', u'human', u'government', u'development', u'global']
topic_2:  [u'jacob', u'me', u'sullivanjj', u'sun', u'11', u'12', u'sunday', u'know', u'should', u'b5']
topic_3:  [u'obama', u'mr', u'clinton', u'when', u'administration', u'him', u'says', u'white', u'some', u'policy']
topic_4:  [u'my', u'afghanistan', u'me', u'were', u'afghan', u'think', u'when', u'know', u'clinton', u'your']
topic_5:  [u'office', u'00', u'meeting', u'15', u'room', u'en', u'12', u'45', u'11', u'arrive']
topic_6:  [u'israel', u'israeli', u'peace', u'palestinian', u'netanyahu', u'palestinians', u'jewish', u'arab', u'jerusalem', u'talks']
topic_7:  [u'hdr22', u'abedinh', u'sat', u'lona', u'18', u'21', u'fri', u'hrod17', u'valmoro', u'aug']
topic_8:  [u'security', u'united', u'states', u'internatio[...]<jupyter_text>---<jupyter_code>model = artm.ARTM(num_topics=25, dictionary=my_dictionary)
model.scores.add(artm.PerplexityScore(name='PerplexityScore',
                                      use_unigram_document_model=False,
                                      dictionary=my_dictionary))
model.scores.add(artm.TopTokensScore(name='TopTokensScore', num_tokens=20))

model.regularizers.add(artm.SmoothSparsePhiRegularizer(name='SparsePhi', tau=-2.0))
model.regularizers.add(artm.SmoothSparseThetaRegularizer(name='SparseTheta', tau=-1))
model.regularizers.add(artm.DecorrelatorPhiRegularizer(name='DecorrelatorPhi', tau=1e+7))
model.fit_offline(batch_vectorizer=batch, num_collection_passes=100)
print model.score_tracker['PerplexityScore'].last_value
for topic_name in model.topic_names:
    print topic_name + ': ',
    print model.score_tracker['TopTokensScore'].last_tokens[topic_name]
import tqdm
cont, counter = {}, 1
for topic_index, topic_name in enumerate(tqdm.tqdm_notebook(model.topic_names, desc="topics")):
    score_tracker = model.score_tracker['TopTokensScore']
    for i in tqdm.tqdm_notebook(xrange(len(score_tracker.last_tokens[topic_name])), desc="tokens"):
        cont[score_tracker.last_tokens[topic_name][i]] = counter
        counter += 1
model_topic_names = model.topic_names
model_score_tracker = model.score_tracker
score_tracker_last_tokens = score_tracker.last_tokens
score_tracker_last_weights = score_tracker.last_weights
with open('gephi.gexf', 'wb') as f:
    num_top_tokens = 20 
    num_topics = 25 
    f.write('<gexf xmlns:viz="http:///www.gexf.net/1.1draft/viz" xmlns="http://www.gexf.net/1.1draft" version="1.1">\n')
    f.write('<meta lastmodifieddate="2010-03-03+23:44">\n')
    f.write('<creator>Gephi 0.7</creator>\n')
    f.write('</meta>\n')
    f.write('<graph defaultedgetype="undirected" idtype="string" type="static">\n')
    f.write('<nodes count="{}">\n'.format(len(cont) + num_topics))
    
    for token, value in cont.iteritems():
        f.write('<node id="{0}" label="{1}"/>\n'.format(value, token))
        for id in xrange(num_topics):
            f.write('<node id="{0}" label="TOPIC_{1}"/>\n'.format(num_top_tokens * num_topics + id, id))
            
    f.write('</nodes>\n')
            
    edge_id = 0
    strs_to_write = []
    score_tracker_last_tokens = score_tracker.last_tokens
    score_tracker_last_weights = score_tracker.last_weights
    for token, value in cont.iteritems():
        for topic_index, topic_name in enumerate(model_topic_names):
            score_tracker = model_score_tracker['TopTokensScore']
            for i in xrange(len(score_tracker_last_tokens[topic_name])):
                if score_tracker_last_tokens[topic_name][i] == token:
                    strs_to_write.append('<edge id="{0}" source="{1}" target="{2}" weight="{3}"/>\n'.format(
                    edge_id, num_top_tokens * num_topics + topic_index, value, score_tracker_last_weights[topic_name][i]))
                    edge_id += 1
    f.write('<edges count="{}">\n'.format(len(strs_to_write)))
    for elem in strs_to_write:
        f.write(elem)
        
    f.write('</edges>\n')
    f.write('</graph>\n')
    f.write('</gexf>\n')<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup