<jupyter_start><jupyter_text>Spell Check Indonesia  Install Gensim<jupyter_code># !pip install --upgrade gensim<jupyter_output><empty_output><jupyter_text>Download Pre-Trained Fasttext Model IndonesiaPre-trained Fasttext model yang diunduh adalah hasil pelatihan menggunakan metode CBOW dengan *position-weight*, dalam dimensi 300, dengan panjang karakter n-gram sebesar 5, *window size* 5 dan *negatives* 10. Link kumpulan pre-trained model Fasttext dapat diunduh [disini](https://fasttext.cc/docs/en/crawl-vectors.html).<jupyter_code>import wget

def bar_progress(current, total, width=80):
    progress_message = "Downloading: %d%% [%d / %d] bytes" % (current / total * 100, current, total)
     # Don't use print() as it will print in new line every time.
    sys.stdout.write("\r" + progress_message)
    sys.stdout.flush()

url = 'https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.id.300.bin.gz'

# filename = wget.download(url, bar=bar_progress)<jupyter_output><empty_output><jupyter_text>Extract model dengan menggunakan `gunzip`<jupyter_code>%%time

import gzip
import shutil

# with gzip.open('cc.id.300.bin.gz', 'rb') as f_in:
#     with open('cc.id.300.bin', 'wb') as f_out:
#         shutil.copyfileobj(f_in, f_out)<jupyter_output>Wall time: 0 ns<jupyter_text>Import Libraries<jupyter_code>from gensim.models import fasttext
from gensim.models.fasttext import load_facebook_model<jupyter_output><empty_output><jupyter_text>Load Fasttext Model using Gensim<jupyter_code>%%time

# model = fasttext.load_facebook_model('cc.id.300.bin')<jupyter_output>Wall time: 0 ns<jupyter_text>Pada pre-trained model Fasttext terdapat 2.000.000 (dua juta) *vocabulary*<jupyter_code>import pickle

with open('models.pkl', 'rb') as file:
    vocab = pickle.load(file)

len(vocab)
list(vocab)[:10]
# with open('models.pkl', 'wb') as file:
#     pickle.dump(vocab, file)<jupyter_output><empty_output><jupyter_text>Create Index for each word in VocabularyBuat *dictionary word-rank* pada tiap kata/karakter yang terdapat pada *vocabulary*.<jupyter_code>%%time

words = list(vocab)

w_rank = {}
for i,word in enumerate(words):
    w_rank[word] = i
    
WORDS = w_rank
import itertools

dict(itertools.islice(WORDS.items(), 10))<jupyter_output><empty_output><jupyter_text>Peter Norvig Spelling CorrectorSalah satu metode paling sederhana untuk Spelling Corrector adalah dengan metode yang diterapkan oleh [<NAME>.](https://norvig.com/spell-correct.html) PerbedaanTerdapat perbedaaan pada code <NAME>vig dengan code CPMP untuk penerapan Spell Check. Perbedaannya jika menggunakan **code Peter Norvig dengan menghitung frekuensi dari tiap kata dari kamus**, sedangkan **code CPMP dengan menggunakan ranking dari Word2Vec model**. Cara KerjaSpell Check ini menggunakan Teorema Bayes untuk menemukan *correction c* dengan memilih *probability* terbesar dari semua *probability candidate correction*. Metode tersebut dapat dijabarkan menjadi 4 bagian:1. **Selection Mechanism**: `argmax`, dengan memilih *candidate* yang memiliki *probability* terbesar.2. **Candidate Model:** `c ∈ candidates`, didapatkan *candidate corrections c*, untuk dipertimbangkan.3. **Languange Model**: `P(c)`, *probability* dari kemunculan *candidate corretion c* pada kamus. Pada penerapannya menggunakan kamus Word2Vec model.4. **Error Model**: `P(w|c)`, *probability* apabila yang diketik adalah kata *w* sedangkan yang dimaksud adalah *c*. Sebagai contoh, *probability* `P(mkan|makan)` lebih tinggi dibandingkan dengan *probability* `P(mkanxxxyz|makan)` yang lebih rendah.<jupyter_code>import re
from collections import Counter

def words(text): return re.findall(r'\w+', text.lower())

def P(word): 
    "Probability of `word`."
    # use inverse of rank as proxy
    # returns 0 if the word isn't in the dictionary
    return - WORDS.get(word, 0)

def correction(word): 
    "Most probable spelling correction for word."
    return max(candidates(word), key=P)

def candidates(word): 
    "Generate possible spelling corrections for word."
    return (known([word]) or known(edits1(word)) or known(edits2(word)) or [word])

def known(words): 
    "The subset of `words` that appear in the dictionary of WORDS."
    return set(w for w in words if w in WORDS)

def edits1(word):
    "All edits that are one edit away from `word`."
    letters    = 'abcdefghijklmnopqrstuvwxyz'
    splits     = [(word[:i], word[i:])    for i in range(len(word) + 1)]
    deletes    = [L + R[1:]               for L, R in splits if R]
    transposes = [L + R[1] + R[0] + R[2:] for L, R in splits if len(R)>1]
    replaces   = [L + c + R[1:]           for L, R in splits if R for c in letters]
    inserts    = [L + c + R               for L, R in splits for c in letters]
    return set(deletes + transposes + replaces + inserts)

def edits2(word): 
    "All edits that are two edits away from `word`."
    return (e2 for e1 in edits1(word) for e2 in edits1(e1))<jupyter_output><empty_output><jupyter_text>Spelling Test<jupyter_code>correction("kcing")
correction('J4karta')
correction('mnyedihknn')<jupyter_output><empty_output><jupyter_text>Slang + Spell Check<jupyter_code>import json

with open("source/slang.txt") as f:
    slangS = json.loads(f.read())
    
with open("source/formal.txt") as f:
    formalS = json.loads(f.read())
    
type(slangS), type(formalS)
import re

def join_punctuation(seq, characters='.,;?!'):
    characters = set(characters)
    seq = iter(seq)
    current = next(seq)

    for nxt in seq:
        if nxt in characters:
            current += nxt
        else:
            yield current
            current = nxt

    yield current

def slang(T):
    Texts = re.findall(r"[\w']+|[.,!?;]",T)
    
    _spelling = []
    for text in Texts:
        _spelling.append(correction(text))
    
    for index,text in enumerate(_spelling):
        if text in slangS.keys():
            _spelling[index] = slangS[text]
            
    for index,text in enumerate(_spelling):
        if text in formalS.keys():
            _spelling[index] = formalS[text]
        
    
    _text = ' '.join(_spelling)
    _text = re.sub(r' ([^A-Za-z0-9])', r'\1', _text)
    return _text
    
slang('Unyil prg ke jkt, utk ttp men1mba ilmu!')<jupyter_output><empty_output><jupyter_text>Sinonim<jupyter_code>import json

with open('source/dict.json') as f:
    mydict = json.load(f)
def getSinonim(word):
    if word in mydict.keys():
        return mydict[word]['sinonim']
    else:
        return []


def getAntonim(word):
    if word in mydict.keys():
        if 'antonim' in mydict[word].keys():
            return mydict[word]['antonim']
    
    return []
print(getSinonim('senang'))
print(getSinonim(getAntonim('senang')[0]))<jupyter_output>['duka', 'getir ', 'gundah', 'lara', 'masygul', 'menyesak', 'merana', 'pedih', 'pilu', 'prihatin', 'sedu', 'susah hati', 'terharu', 'trenyuh']<jupyter_text>Sources- https://github.com/louisowen6/NLP_bahasa_resourcespos-tagging- https://medium.com/@yasirabd/spell-check-indonesia-menggunakan-pre-trained-fasttext-model-14e90a3f1ac0- https://norvig.com/spell-correct.html  Kata Baku<jupyter_code>with open('dict.txt', 'r') as f:
    err = f.read()
a = err.split("\n")
a[1]
test = []
for i in a:
    x = i.split('|')
    test.append(x)
test[:10]
# Nested list to dict
data= { k[0]: k[1:] for k in test }
# convert dict.value list to string
data_exp = {key:''.join(value) for key,value in data.items()}
# reverse keys and values
new_dict = { v:k for k,v in data_exp.items()}
# export as txt
with open('formal.txt', 'w') as convert_file:
     convert_file.write(json.dumps(new_dict))<jupyter_output><empty_output><jupyter_text>Imporve Slang<jupyter_code>import pandas as pd
df = pd.read_csv('source/message.csv')
new_df = df.loc[:, ["transformed", "original-for"]]
new_df
slang = new_df["transformed"]
formal = new_df["original-for"]
ds = {}

for i,y in zip(slang, formal):
    ds[i] = y
import json
with open('slang1.txt', 'w') as convert_file:
     convert_file.write(json.dumps(ds))
with open("slang.txt") as f:
    slang = json.loads(f.read())
    
with open("slang1.txt") as f:
    slang1 = json.loads(f.read())
combine_slang = slang | slang1
len(combine_slang), len(slang), len(slang1)
import json
with open('slang_corpus.txt', 'w') as convert_file:
     convert_file.write(json.dumps(combine_slang))<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup