Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
+unigram.json filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,8 @@
+/src/data/extracted
+/src/data/kkwiki-latest-pages-articles.xml.bz2
+/src/data/kazakh_latin_pairs.jsonl
+/src/data/clean_pairs.jsonl
+/src/data/kk.txt
+/logs/**
+/src/test_minilm.py
+/MiniDalaLM/README.md

1_Pooling/config.json

@@ -0,0 +1,10 @@
+{
+  "word_embedding_dimension": 384,
+  "pooling_mode_cls_token": false,
+  "pooling_mode_mean_tokens": true,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false,
+  "pooling_mode_weightedmean_tokens": false,
+  "pooling_mode_lasttoken": false,
+  "include_prompt": true
+}

README.md

@@ -1,3 +1,148 @@
----
-license: mit
----
+---
+base_model: paraphrase-multilingual-MiniLM-L12-v2
+license: mit
+language: kaz
+tags:
+  - sentence-transformers
+  - sentence-similarity
+  - feature-extraction
+  - kazakh
+  - low-resource
+  - cultural-nlp
+  - multilingual-minilm
+pipeline_tag: sentence-similarity
+model-index:
+- name: DalaT5
+  results:
+    - task:
+        name: Binary Classification
+        type: binary-classification
+      dataset:
+        name: Kazakh Latin Corpus
+        type: custom
+      metrics:
+        - name: Training Loss
+          type: loss
+          value: 0.0023
+        - name: Cosine Accuracy (Evaluation)
+          type: accuracy
+          value: 0.9999
+        - name: Cosine Accuracy Threshold
+          type: accuracy
+          value: 0.9997
+---
+# MiniDalaLM - Embedding Extractor for Latin Kazakh 🇰🇿
+
+> 'Dala' means 'steppe' in Kazakh - a nod to where the voice of this model might echo.
+
+**MiniDalaLM** is a fine-tuned version of `paraphrase-multilingual-MiniLM-L12-v2`, trained to **extract embeddings** from Kazakh text that make use of the officially adopted, [2021 alphabet reform-based](https://astanatimes.com/2021/02/kazakhstan-presents-new-latin-alphabet-plans-gradual-transition-through-2031/) Latin script. It is meant to serve as a **foundational model** to be improved upon as needed and used alongside its more powerful transliteration-based cousin, [DalaT5](https://huggingface.co/crossroderick/dalat5).
+
+⚠️ Limitations
+- May produce unexpected outputs for very short inputs or mixed-script text
+- Accuracy may vary across dialects or uncommon characters
+
+---
+
+## 🧠 Purpose
+
+Much like DalaT5, this model wasn’t built for production-grade embedding extraction or for linguistic study alone.
+
+It was born from something else:
+- A deep **respect for Kazakh culture**
+- A belief that **no language should ever be forgotten**
+- A desire to **aid the country's modernisation efforts** through AI
+
+> *I'm not Kazakh, but I believe that there is beauty in helping those that may be in need - with the sole expectation being that it may prove useful to them. So, I help and give away freely.*
+
+---
+
+## 🌍 Жоба туралы / About the Project
+
+### 🏕 Қазақша
+
+**MiniDalaLM** - Қазақстанның ұлттық модернизациялау күш-жігерін қолдауға арналған, қазақша латын деректеріне дәл бапталған трансформатор. Модель ендірілгендер арқылы мәтіндік мүмкіндіктерді шығаруға бағытталған, бұл оны күшті лингвистикалық құралдардың негізі ретінде тамаша етеді.
+
+Бұл жоба:
+- AI жүйесінде **аз ұсынылған тілдерге** қолдау көрсетеді
+- Қазақтың латыншаланған болашағына **ашық қолжетімділік** ұсынады
+- Шетелдік – кішіпейілділікпен, ізденімпаздықпен, терең қамқорлықпен жасаған
+
+---
+
+### 🌐 English
+
+**MiniDalaLM** is a transformer fine-tuned on Kazakh Latin data, designed to support Kazakhstan’s national modernisation efforts. The model focuses on textual feature extraction via embeddings, making it ideal as the backbone of more powerful linguistic tools.
+
+This project:
+- Supports **underrepresented languages** in AI  
+- Offers **open access** to the Latinised future of Kazakh  
+- Was created by a foreigner - with humility, curiosity, and deep care
+
+---
+
+## 💻 Байқап көріңіз / Try it out
+
+Құшақтап тұрған бет арқылы тікелей пайдаланыңыз🤗 Sentence Transformers / Use directly via Hugging Face 🤗 Sentence Transformers:
+
+```python
+from sentence_transformers import SentenceTransformer
+
+model = SentenceTransformer("crossroderick/minidalalm")
+
+sentences = [
+    "Vakkaritstso-Albaneze (ital. 'Vaccarizzo Albanese') — Italiiadağy kommuna, Kalabriia äkımşılık aimağyna qarasty Kozentsa provintsiiasynda ornalasqan.",
+    "Qalanyñ tūraqty tūrğyndarynyñ sany 1236 adamdy qūraidy (2008). Halyq tyğyzdyğy 154 adam/km². Alyp jatqan jer aumağy 8 km² şamasynda. Poşta indeksı — 87060.",
+    "Eldı mekennıñ qamqorşysy — Madonna di Costantinopoli.",
+]
+
+embeddings = model.encode(sentences)
+
+print(embeddings)
+```
+
+---
+
+## 🙏 Алғыс / Acknowledgements
+
+Тәуелсіз жоба болғанына қарамастан, MiniDalaLM өте маңызды үш деректер жиынтығын пайдаланады / Despite being an independent project, MiniDalaLM makes use of three very important datasets:
+
+- The first ~50 thousand records of the Kazakh subset of the CC100 dataset by [Conneau et al. (2020)](https://paperswithcode.com/paper/unsupervised-cross-lingual-representation-1)
+- The first ~55 thousand records of the raw, Kazakh-focused part of the [Kazakh Parallel Corpus (KazParC)](https://huggingface.co/datasets/issai/kazparc) from Nazarbayev University's Institute of Smart Systems and Artificial Intelligence (ISSAI), graciously made available on Hugging Face
+- The Wikipedia dump of articles in the Kazakh language, obtained via the `wikiextractor` Python package
+
+---
+
+## 🤖 Нақты баптау нұсқаулары / Fine-tuning instructions
+
+Деректер жиынының жалпы өлшемін ескере отырып, олар осы үлгінің репозиторийіне қосылмаған. Дегенмен, MiniDalaLM-ті өзіңіз дәл баптағыңыз келсе, келесі әрекеттерді орындаңыз / Given the total size of the datasets, they haven't been included in this model's repository. However, should you wish to fine-tune MiniDalaLM yourself, please do the following:
+
+1. `get_data.sh` қабық сценарий файлын "src/data" қалтасында іске қосыңыз / Run the `get_data.sh` shell script file in the "src/data" folder
+2. Сол қалтадағы `generate_lat_pairs.py` файлын іске қосыңыз / Run the `generate_lat_pairs.py` file in the same folder 
+3. Қазақ корпус файлын тазалау және деректер жинағын араластыру үшін `generate_clean_corpus.sh` іске қосыңыз / Run `generate_clean_corpus.sh` to clean the Kazakh corpus file and shuffle the dataset
+
+KazParC деректер жинағын жүктеп алу үшін сізге Hugging Face есептік жазбасы қажет екенін ескеріңіз. Бұған қоса, жүктеп алуды бастау үшін өзіңізді аутентификациялау үшін `huggingface-cli` орнатуыңыз қажет. Бұл туралы толығырақ [мына жерден](https://huggingface.co/docs/huggingface_hub/en/guides/cli) оқыңыз / Please note that you'll need a Hugging Face account to download the KazParC dataset. Additionally, you'll need to install `huggingface-cli` to authenticate yourself for the download to commence. Read more about it [here](https://huggingface.co/docs/huggingface_hub/en/guides/cli). 
+
+Егер сіз Windows жүйесінде болсаңыз, «get_data.sh» сценарийі жұмыс істемеуі мүмкін. Дегенмен, файлдағы сілтемелерді орындап, ондағы қадамдарды қолмен орындау арқылы әлі де деректерді алуға болады. Сол сияқты, `generate_clean_corpus.sh` файлында да қате пайда болады, бұл сізге `kazakh_latin_pairs.json` файлындағы бос немесе бос жолдарды сүзу, сондай-ақ оны араластыру үшін баламалы Windows функциясын табуды талап етеді. Бұған қоса, 'wikiextractor' және `sentencetransformers` бумаларын алдын ала орнатуды ұмытпаңыз (нақты нұсқаларды 'requirements.txt' файлынан табуға болады) / If you're on Windows, the `get_data.sh` script likely won't work. However, you can still get the data by following the links in the file and manually doing the steps in there. Likewise, `generate_clean_corpus.sh` will also error out, requiring you to find an equivalent Windows functionality to filter out blank or empty lines in the `kazakh_latin_pairs.json` file, as well as shuffle it. Additionally, be sure to install the `wikiextractor` and `sentencetransformers` packages beforehand (the exact versions can be found in the `requirements.txt` file).
+
+---
+
+## 📋 Өзгеріс журналы / Changelog
+
+* **MiniDalaLM v1:** 5 мамырда жөнделді және сол күні қолжетімді болды. Қазақ морфологиясына тез бейімделіп, өзінің негізгі үлгісінің табиғатын пайдаланған бастапқы нұсқа / Fine-tuned on May 5 and made available on the same day. Initial version that benefitted from the nature of its base model, quickly adapting to Kazakh morphology
+
+
+---
+
+## 📚 Несиелер / Credits
+
+Егер сіз MiniDalaLM-ті туынды жұмыстарды зерттеуде қолдансаңыз - біріншіден, рахмет. Екіншіден, егер сіз қаласаңыз, дәйексөз келтіріңіз / If you use MiniDalaLM in research of derivative works - first off, thank you. Secondly, should you be willing, feel free to cite:
+
+```
+@misc{pereira_cruz_dalat5_2025,
+  author = {Rodrigo Pereira Cruz},
+  title = {MiniDalaLM: Feature extraction on Latin Kazakh via embeddings},
+  year = 2025,
+  url = {https://huggingface.co/crossroderick/minidalalm},
+  publisher = {Hugging Face}
+}
+```

checkpoints/model_1/eval/binary_classification_evaluation_eval_results.csv

@@ -0,0 +1,4 @@
+epoch,steps,cosine_accuracy,cosine_accuracy_threshold,cosine_f1,cosine_precision,cosine_recall,cosine_f1_threshold,cosine_ap,cosine_mcc
+0.12711325791280031,1000,0.9999880826113382,0.9995862245559692,0,0,0,0,0.0,0.0
+0.25422651582560063,2000,0.9999880826113382,0.9996930360794067,0,0,0,0,0.0,0.0
+0.3813397737384009,3000,0.9999880826113382,0.9997444152832031,0,0,0,0,0.0,0.0

config.json

@@ -0,0 +1,25 @@
+{
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 384,
+  "initializer_range": 0.02,
+  "intermediate_size": 1536,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.51.2",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 250037
+}

config_sentence_transformers.json

@@ -0,0 +1,10 @@
+{
+  "__version__": {
+    "sentence_transformers": "4.1.0",
+    "transformers": "4.51.2",
+    "pytorch": "2.5.1+cu124"
+  },
+  "prompts": {},
+  "default_prompt_name": null,
+  "similarity_fn_name": "cosine"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:196c0c9ee62fe9083b00d15b46cda9f5cbfb0fdec38b1b392c1d43aaddee1ba2
+size 470637416

modules.json

@@ -0,0 +1,14 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  }
+]

requirements.txt

@@ -0,0 +1,5 @@
+datasets==3.5.0
+sentence_transformers==4.1.0
+torch==2.5.1
+tqdm==4.67.1
+wikiextractor==3.0.7

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 128,
+  "do_lower_case": false
+}

special_tokens_map.json

@@ -0,0 +1,51 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

src/data/generate_clean_corpus.sh

@@ -0,0 +1,2 @@
+shuf kazakh_latin_pairs.jsonl -o kazakh_latin_pairs.jsonl
+grep '\S' kazakh_latin_pairs.jsonl > clean_pairs.jsonl

src/data/generate_lat_pairs.py

@@ -0,0 +1,177 @@
+import os
+import json
+import random
+from tqdm import tqdm
+from itertools import islice
+from datasets import load_dataset
+
+from typing import List
+
+
+# Kazakh Cyrillic character to the Kazakh Latin character mapping from 2021 onwards
+cyrillic_to_latin = {
+    "А": "A", "а": "a",
+    "Ә": "Ä", "ә": "ä",
+    "Б": "B", "б": "b",
+    "Д": "D", "д": "d",
+    "Е": "E", "е": "e",
+    "Ф": "F", "ф": "f",
+    "Г": "G", "г": "g",
+    "Ғ": "Ğ", "ғ": "ğ",
+    "Х": "H", "х": "h",      # also Һ, see below
+    "Һ": "H", "һ": "h",
+
+    "И": "I", "и": "i",      # used for [и], [й]
+    "І": "I", "і": "ı",      # distinct from И in sound, both map to 'I/i'
+    "Ж": "J", "ж": "j",
+
+    "К": "K", "к": "k",
+    "Қ": "Q", "қ": "q",
+    "Л": "L", "л": "l",
+    "М": "M", "м": "m",
+    "Н": "N", "н": "n",
+    "Ң": "Ñ", "ң": "ñ",
+
+    "О": "O", "о": "o",
+    "Ө": "Ö", "ө": "ö",
+
+    "П": "P", "п": "p",
+    "Р": "R", "р": "r",
+    "С": "S", "с": "s",
+    "Ш": "Ş", "ш": "ş",
+    "Т": "T", "т": "t",
+
+    "У": "U", "у": "u",      # basic 'u' sound, distinct from Ұ
+    "Ұ": "Ū", "ұ": "ū",      # back rounded, used frequently
+    "Ү": "Ü", "ү": "ü",      # front rounded
+
+    "В": "V", "в": "v",
+    "Ы": "Y", "ы": "y",
+    "Й": "I", "й": "i",      # same treatment as И
+    "Ц": "Ts", "ц": "ts",    # for Russian borrowings
+    "Ч": "Ch", "ч": "ch",
+    "Щ": "Ş", "щ": "ş",      # typically simplified to 'ş'
+
+    "Э": "E", "э": "e",
+    "Ю": "Iu", "ю": "iu",    # borrowed words only
+    "Я": "Ia", "я": "ia",
+
+    "Ъ": "", "ъ": "",
+    "Ь": "", "ь": "",
+
+    "З": "Z", "з": "z",
+
+    # Additional (not in table but used in borrowings)
+    "Ё": "Io", "ё": "io",
+}
+
+
+def convert_to_latin(text: str) -> str:
+    """
+    Simple function to apply the Cyrillic -> Latin mapping for Kazakh characters.
+    """
+    return ''.join(cyrillic_to_latin.get(char, char) for char in text)
+
+
+def create_augmented_pairs(sentences: List) -> List:
+    """
+    Create Kazakh Latin pairs between original sentences and slightly changed ones.
+    """
+    pairs = []
+
+    # Randomly change sentences
+    for _ in range(len(sentences)):
+        s = random.choice(sentences)
+
+        # Create a minor variation
+        s_aug = s.replace(".", "").replace(",", "")  # remove punctuation
+        s_aug = s_aug.replace("ğa", "ga").replace("ñ", "n")  # light spelling variants
+        s_aug = s_aug.capitalize()
+
+        if s != s_aug:
+            pairs.append({"texts": [s, s_aug]})
+
+    return pairs
+
+
+# Process all files in "extracted" dir
+# Output file path
+output_path = "src/data/kazakh_latin_pairs.jsonl"
+
+# List to hold all Latin sentences
+latin_sentences = []
+
+# First step: process the Wikipedia dump
+print("Processing the Wikipedia dump of Kazakh articles...")
+
+# Iterate over all folders
+for root, _, files in os.walk("src/data/extracted"):
+    for fname in tqdm(files, desc = "Files in Wikipedia dump"):
+        with open(os.path.join(root, fname), 'r', encoding = "utf-8") as f:
+            for line in f:
+                try:
+                    data = json.loads(line)
+                    cyr_text = data["text"].strip()
+                    lat_text = convert_to_latin(cyr_text).strip()
+
+                    if lat_text:
+                        latin_sentences.append(lat_text)
+
+                except Exception as e:
+                    tqdm.write(f"Skipping due to: {e}")
+
+                    continue
+
+print("Done")
+
+# Second step: process the "CC100-Kazakh" dataset
+print("Loading 'CC100-Kazakh' dataset...")
+
+with open("src/data/kk.txt", 'r', encoding = "utf-8") as f:
+    for line in tqdm(islice(f, 50_000), total = 50_000, desc = "Lines in CC100-Kazakh"):
+        try:
+            cyr_text = line.strip()
+            lat_text = convert_to_latin(cyr_text).strip()
+
+            if lat_text:
+                latin_sentences.append(lat_text)
+
+        except Exception as e:
+            tqdm.write(f"Skipping due to: {e}")
+
+            continue
+
+# Third step: process 15% of the raw, Kazakh-centred part of the "KazParC" dataset
+print("Loading 'KazParC' dataset...")
+
+kazparc = load_dataset("issai/kazparc", "kazparc_raw", split = "train[:15%]")
+
+for entry in tqdm(kazparc, desc = "Entries in KazParC"):
+    try:
+        if "kk" in entry and isinstance(entry["kk"], str):
+            cyr_text = entry["kk"].strip()
+            lat_text = convert_to_latin(cyr_text).strip()
+
+            if lat_text:
+                latin_sentences.append(lat_text)
+
+    except Exception as e:
+        tqdm.write(f"Skipping due to: {e}")
+
+        continue
+
+
+# Fourth and last step: create Latin sentences with variations
+print("Creating Latin pairs...")
+
+augmented_pairs = create_augmented_pairs(latin_sentences)
+
+with open(output_path, 'w', encoding = "utf-8") as f:
+    for pair in tqdm(augmented_pairs, desc = "Dataset entries"):
+        try:
+            f.write(json.dumps(pair, ensure_ascii = False) + "\n")
+
+        except Exception as e:
+            tqdm.write(f"Skipping due to: {e}")
+
+            continue

src/data/get_data.sh

@@ -0,0 +1,4 @@
+wget https://dumps.wikimedia.org/kkwiki/latest/kkwiki-latest-pages-articles.xml.bz2
+wget http://data.statmt.org/cc-100/kk.txt.xz
+unxz kk.txt.xz
+python3 -m wikiextractor.WikiExtractor kkwiki-latest-pages-articles.xml.bz2 --output extracted --json

src/train_minilm.py

@@ -0,0 +1,43 @@
+from datasets import load_dataset
+from torch.utils.data import DataLoader
+from sentence_transformers import SentenceTransformer, InputExample, losses, evaluation
+
+
+# Path config
+base_model = "paraphrase-multilingual-MiniLM-L12-v2"
+data_path = "src/data/clean_pairs.jsonl"
+
+# Load the full dataset and convert to input examples
+dataset = load_dataset("json", data_files = "src/data/clean_pairs.jsonl", split = "train")
+
+# Create input examples
+all_samples = [
+    InputExample(texts = entry["texts"])
+    for entry in dataset
+]
+
+# Split into train and eval sets (75/25)
+split_idx = int(len(all_samples) * 0.75)
+train_samples = all_samples[:split_idx]
+eval_samples = all_samples[split_idx:]
+
+# Model and loss
+model = SentenceTransformer(base_model)
+train_dataloader = DataLoader(train_samples, shuffle = True, batch_size = 32)
+train_loss = losses.MultipleNegativesRankingLoss(model)
+
+# Evaluation setup
+evaluator = evaluation.BinaryClassificationEvaluator.from_input_examples(eval_samples, name = "eval")
+
+# Train with eval
+model.fit(
+    train_objectives = [(train_dataloader, train_loss)],
+    epochs = 0.5,
+    warmup_steps = 100,
+    evaluator = evaluator,
+    evaluation_steps = 1000,
+    show_progress_bar = True
+)
+
+# Save final model
+model.save("MiniDalaLM")

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cad551d5600a84242d0973327029452a1e3672ba6313c2a3c3d69c4310e12719
+size 17082987

tokenizer_config.json

@@ -0,0 +1,65 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "250001": {
+      "content": "<mask>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "<s>",
+  "do_lower_case": true,
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "mask_token": "<mask>",
+  "max_length": 128,
+  "model_max_length": 128,
+  "pad_to_multiple_of": null,
+  "pad_token": "<pad>",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "</s>",
+  "stride": 0,
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "<unk>"
+}

unigram.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:da145b5e7700ae40f16691ec32a0b1fdc1ee3298db22a31ea55f57a966c4a65d
+size 14763260