1.1 MB

Ctrl+K

1 contributor

History: 9 commits

janakhpon

feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue

9d74fcf 7 months ago

.gitattributes
285 Bytes
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 7 months ago
.gitignore
114 Bytes
feat: simplified mon tokenizer in hf format, updated tags 7 months ago
.python-version
5 Bytes
feat: simplified mon tokenizer in hf format 7 months ago
README.md
2.65 kB
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 7 months ago
convert_to_hf.py
19.3 kB
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 7 months ago
generation_config.json
291 Bytes
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 7 months ago
mon_tokenizer.meta.json
12.9 kB
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 7 months ago
pyproject.toml
1.1 kB
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 7 months ago
sample_usage.py
6.14 kB
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 7 months ago
special_tokens_map.json
552 Bytes
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 7 months ago
test_tokenizer.py
18.2 kB
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 7 months ago
tokenizer.json
278 kB
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 7 months ago
tokenizer.model
338 kB
xet

feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 7 months ago
tokenizer_config.json
1.7 kB
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 7 months ago
upload_to_hub.py
18 kB
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 7 months ago
uv.lock
406 kB
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 7 months ago