metythorn
/

gpt2-tokenizer

Model card Files Files and versions

gpt2-tokenizer / README.md

metythorn's picture

Add model card

3694cd0 verified about 1 month ago

|

history blame contribute delete

232 Bytes

Khmer‑English GPT‑2 Tokenizer

Vocab size: 50,257
Algorithm: Byte‑Level BPE (byte_fallback)
Special tokens: <|endoftext|>, <|bos|>, <|pad|>, <|unk|>
Trained on: metythorn/khmer‑english‑corpus