Multilingual tokenizer
Hi! Thanks for releasing the model.
I can see 13k tokens in vocab in this nemo file. I wonder if you could share how you built tokenizers so that we can add other languages and finetune the model with it.
Thank you!
The existing tokenizer already covers 40 language-locales and ~13k tokens. For fine-tuning on a supported language, you don't need to modify the tokenizer, just provide data with the correct target_lang tag.
If you want to add a genuinely new language whose script isn't covered:
Train a new SentencePiece tokenizer that includes both the existing vocabulary and your new language's characters.
Resize the model's embedding and output projection layers to match.
Fine-tune from the base checkpoint with the expanded tokenizer.
NeMo's tokenizer training: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/asr_language_modeling_and_customization.html#training-a-custom-tokenizer
For most use cases (languages whose scripts are already in the 13k vocab), the existing tokenizer works as-is. the Swahili example I linked demonstrates this since Swahili uses Latin script which is already in the vocabulary.