| --- |
| language: |
| - dv |
| license: apache-2.0 |
| --- |
| |
| # BERT base for Dhivehi |
|
|
| Pretrained model on Dhivehi language using masked language modeling (MLM). |
|
|
| ## Tokenizer |
|
|
| The *WordPiece* tokenizer uses several components: |
|
|
| * **Normalization**: lowercase and then NFKD unicode normalization. |
| * **Pretokenization**: splits by whitespace and punctuation. |
| * **Postprocessing**: single sentences are output in format `[CLS] sentence A [SEP]` and pair sentences in format `[CLS] sentence A [SEP] sentence B [SEP]`. |
|
|
| ## Training |
|
|
| Training was performed over 16M+ Dhivehi sentences/paragraphs put together by [@ashraq](https://huggingface.co/ashraq). An Adam optimizer with weighted decay was used with following parameters: |
|
|
| * Learning rate: 1e-5 |
| * Weight decay: 0.1 |
| * Warmup steps: 10% of data |
|
|
