Update README.md
Browse files
README.md
CHANGED
|
@@ -25,6 +25,9 @@ This official repository unveils the TransNormerLLM3 model along with its open-s
|
|
| 25 |
- **TransNormerLLM3-15B** features **14.83 billion** parameters. It is structured with **42 layers**, includes **40 attention heads**, and has a total **embedding size of 5120**.
|
| 26 |
- **TransNormerLLM3-15B** is purely intergrated with **[Lightning Attention-2](http://arxiv.org/abs/2401.04658)**, which can maintain a **stable TGS** during training of **unlimited sequence lengths**, up until encountering firm limitations like GPU memory constraints.
|
| 27 |
- **Titoken** tokenizer is used with a total **vocabulary size** of about **100,000**.
|
|
|
|
|
|
|
|
|
|
| 28 |
|
| 29 |
### Pre-training Logbook
|
| 30 |
* Realtime Track: https://api.wandb.ai/links/opennlplab/kip314lq
|
|
@@ -93,5 +96,6 @@ If you wish to cite our work, please use the following reference:
|
|
| 93 |
```
|
| 94 |
|
| 95 |
<p align="center">
|
|
|
|
| 96 |
- OpenNLPLab @2024 -
|
| 97 |
</p>
|
|
|
|
| 25 |
- **TransNormerLLM3-15B** features **14.83 billion** parameters. It is structured with **42 layers**, includes **40 attention heads**, and has a total **embedding size of 5120**.
|
| 26 |
- **TransNormerLLM3-15B** is purely intergrated with **[Lightning Attention-2](http://arxiv.org/abs/2401.04658)**, which can maintain a **stable TGS** during training of **unlimited sequence lengths**, up until encountering firm limitations like GPU memory constraints.
|
| 27 |
- **Titoken** tokenizer is used with a total **vocabulary size** of about **100,000**.
|
| 28 |
+
<p align="center">
|
| 29 |
+
<img src="./images/TransNormer3.jpg" width="75%" />
|
| 30 |
+
</p>
|
| 31 |
|
| 32 |
### Pre-training Logbook
|
| 33 |
* Realtime Track: https://api.wandb.ai/links/opennlplab/kip314lq
|
|
|
|
| 96 |
```
|
| 97 |
|
| 98 |
<p align="center">
|
| 99 |
+
<img src="./images/lightning3-leopard.jpg" width="67%" />
|
| 100 |
- OpenNLPLab @2024 -
|
| 101 |
</p>
|