File size: 1,097 Bytes
eb2d0f0 5d491cd eb2d0f0 5d491cd b2b4c5e 5d491cd 114ae50 5d491cd b2b4c5e 5d491cd |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
---
license: other
license_name: modified-mit
library_name: mlx
base_model: moonshotai/Kimi-K2-Instruct
pipeline_tag: text-generation
tags:
- mlx
---
**See Kimi-K2 Dynamic MLX in action - [https://youtu.be/-zfUvA2CDqE](https://youtu.be/-zfUvA2CDqE)**
*q3.95bit dynamic quant achieves 1.243 perplexity in our testing, slotting closer to q4 perplexity (1.168) than q3 perplexity (1.900).*
| Quantization | Perplexity |
|:------------:|:----------:|
| **q2** | 41.293 |
| **q3** | 1.900 |
| **q3.95** | 1.243 |
| **q4** | 1.168 |
| **q6** | 1.128 |
| **q8** | 1.128 |
## Usage Notes
* Built with a modified version of [MLX](https://github.com/ml-explore/mlx) 0.26
* Runs on a single M3 Ultra 512GB RAM
* Requires expanding VRAM limit to at least ~500000 MB
* For a larger context window, 507000 is used in VRAM limit command below.
* `sudo sysctl iogpu.wired_limit_mb=507000`
* Expect ~20 tokens/s
* For more details see [demonstration video](https://youtu.be/-zfUvA2CDqE) or visit [Kimi K2](https://moonshotai.github.io/Kimi-K2/). |