|
|
--- |
|
|
license: mit |
|
|
library_name: mlx |
|
|
base_model: deepseek-ai/DeepSeek-V3.1 |
|
|
tags: |
|
|
- mlx |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
**See DeepSeek-V3.1 5.5bit MLX in action - [demonstration video](https://youtu.be/ufXZI6aqOU8)** |
|
|
|
|
|
*q5.5bit quant typically achieves 1.141 perplexity in our testing* |
|
|
| Quantization | Perplexity | |
|
|
|:------------:|:----------:| |
|
|
| **q2.5** | 41.293 | |
|
|
| **q3.5** | 1.900 | |
|
|
| **q4.5** | 1.168 | |
|
|
| **q5.5** | 1.141 | |
|
|
| **q6.5** | 1.128 | |
|
|
| **q8.5** | 1.128 | |
|
|
|
|
|
## Usage Notes |
|
|
|
|
|
* Runs on a single M3 Ultra 512GB RAM using [Inferencer app](https://inferencer.com) |
|
|
* Memory usage: ~480 GB |
|
|
* Expect ~13-19 tokens/s |
|
|
* Quantized with a modified version of [MLX](https://github.com/ml-explore/mlx) 0.26 |
|
|
* For more details see [demonstration video](https://youtu.be/ufXZI6aqOU8) or visit [DeepSeek-V3.1](https://huggingface.co/deepseek-ai/DeepSeek-V3.1). |
