Hugging Face's logo

RedHatAI
/
Qwen2.5-VL-72B-Instruct-quantized.w8a8

Image-to-Text
Transformers
Safetensors
English
qwen2_5_vl
vllm
vision
w8a8
text-generation-inference
8-bit precision
compressed-tensors
Model card Files
xet
Community
2
shubhrapandit commited on
Commit
f005e30
·
verified ·
1 Parent(s): 9d691ab

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +28 -28
README.md CHANGED
@@ -400,10 +400,10 @@ The following performance benchmarks were conducted with [vLLM](https://docs.vll
400
  <td>5.66</td>
401
  <td>4.3</td>
402
  <td>252</td>
403
- <td>4.3</td>
404
- <td>252</td>
405
- <td>1.0</td>
406
- <td>1065</td>
407
  </tr>
408
  </tbody>
409
  </table>
@@ -451,22 +451,22 @@ The following performance benchmarks were conducted with [vLLM](https://docs.vll
451
  <tr>
452
  <td>neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w8a8</td>
453
  <td>1.80</td>
454
- <td>1.2</td>
455
- <td>578</td>
456
- <td>4.0</td>
457
- <td>2040</td>
458
- <td>4.6</td>
459
- <td>2266</td>
460
  </tr>
461
  <tr>
462
  <td>neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w4a16</td>
463
  <td>2.75</td>
464
- <td>2.8</td>
465
- <td>1364</td>
466
- <td>12.8</td>
467
- <td>6352</td>
468
- <td>16.4</td>
469
- <td>8148</td>
470
  </tr>
471
  <tr>
472
  <th rowspan="3" valign="top">H100x4</th>
@@ -482,22 +482,22 @@ The following performance benchmarks were conducted with [vLLM](https://docs.vll
482
  <tr>
483
  <td>neuralmagic/Qwen2.5-VL-72B-Instruct-FP8-Dynamic</td>
484
  <td>1.73</td>
485
- <td>1.8</td>
486
- <td>479</td>
487
- <td>4.4</td>
488
- <td>1203</td>
489
- <td>4.8</td>
490
- <td>1296</td>
491
  </tr>
492
  <tr>
493
  <td>neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w4a16</td>
494
  <td>8.27</td>
495
- <td>13.2</td>
496
- <td>3652</td>
497
- <td>13.2</td>
498
- <td>3652</td>
499
- <td>99.2</td>
500
- <td>27108</td>
501
  </tr>
502
  </tbody>
503
  </table>
 
400
  <td>5.66</td>
401
  <td>4.3</td>
402
  <td>252</td>
403
+ <td>4.4</td>
404
+ <td>251</td>
405
+ <td>4.2</td>
406
+ <td>259</td>
407
  </tr>
408
  </tbody>
409
  </table>
 
451
  <tr>
452
  <td>neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w8a8</td>
453
  <td>1.80</td>
454
+ <td>0.6</td>
455
+ <td>289</td>
456
+ <td>2.0</td>
457
+ <td>1020</td>
458
+ <td>2.3</td>
459
+ <td>1133</td>
460
  </tr>
461
  <tr>
462
  <td>neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w4a16</td>
463
  <td>2.75</td>
464
+ <td>0.7</td>
465
+ <td>341</td>
466
+ <td>3.2</td>
467
+ <td>1588</td>
468
+ <td>4.1</td>
469
+ <td>2037</td>
470
  </tr>
471
  <tr>
472
  <th rowspan="3" valign="top">H100x4</th>
 
482
  <tr>
483
  <td>neuralmagic/Qwen2.5-VL-72B-Instruct-FP8-Dynamic</td>
484
  <td>1.73</td>
485
+ <td>0.9</td>
486
+ <td>247</td>
487
+ <td>2.2</td>
488
+ <td>621</td>
489
+ <td>2.4</td>
490
+ <td>669</td>
491
  </tr>
492
  <tr>
493
  <td>neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w4a16</td>
494
  <td>8.27</td>
495
+ <td>3.3</td>
496
+ <td>913</td>
497
+ <td>3.3</td>
498
+ <td>898</td>
499
+ <td>3.6</td>
500
+ <td>991</td>
501
  </tr>
502
  </tbody>
503
  </table>