Hugging Face's logo

genovalabs
/
Qwen2.5-VL-72B-Instruct-FP8-Dynamic

Image-to-Text
Transformers
Safetensors
English
qwen2_5_vl
vllm
vision
fp8
text-generation-inference
compressed-tensors
Model card Files Community
azgo14 commited on
Commit
26f87d1
·
verified ·
1 Parent(s): 3d8edb1

Delete files * with huggingface_hub

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. README.md +0 -450
  2. added_tokens.json +0 -24
  3. blobs/09782507221803a2287fcacb4ef166317fee2e62839ad43caa4c2f9ec461f77c +0 -3
  4. blobs/13303be6f396b038cf397b34a8096819d403836b +0 -3
  5. blobs/13bbafc2668180e8d23af7792e7a422d0e5519bc +0 -28
  6. blobs/2f452321174378d32090e7c94cffa3b43a4539e2f407f936b12a35639122005a +0 -3
  7. blobs/31349551d90c7606f325fe0f11bbb8bd5fa0d7c7 +0 -0
  8. blobs/3168a6cbc214a1e83a04235e676fad12952ef36010442e02e007701ea23037e4 +0 -3
  9. blobs/39f8a745a63bdd2552d0aaa25728108ccd66718611bb8f1b1d1998a86e5054a1 +0 -3
  10. blobs/46d25976bd3a6cb3055a1376a5a3ffec0f350d6a34a9749cbdd0495409e5e744 +0 -3
  11. blobs/4783fe10ac3adce15ac8f358ef5462739852c569 +0 -0
  12. blobs/482ced4679301bf287ebb310bdd1790eb4514232 +0 -24
  13. blobs/48dd3dd98f76920c997a634e4f390f4ab11e629c7d76d752ef94844477bea183 +0 -3
  14. blobs/4a79addd10e3e66dcc4b53822ed86a7e31197ca3a54eaacc2a8140921d158f47 +0 -3
  15. blobs/4f76e418f06f6b9b7f8a524f6405b1baee70ca7418a9a660f7537c490263ae35 +0 -3
  16. blobs/52373fe24473b1aa44333d318f578ae6bf04b49b +0 -36
  17. blobs/539e86b34d04abec945d015158b13507403596e4 +0 -0
  18. blobs/612675f4c6d89aa25675c00cb67e624892396d77 +0 -6
  19. blobs/66dcd62e9496e9b8c456f6c95df848034fe648f4414ff1088a35f7a6efa44156 +0 -3
  20. blobs/83154d2dd5e7064947716c0930013e17943c378f80343c5f6067e8afdc291c1f +0 -3
  21. blobs/87afe66c34acb2c5a7b118db74939afe8f47613f442eca4ed0f38a02e4e93fed +0 -3
  22. blobs/98d1b9f92f0154fc8f7cbfe0c6d7d3955b088cf2c520e1a41c2f157c07c5b6bf +0 -3
  23. blobs/9c5ae00e602b8860cbd784ba82a8aa14e8feecec692e7076590d014d7b7fdafa +0 -3
  24. blobs/ab1e3dfb60f12733be7aae9274996f14814cc4680a707e1a201a0b627b4373ea +0 -3
  25. blobs/ac23c0aaa2434523c494330aeb79c58395378103 +0 -31
  26. blobs/c19a13d7ad9781c9cb3df407dbd759b8eb7289ef +0 -450
  27. blobs/c4fdcf55d00ffa11e55fcc79ab5cca51d50a1d267219820ac66c36be0177fb23 +0 -3
  28. blobs/c51a8e5862935e7465e28a6fdbe227f77d15ccc3 +0 -209
  29. blobs/d17169ad645848ed1c32ef9d34529d80bbe052b3f5309d9f7973ed0fde30c09a +0 -3
  30. blobs/de3d7b2e27518f882a09cccf449e98ee4f5d72c3 +0 -255
  31. blobs/ee397dd8b35e26aea29863b307013b1bc2071c8c +0 -13
  32. blobs/f3611e6a6ce31c332c9fcbaa2d5975ad981a6aa05470b66df74dab8811f7cb0a +0 -3
  33. chat_template.json +0 -3
  34. config.json +0 -255
  35. generation_config.json +0 -13
  36. merges.txt +0 -0
  37. model-00001-of-00016.safetensors +0 -3
  38. model-00002-of-00016.safetensors +0 -3
  39. model-00003-of-00016.safetensors +0 -3
  40. model-00004-of-00016.safetensors +0 -3
  41. model-00005-of-00016.safetensors +0 -3
  42. model-00006-of-00016.safetensors +0 -3
  43. model-00007-of-00016.safetensors +0 -3
  44. model-00008-of-00016.safetensors +0 -3
  45. model-00009-of-00016.safetensors +0 -3
  46. model-00010-of-00016.safetensors +0 -3
  47. model-00011-of-00016.safetensors +0 -3
  48. model-00012-of-00016.safetensors +0 -3
  49. model-00013-of-00016.safetensors +0 -3
  50. model-00014-of-00016.safetensors +0 -3
README.md DELETED
@@ -1,450 +0,0 @@
1
- ---
2
- tags:
3
- - vllm
4
- - vision
5
- - fp8
6
- license: apache-2.0
7
- license_link: >-
8
- https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/apache-2.0.md
9
- language:
10
- - en
11
- base_model: Qwen/Qwen2.5-VL-72B-Instruct
12
- library_name: transformers
13
- ---
14
-
15
- # Qwen2.5-VL-72B-Instruct-quantized-FP8-Dynamic
16
-
17
- ## Model Overview
18
- - **Model Architecture:** Qwen2.5-VL-72B-Instruct
19
- - **Input:** Vision-Text
20
- - **Output:** Text
21
- - **Model Optimizations:**
22
- - **Weight quantization:** FP8
23
- - **Activation quantization:** FP8
24
- - **Release Date:** 2/24/2025
25
- - **Version:** 1.0
26
- - **Model Developers:** Neural Magic
27
-
28
- Quantized version of [Qwen/Qwen2.5-VL-72B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct).
29
-
30
- ### Model Optimizations
31
-
32
- This model was obtained by quantizing the weights of [Qwen/Qwen2.5-VL-72B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct) to FP8 data type, ready for inference with vLLM >= 0.5.2.
33
-
34
- ## Deployment
35
-
36
- ### Use with vLLM
37
-
38
- This model can be deployed efficiently using the [vLLM](https://docs.vllm.ai/en/latest/) backend, as shown in the example below.
39
-
40
- ```python
41
- from vllm.assets.image import ImageAsset
42
- from vllm import LLM, SamplingParams
43
-
44
- # prepare model
45
- llm = LLM(
46
- model="neuralmagic/Qwen2.5-VL-72B-Instruct-FP8-Dynamic",
47
- trust_remote_code=True,
48
- max_model_len=4096,
49
- max_num_seqs=2,
50
- )
51
-
52
- # prepare inputs
53
- question = "What is the content of this image?"
54
- inputs = {
55
- "prompt": f"<|user|>\n<|image_1|>\n{question}<|end|>\n<|assistant|>\n",
56
- "multi_modal_data": {
57
- "image": ImageAsset("cherry_blossom").pil_image.convert("RGB")
58
- },
59
- }
60
-
61
- # generate response
62
- print("========== SAMPLE GENERATION ==============")
63
- outputs = llm.generate(inputs, SamplingParams(temperature=0.2, max_tokens=64))
64
- print(f"PROMPT : {outputs[0].prompt}")
65
- print(f"RESPONSE: {outputs[0].outputs[0].text}")
66
- print("==========================================")
67
- ```
68
-
69
- vLLM also supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details.
70
-
71
- ## Creation
72
-
73
- This model was created with [llm-compressor](https://github.com/vllm-project/llm-compressor) by running the code snippet below as part a multimodal announcement blog.
74
-
75
- <details>
76
- <summary>Model Creation Code</summary>
77
-
78
- ```python
79
- import requests
80
- import torch
81
- from PIL import Image
82
- from transformers import AutoProcessor
83
- from llmcompressor.transformers import oneshot
84
- from llmcompressor.transformers.tracing import (
85
- TraceableQwen2_5_VLForConditionalGeneration,
86
- )
87
- from llmcompressor.modifiers.quantization import QuantizationModifier
88
-
89
- # Load model.
90
- model_id = Qwen/Qwen2.5-VL-72B-Instruct
91
- model = TraceableQwen2_5_VLForConditionalGeneration.from_pretrained(
92
- model_id, device_map="auto", torch_dtype="auto"
93
- )
94
- processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
95
-
96
- # Recipe
97
- recipe = [
98
- QuantizationModifier(
99
- targets="Linear",
100
- scheme="FP8_DYNAMIC",
101
- sequential_targets=["MistralDecoderLayer"],
102
- ignore=["re:.*lm_head", "re:vision_tower.*", "re:multi_modal_projector.*"],
103
- ),
104
- ]
105
-
106
- SAVE_DIR=f"{model_id.split('/')[1]}-FP8-Dynamic"
107
-
108
- # Perform oneshot
109
- oneshot(
110
- model=model,
111
- recipe=recipe,
112
- trust_remote_code_model=True,
113
- output_dir=SAVE_DIR
114
- )
115
-
116
-
117
- ```
118
- </details>
119
-
120
- ## Evaluation
121
-
122
- The model was evaluated using [mistral-evals](https://github.com/neuralmagic/mistral-evals) for vision-related tasks and using [lm_evaluation_harness](https://github.com/neuralmagic/lm-evaluation-harness) for select text-based benchmarks. The evaluations were conducted using the following commands:
123
-
124
- <details>
125
- <summary>Evaluation Commands</summary>
126
-
127
- ### Vision Tasks
128
- - vqav2
129
- - docvqa
130
- - mathvista
131
- - mmmu
132
- - chartqa
133
-
134
- ```
135
- vllm serve neuralmagic/pixtral-12b-quantized.w8a8 --tensor_parallel_size 1 --max_model_len 25000 --trust_remote_code --max_num_seqs 8 --gpu_memory_utilization 0.9 --dtype float16 --limit_mm_per_prompt image=7
136
-
137
- python -m eval.run eval_vllm \
138
- --model_name neuralmagic/pixtral-12b-quantized.w8a8 \
139
- --url http://0.0.0.0:8000 \
140
- --output_dir ~/tmp \
141
- --eval_name <vision_task_name>
142
- ```
143
-
144
- ### Text-based Tasks
145
- #### MMLU
146
-
147
- ```
148
- lm_eval \
149
- --model vllm \
150
- --model_args pretrained="<model_name>",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=<n>,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True \
151
- --tasks mmlu \
152
- --num_fewshot 5 \
153
- --batch_size auto \
154
- --output_path output_dir
155
-
156
- ```
157
-
158
- #### MGSM
159
-
160
- ```
161
- lm_eval \
162
- --model vllm \
163
- --model_args pretrained="<model_name>",dtype=auto,max_model_len=4096,max_gen_toks=2048,max_num_seqs=128,tensor_parallel_size=<n>,gpu_memory_utilization=0.9 \
164
- --tasks mgsm_cot_native \
165
- --num_fewshot 0 \
166
- --batch_size auto \
167
- --output_path output_dir
168
-
169
- ```
170
- </details>
171
-
172
- ### Accuracy
173
-
174
- <table>
175
- <thead>
176
- <tr>
177
- <th>Category</th>
178
- <th>Metric</th>
179
- <th>Qwen/Qwen2.5-VL-72B-Instruct</th>
180
- <th>neuralmagic/Qwen2.5-VL-72B-Instruct-FP8-Dynamic</th>
181
- <th>Recovery (%)</th>
182
- </tr>
183
- </thead>
184
- <tbody>
185
- <tr>
186
- <td rowspan="6"><b>Vision</b></td>
187
- <td>MMMU (val, CoT)<br><i>explicit_prompt_relaxed_correctness</i></td>
188
- <td>64.33</td>
189
- <td>66.88</td>
190
- <td>103.96%</td>
191
- </tr>
192
- <tr>
193
- <td>VQAv2 (val)<br><i>vqa_match</i></td>
194
- <td>81.94</td>
195
- <td>81.94</td>
196
- <td>100.00%</td>
197
- </tr>
198
- <tr>
199
- <td>DocVQA (val)<br><i>anls</i></td>
200
- <td>94.71</td>
201
- <td>94.64</td>
202
- <td>99.93%</td>
203
- </tr>
204
- <tr>
205
- <td>ChartQA (test, CoT)<br><i>anywhere_in_answer_relaxed_correctness</i></td>
206
- <td>88.96</td>
207
- <td>89.04</td>
208
- <td>100.09%</td>
209
- </tr>
210
- <tr>
211
- <td>Mathvista (testmini, CoT)<br><i>explicit_prompt_relaxed_correctness</i></td>
212
- <td>78.18</td>
213
- <td>77.78</td>
214
- <td>99.49%</td>
215
- </tr>
216
- <tr>
217
- <td><b>Average Score</b></td>
218
- <td><b>81.62</b></td>
219
- <td><b>81.86</b></td>
220
- <td><b>100.29%</b></td>
221
- </tr>
222
- <tr>
223
- <td rowspan="2"><b>Text</b></td>
224
- <td>MGSM (CoT)</td>
225
- <td>75.45</td>
226
- <td>49.65</td>
227
- <td>65.81%</td>
228
- </tr>
229
- <tr>
230
- <td>MMLU (5-shot)</td>
231
- <td>86.16</td>
232
- <td>86.12</td>
233
- <td>99.95%</td>
234
- </tr>
235
- </tbody>
236
- </table>
237
-
238
-
239
- ## Inference Performance
240
-
241
-
242
- This model achieves up to 1.79x speedup in single-stream deployment and up to 1.84x speedup in multi-stream asynchronous deployment, depending on hardware and use-case scenario.
243
- The following performance benchmarks were conducted with [vLLM](https://docs.vllm.ai/en/latest/) version 0.7.2, and [GuideLLM](https://github.com/neuralmagic/guidellm).
244
-
245
- <details>
246
- <summary>Benchmarking Command</summary>
247
- ```
248
- guidellm --model neuralmagic/Qwen2.5-VL-72B-Instruct-FP8-Dynamic --target "http://localhost:8000/v1" --data-type emulated --data prompt_tokens=<prompt_tokens>,generated_tokens=<generated_tokens>,images=<num_images>,width=<image_width>,height=<image_height> --max seconds 120 --backend aiohttp_server
249
- ```
250
-
251
- </details>
252
-
253
-
254
- ### Single-stream performance (measured with vLLM version 0.7.2)
255
-
256
- <table border="1" class="dataframe">
257
- <thead>
258
- <tr>
259
- <th></th>
260
- <th></th>
261
- <th></th>
262
- <th></th>
263
- <th style="text-align: center;" colspan="2" >Document Visual Question Answering<br>1680W x 2240H<br>64/128</th>
264
- <th style="text-align: center;" colspan="2" >Visual Reasoning <br>640W x 480H<br>128/128</th>
265
- <th style="text-align: center;" colspan="2" >Image Captioning<br>480W x 360H<br>0/128</th>
266
- </tr>
267
- <tr>
268
- <th>Hardware</th>
269
- <th>Number of GPUs</th>
270
- <th>Model</th>
271
- <th>Average Cost Reduction</th>
272
- <th>Latency (s)</th>
273
- <th>Queries Per Dollar</th>
274
- <th>Latency (s)th>
275
- <th>Queries Per Dollar</th>
276
- <th>Latency (s)</th>
277
- <th>Queries Per Dollar</th>
278
- </tr>
279
- </thead>
280
- <tbody>
281
- <tr>
282
- <th rowspan="3" valign="top">A100</td>
283
- <td>4</td>
284
- <td>Qwen/Qwen2.5-VL-72B-Instruct</td>
285
- <td></td>
286
- <td>6.4</td>
287
- <td>78</td>
288
- <td>4.5</td>
289
- <td>111</td>
290
- <td>4.4</td>
291
- <td>113</td>
292
- </tr>
293
- <tr>
294
- <td>2</td>
295
- <td>neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w8a8</td>
296
- <td>1.85</td>
297
- <td>7.0</td>
298
- <td>143</td>
299
- <td>4.9</td>
300
- <td>205</td>
301
- <td>4.8</td>
302
- <td>211</td>
303
- </tr>
304
- <tr>
305
- <td>1</td>
306
- <td>neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w4a16</td>
307
- <td>3.33</td>
308
- <td>9.4</td>
309
- <td>213</td>
310
- <td>5.1</td>
311
- <td>396</td>
312
- <td>4.8</td>
313
- <td>420</td>
314
- </tr>
315
- <tr>
316
- <th rowspan="3" valign="top">H100</td>
317
- <td>4</td>
318
- <td>Qwen/Qwen2.5-VL-72B-Instruct</td>
319
- <td></td>
320
- <td>4.3</td>
321
- <td>68</td>
322
- <td>3.0</td>
323
- <td>97</td>
324
- <td>2.9</td>
325
- <td>100</td>
326
- </tr>
327
- <tr>
328
- <td>2</td>
329
- <td>neuralmagic/Qwen2.5-VL-72B-Instruct-FP8-Dynamic</td>
330
- <td>1.79</td>
331
- <td>4.6</td>
332
- <td>122</td>
333
- <td>3.3</td>
334
- <td>173</td>
335
- <td>3.2</td>
336
- <td>177</td>
337
- </tr>
338
- <tr>
339
- <td>1</td>
340
- <td>neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w4a16</td>
341
- <td>5.66</td>
342
- <td>4.3</td>
343
- <td>252</td>
344
- <td>4.4</td>
345
- <td>251</td>
346
- <td>4.2</td>
347
- <td>259</td>
348
- </tr>
349
- </tbody>
350
- </table>
351
-
352
- **Use case profiles: Image Size (WxH) / prompt tokens / generation tokens
353
-
354
- **QPD: Queries per dollar, based on on-demand cost at [Lambda Labs](https://lambdalabs.com/service/gpu-cloud) (observed on 2/18/2025).
355
-
356
- ### Multi-stream asynchronous performance (measured with vLLM version 0.7.2)
357
-
358
- <table border="1" class="dataframe">
359
- <thead>
360
- <tr>
361
- <th></th>
362
- <th></th>
363
- <th></th>
364
- <th style="text-align: center;" colspan="2" >Document Visual Question Answering<br>1680W x 2240H<br>64/128</th>
365
- <th style="text-align: center;" colspan="2" >Visual Reasoning <br>640W x 480H<br>128/128</th>
366
- <th style="text-align: center;" colspan="2" >Image Captioning<br>480W x 360H<br>0/128</th>
367
- </tr>
368
- <tr>
369
- <th>Hardware</th>
370
- <th>Model</th>
371
- <th>Average Cost Reduction</th>
372
- <th>Maximum throughput (QPS)</th>
373
- <th>Queries Per Dollar</th>
374
- <th>Maximum throughput (QPS)</th>
375
- <th>Queries Per Dollar</th>
376
- <th>Maximum throughput (QPS)</th>
377
- <th>Queries Per Dollar</th>
378
- </tr>
379
- </thead>
380
- <tbody style="text-align: center">
381
- <tr>
382
- <th rowspan="3" valign="top">A100x4</th>
383
- <td>Qwen/Qwen2.5-VL-72B-Instruct</td>
384
- <td></td>
385
- <td>0.4</td>
386
- <td>180</td>
387
- <td>1.1</td>
388
- <td>539</td>
389
- <td>1.2</td>
390
- <td>595</td>
391
- </tr>
392
- <tr>
393
- <td>neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w8a8</td>
394
- <td>1.80</td>
395
- <td>0.6</td>
396
- <td>289</td>
397
- <td>2.0</td>
398
- <td>1020</td>
399
- <td>2.3</td>
400
- <td>1133</td>
401
- </tr>
402
- <tr>
403
- <td>neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w4a16</td>
404
- <td>2.75</td>
405
- <td>0.7</td>
406
- <td>341</td>
407
- <td>3.2</td>
408
- <td>1588</td>
409
- <td>4.1</td>
410
- <td>2037</td>
411
- </tr>
412
- <tr>
413
- <th rowspan="3" valign="top">H100x4</th>
414
- <td>Qwen/Qwen2.5-VL-72B-Instruct</td>
415
- <td></td>
416
- <td>0.5</td>
417
- <td>134</td>
418
- <td>1.2</td>
419
- <td>357</td>
420
- <td>1.3</td>
421
- <td>379</td>
422
- </tr>
423
- <tr>
424
- <td>neuralmagic/Qwen2.5-VL-72B-Instruct-FP8-Dynamic</td>
425
- <td>1.73</td>
426
- <td>0.9</td>
427
- <td>247</td>
428
- <td>2.2</td>
429
- <td>621</td>
430
- <td>2.4</td>
431
- <td>669</td>
432
- </tr>
433
- <tr>
434
- <td>neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w4a16</td>
435
- <td>8.27</td>
436
- <td>3.3</td>
437
- <td>913</td>
438
- <td>3.3</td>
439
- <td>898</td>
440
- <td>3.6</td>
441
- <td>991</td>
442
- </tr>
443
- </tbody>
444
- </table>
445
-
446
- **Use case profiles: Image Size (WxH) / prompt tokens / generation tokens
447
-
448
- **QPS: Queries per second.
449
-
450
- **QPD: Queries per dollar, based on on-demand cost at [Lambda Labs](https://lambdalabs.com/service/gpu-cloud) (observed on 2/18/2025).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
added_tokens.json DELETED
@@ -1,24 +0,0 @@
1
- {
2
- "</tool_call>": 151658,
3
- "<tool_call>": 151657,
4
- "<|box_end|>": 151649,
5
- "<|box_start|>": 151648,
6
- "<|endoftext|>": 151643,
7
- "<|file_sep|>": 151664,
8
- "<|fim_middle|>": 151660,
9
- "<|fim_pad|>": 151662,
10
- "<|fim_prefix|>": 151659,
11
- "<|fim_suffix|>": 151661,
12
- "<|im_end|>": 151645,
13
- "<|im_start|>": 151644,
14
- "<|image_pad|>": 151655,
15
- "<|object_ref_end|>": 151647,
16
- "<|object_ref_start|>": 151646,
17
- "<|quad_end|>": 151651,
18
- "<|quad_start|>": 151650,
19
- "<|repo_name|>": 151663,
20
- "<|video_pad|>": 151656,
21
- "<|vision_end|>": 151653,
22
- "<|vision_pad|>": 151654,
23
- "<|vision_start|>": 151652
24
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
blobs/09782507221803a2287fcacb4ef166317fee2e62839ad43caa4c2f9ec461f77c DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:09782507221803a2287fcacb4ef166317fee2e62839ad43caa4c2f9ec461f77c
3
- size 4873986032
 
 
 
 
blobs/13303be6f396b038cf397b34a8096819d403836b DELETED
@@ -1,3 +0,0 @@
1
- {
2
- "chat_template": "{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n{% endif %}<|im_start|>{{ message['role'] }}\n{% if message['content'] is string %}{{ message['content'] }}<|im_end|>\n{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}"
3
- }
 
 
 
 
blobs/13bbafc2668180e8d23af7792e7a422d0e5519bc DELETED
@@ -1,28 +0,0 @@
1
- {
2
- "do_convert_rgb": true,
3
- "do_normalize": true,
4
- "do_rescale": true,
5
- "do_resize": true,
6
- "image_mean": [
7
- 0.48145466,
8
- 0.4578275,
9
- 0.40821073
10
- ],
11
- "image_std": [
12
- 0.26862954,
13
- 0.26130258,
14
- 0.27577711
15
- ],
16
- "max_pixels": 12845056,
17
- "merge_size": 2,
18
- "min_pixels": 3136,
19
- "patch_size": 14,
20
- "processor_class": "Qwen2VLImageProcessor",
21
- "resample": 3,
22
- "rescale_factor": 0.00392156862745098,
23
- "size": {
24
- "longest_edge": 12845056,
25
- "shortest_edge": 3136
26
- },
27
- "temporal_patch_size": 2
28
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
blobs/2f452321174378d32090e7c94cffa3b43a4539e2f407f936b12a35639122005a DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:2f452321174378d32090e7c94cffa3b43a4539e2f407f936b12a35639122005a
3
- size 4782749368
 
 
 
 
blobs/31349551d90c7606f325fe0f11bbb8bd5fa0d7c7 DELETED
The diff for this file is too large to render. See raw diff
 
blobs/3168a6cbc214a1e83a04235e676fad12952ef36010442e02e007701ea23037e4 DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:3168a6cbc214a1e83a04235e676fad12952ef36010442e02e007701ea23037e4
3
- size 4873986032
 
 
 
 
blobs/39f8a745a63bdd2552d0aaa25728108ccd66718611bb8f1b1d1998a86e5054a1 DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:39f8a745a63bdd2552d0aaa25728108ccd66718611bb8f1b1d1998a86e5054a1
3
- size 4873986032
 
 
 
 
blobs/46d25976bd3a6cb3055a1376a5a3ffec0f350d6a34a9749cbdd0495409e5e744 DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:46d25976bd3a6cb3055a1376a5a3ffec0f350d6a34a9749cbdd0495409e5e744
3
- size 4873986032
 
 
 
 
blobs/4783fe10ac3adce15ac8f358ef5462739852c569 DELETED
The diff for this file is too large to render. See raw diff
 
blobs/482ced4679301bf287ebb310bdd1790eb4514232 DELETED
@@ -1,24 +0,0 @@
1
- {
2
- "</tool_call>": 151658,
3
- "<tool_call>": 151657,
4
- "<|box_end|>": 151649,
5
- "<|box_start|>": 151648,
6
- "<|endoftext|>": 151643,
7
- "<|file_sep|>": 151664,
8
- "<|fim_middle|>": 151660,
9
- "<|fim_pad|>": 151662,
10
- "<|fim_prefix|>": 151659,
11
- "<|fim_suffix|>": 151661,
12
- "<|im_end|>": 151645,
13
- "<|im_start|>": 151644,
14
- "<|image_pad|>": 151655,
15
- "<|object_ref_end|>": 151647,
16
- "<|object_ref_start|>": 151646,
17
- "<|quad_end|>": 151651,
18
- "<|quad_start|>": 151650,
19
- "<|repo_name|>": 151663,
20
- "<|video_pad|>": 151656,
21
- "<|vision_end|>": 151653,
22
- "<|vision_pad|>": 151654,
23
- "<|vision_start|>": 151652
24
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
blobs/48dd3dd98f76920c997a634e4f390f4ab11e629c7d76d752ef94844477bea183 DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:48dd3dd98f76920c997a634e4f390f4ab11e629c7d76d752ef94844477bea183
3
- size 4782749368
 
 
 
 
blobs/4a79addd10e3e66dcc4b53822ed86a7e31197ca3a54eaacc2a8140921d158f47 DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:4a79addd10e3e66dcc4b53822ed86a7e31197ca3a54eaacc2a8140921d158f47
3
- size 4782749368
 
 
 
 
blobs/4f76e418f06f6b9b7f8a524f6405b1baee70ca7418a9a660f7537c490263ae35 DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:4f76e418f06f6b9b7f8a524f6405b1baee70ca7418a9a660f7537c490263ae35
3
- size 4782749304
 
 
 
 
blobs/52373fe24473b1aa44333d318f578ae6bf04b49b DELETED
@@ -1,36 +0,0 @@
1
- *.7z filter=lfs diff=lfs merge=lfs -text
2
- *.arrow filter=lfs diff=lfs merge=lfs -text
3
- *.bin filter=lfs diff=lfs merge=lfs -text
4
- *.bz2 filter=lfs diff=lfs merge=lfs -text
5
- *.ckpt filter=lfs diff=lfs merge=lfs -text
6
- *.ftz filter=lfs diff=lfs merge=lfs -text
7
- *.gz filter=lfs diff=lfs merge=lfs -text
8
- *.h5 filter=lfs diff=lfs merge=lfs -text
9
- *.joblib filter=lfs diff=lfs merge=lfs -text
10
- *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
- *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
- *.model filter=lfs diff=lfs merge=lfs -text
13
- *.msgpack filter=lfs diff=lfs merge=lfs -text
14
- *.npy filter=lfs diff=lfs merge=lfs -text
15
- *.npz filter=lfs diff=lfs merge=lfs -text
16
- *.onnx filter=lfs diff=lfs merge=lfs -text
17
- *.ot filter=lfs diff=lfs merge=lfs -text
18
- *.parquet filter=lfs diff=lfs merge=lfs -text
19
- *.pb filter=lfs diff=lfs merge=lfs -text
20
- *.pickle filter=lfs diff=lfs merge=lfs -text
21
- *.pkl filter=lfs diff=lfs merge=lfs -text
22
- *.pt filter=lfs diff=lfs merge=lfs -text
23
- *.pth filter=lfs diff=lfs merge=lfs -text
24
- *.rar filter=lfs diff=lfs merge=lfs -text
25
- *.safetensors filter=lfs diff=lfs merge=lfs -text
26
- saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
- *.tar.* filter=lfs diff=lfs merge=lfs -text
28
- *.tar filter=lfs diff=lfs merge=lfs -text
29
- *.tflite filter=lfs diff=lfs merge=lfs -text
30
- *.tgz filter=lfs diff=lfs merge=lfs -text
31
- *.wasm filter=lfs diff=lfs merge=lfs -text
32
- *.xz filter=lfs diff=lfs merge=lfs -text
33
- *.zip filter=lfs diff=lfs merge=lfs -text
34
- *.zst filter=lfs diff=lfs merge=lfs -text
35
- *tfevents* filter=lfs diff=lfs merge=lfs -text
36
- tokenizer.json filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
blobs/539e86b34d04abec945d015158b13507403596e4 DELETED
The diff for this file is too large to render. See raw diff
 
blobs/612675f4c6d89aa25675c00cb67e624892396d77 DELETED
@@ -1,6 +0,0 @@
1
- DEFAULT_stage:
2
- DEFAULT_modifiers:
3
- QuantizationModifier:
4
- ignore: ['re:.*lm_head', 're:visual.*']
5
- targets: [Linear]
6
- scheme: FP8_DYNAMIC
 
 
 
 
 
 
 
blobs/66dcd62e9496e9b8c456f6c95df848034fe648f4414ff1088a35f7a6efa44156 DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:66dcd62e9496e9b8c456f6c95df848034fe648f4414ff1088a35f7a6efa44156
3
- size 4873985928
 
 
 
 
blobs/83154d2dd5e7064947716c0930013e17943c378f80343c5f6067e8afdc291c1f DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:83154d2dd5e7064947716c0930013e17943c378f80343c5f6067e8afdc291c1f
3
- size 4873986032
 
 
 
 
blobs/87afe66c34acb2c5a7b118db74939afe8f47613f442eca4ed0f38a02e4e93fed DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:87afe66c34acb2c5a7b118db74939afe8f47613f442eca4ed0f38a02e4e93fed
3
- size 4873986032
 
 
 
 
blobs/98d1b9f92f0154fc8f7cbfe0c6d7d3955b088cf2c520e1a41c2f157c07c5b6bf DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:98d1b9f92f0154fc8f7cbfe0c6d7d3955b088cf2c520e1a41c2f157c07c5b6bf
3
- size 4782749368
 
 
 
 
blobs/9c5ae00e602b8860cbd784ba82a8aa14e8feecec692e7076590d014d7b7fdafa DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:9c5ae00e602b8860cbd784ba82a8aa14e8feecec692e7076590d014d7b7fdafa
3
- size 11421896
 
 
 
 
blobs/ab1e3dfb60f12733be7aae9274996f14814cc4680a707e1a201a0b627b4373ea DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:ab1e3dfb60f12733be7aae9274996f14814cc4680a707e1a201a0b627b4373ea
3
- size 4782749368
 
 
 
 
blobs/ac23c0aaa2434523c494330aeb79c58395378103 DELETED
@@ -1,31 +0,0 @@
1
- {
2
- "additional_special_tokens": [
3
- "<|im_start|>",
4
- "<|im_end|>",
5
- "<|object_ref_start|>",
6
- "<|object_ref_end|>",
7
- "<|box_start|>",
8
- "<|box_end|>",
9
- "<|quad_start|>",
10
- "<|quad_end|>",
11
- "<|vision_start|>",
12
- "<|vision_end|>",
13
- "<|vision_pad|>",
14
- "<|image_pad|>",
15
- "<|video_pad|>"
16
- ],
17
- "eos_token": {
18
- "content": "<|im_end|>",
19
- "lstrip": false,
20
- "normalized": false,
21
- "rstrip": false,
22
- "single_word": false
23
- },
24
- "pad_token": {
25
- "content": "<|endoftext|>",
26
- "lstrip": false,
27
- "normalized": false,
28
- "rstrip": false,
29
- "single_word": false
30
- }
31
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
blobs/c19a13d7ad9781c9cb3df407dbd759b8eb7289ef DELETED
@@ -1,450 +0,0 @@
1
- ---
2
- tags:
3
- - vllm
4
- - vision
5
- - fp8
6
- license: apache-2.0
7
- license_link: >-
8
- https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/apache-2.0.md
9
- language:
10
- - en
11
- base_model: Qwen/Qwen2.5-VL-72B-Instruct
12
- library_name: transformers
13
- ---
14
-
15
- # Qwen2.5-VL-72B-Instruct-quantized-FP8-Dynamic
16
-
17
- ## Model Overview
18
- - **Model Architecture:** Qwen2.5-VL-72B-Instruct
19
- - **Input:** Vision-Text
20
- - **Output:** Text
21
- - **Model Optimizations:**
22
- - **Weight quantization:** FP8
23
- - **Activation quantization:** FP8
24
- - **Release Date:** 2/24/2025
25
- - **Version:** 1.0
26
- - **Model Developers:** Neural Magic
27
-
28
- Quantized version of [Qwen/Qwen2.5-VL-72B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct).
29
-
30
- ### Model Optimizations
31
-
32
- This model was obtained by quantizing the weights of [Qwen/Qwen2.5-VL-72B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct) to FP8 data type, ready for inference with vLLM >= 0.5.2.
33
-
34
- ## Deployment
35
-
36
- ### Use with vLLM
37
-
38
- This model can be deployed efficiently using the [vLLM](https://docs.vllm.ai/en/latest/) backend, as shown in the example below.
39
-
40
- ```python
41
- from vllm.assets.image import ImageAsset
42
- from vllm import LLM, SamplingParams
43
-
44
- # prepare model
45
- llm = LLM(
46
- model="neuralmagic/Qwen2.5-VL-72B-Instruct-FP8-Dynamic",
47
- trust_remote_code=True,
48
- max_model_len=4096,
49
- max_num_seqs=2,
50
- )
51
-
52
- # prepare inputs
53
- question = "What is the content of this image?"
54
- inputs = {
55
- "prompt": f"<|user|>\n<|image_1|>\n{question}<|end|>\n<|assistant|>\n",
56
- "multi_modal_data": {
57
- "image": ImageAsset("cherry_blossom").pil_image.convert("RGB")
58
- },
59
- }
60
-
61
- # generate response
62
- print("========== SAMPLE GENERATION ==============")
63
- outputs = llm.generate(inputs, SamplingParams(temperature=0.2, max_tokens=64))
64
- print(f"PROMPT : {outputs[0].prompt}")
65
- print(f"RESPONSE: {outputs[0].outputs[0].text}")
66
- print("==========================================")
67
- ```
68
-
69
- vLLM also supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details.
70
-
71
- ## Creation
72
-
73
- This model was created with [llm-compressor](https://github.com/vllm-project/llm-compressor) by running the code snippet below as part a multimodal announcement blog.
74
-
75
- <details>
76
- <summary>Model Creation Code</summary>
77
-
78
- ```python
79
- import requests
80
- import torch
81
- from PIL import Image
82
- from transformers import AutoProcessor
83
- from llmcompressor.transformers import oneshot
84
- from llmcompressor.transformers.tracing import (
85
- TraceableQwen2_5_VLForConditionalGeneration,
86
- )
87
- from llmcompressor.modifiers.quantization import QuantizationModifier
88
-
89
- # Load model.
90
- model_id = Qwen/Qwen2.5-VL-72B-Instruct
91
- model = TraceableQwen2_5_VLForConditionalGeneration.from_pretrained(
92
- model_id, device_map="auto", torch_dtype="auto"
93
- )
94
- processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
95
-
96
- # Recipe
97
- recipe = [
98
- QuantizationModifier(
99
- targets="Linear",
100
- scheme="FP8_DYNAMIC",
101
- sequential_targets=["MistralDecoderLayer"],
102
- ignore=["re:.*lm_head", "re:vision_tower.*", "re:multi_modal_projector.*"],
103
- ),
104
- ]
105
-
106
- SAVE_DIR=f"{model_id.split('/')[1]}-FP8-Dynamic"
107
-
108
- # Perform oneshot
109
- oneshot(
110
- model=model,
111
- recipe=recipe,
112
- trust_remote_code_model=True,
113
- output_dir=SAVE_DIR
114
- )
115
-
116
-
117
- ```
118
- </details>
119
-
120
- ## Evaluation
121
-
122
- The model was evaluated using [mistral-evals](https://github.com/neuralmagic/mistral-evals) for vision-related tasks and using [lm_evaluation_harness](https://github.com/neuralmagic/lm-evaluation-harness) for select text-based benchmarks. The evaluations were conducted using the following commands:
123
-
124
- <details>
125
- <summary>Evaluation Commands</summary>
126
-
127
- ### Vision Tasks
128
- - vqav2
129
- - docvqa
130
- - mathvista
131
- - mmmu
132
- - chartqa
133
-
134
- ```
135
- vllm serve neuralmagic/pixtral-12b-quantized.w8a8 --tensor_parallel_size 1 --max_model_len 25000 --trust_remote_code --max_num_seqs 8 --gpu_memory_utilization 0.9 --dtype float16 --limit_mm_per_prompt image=7
136
-
137
- python -m eval.run eval_vllm \
138
- --model_name neuralmagic/pixtral-12b-quantized.w8a8 \
139
- --url http://0.0.0.0:8000 \
140
- --output_dir ~/tmp \
141
- --eval_name <vision_task_name>
142
- ```
143
-
144
- ### Text-based Tasks
145
- #### MMLU
146
-
147
- ```
148
- lm_eval \
149
- --model vllm \
150
- --model_args pretrained="<model_name>",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=<n>,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True \
151
- --tasks mmlu \
152
- --num_fewshot 5 \
153
- --batch_size auto \
154
- --output_path output_dir
155
-
156
- ```
157
-
158
- #### MGSM
159
-
160
- ```
161
- lm_eval \
162
- --model vllm \
163
- --model_args pretrained="<model_name>",dtype=auto,max_model_len=4096,max_gen_toks=2048,max_num_seqs=128,tensor_parallel_size=<n>,gpu_memory_utilization=0.9 \
164
- --tasks mgsm_cot_native \
165
- --num_fewshot 0 \
166
- --batch_size auto \
167
- --output_path output_dir
168
-
169
- ```
170
- </details>
171
-
172
- ### Accuracy
173
-
174
- <table>
175
- <thead>
176
- <tr>
177
- <th>Category</th>
178
- <th>Metric</th>
179
- <th>Qwen/Qwen2.5-VL-72B-Instruct</th>
180
- <th>neuralmagic/Qwen2.5-VL-72B-Instruct-FP8-Dynamic</th>
181
- <th>Recovery (%)</th>
182
- </tr>
183
- </thead>
184
- <tbody>
185
- <tr>
186
- <td rowspan="6"><b>Vision</b></td>
187
- <td>MMMU (val, CoT)<br><i>explicit_prompt_relaxed_correctness</i></td>
188
- <td>64.33</td>
189
- <td>66.88</td>
190
- <td>103.96%</td>
191
- </tr>
192
- <tr>
193
- <td>VQAv2 (val)<br><i>vqa_match</i></td>
194
- <td>81.94</td>
195
- <td>81.94</td>
196
- <td>100.00%</td>
197
- </tr>
198
- <tr>
199
- <td>DocVQA (val)<br><i>anls</i></td>
200
- <td>94.71</td>
201
- <td>94.64</td>
202
- <td>99.93%</td>
203
- </tr>
204
- <tr>
205
- <td>ChartQA (test, CoT)<br><i>anywhere_in_answer_relaxed_correctness</i></td>
206
- <td>88.96</td>
207
- <td>89.04</td>
208
- <td>100.09%</td>
209
- </tr>
210
- <tr>
211
- <td>Mathvista (testmini, CoT)<br><i>explicit_prompt_relaxed_correctness</i></td>
212
- <td>78.18</td>
213
- <td>77.78</td>
214
- <td>99.49%</td>
215
- </tr>
216
- <tr>
217
- <td><b>Average Score</b></td>
218
- <td><b>81.62</b></td>
219
- <td><b>81.86</b></td>
220
- <td><b>100.29%</b></td>
221
- </tr>
222
- <tr>
223
- <td rowspan="2"><b>Text</b></td>
224
- <td>MGSM (CoT)</td>
225
- <td>75.45</td>
226
- <td>49.65</td>
227
- <td>65.81%</td>
228
- </tr>
229
- <tr>
230
- <td>MMLU (5-shot)</td>
231
- <td>86.16</td>
232
- <td>86.12</td>
233
- <td>99.95%</td>
234
- </tr>
235
- </tbody>
236
- </table>
237
-
238
-
239
- ## Inference Performance
240
-
241
-
242
- This model achieves up to 1.79x speedup in single-stream deployment and up to 1.84x speedup in multi-stream asynchronous deployment, depending on hardware and use-case scenario.
243
- The following performance benchmarks were conducted with [vLLM](https://docs.vllm.ai/en/latest/) version 0.7.2, and [GuideLLM](https://github.com/neuralmagic/guidellm).
244
-
245
- <details>
246
- <summary>Benchmarking Command</summary>
247
- ```
248
- guidellm --model neuralmagic/Qwen2.5-VL-72B-Instruct-FP8-Dynamic --target "http://localhost:8000/v1" --data-type emulated --data prompt_tokens=<prompt_tokens>,generated_tokens=<generated_tokens>,images=<num_images>,width=<image_width>,height=<image_height> --max seconds 120 --backend aiohttp_server
249
- ```
250
-
251
- </details>
252
-
253
-
254
- ### Single-stream performance (measured with vLLM version 0.7.2)
255
-
256
- <table border="1" class="dataframe">
257
- <thead>
258
- <tr>
259
- <th></th>
260
- <th></th>
261
- <th></th>
262
- <th></th>
263
- <th style="text-align: center;" colspan="2" >Document Visual Question Answering<br>1680W x 2240H<br>64/128</th>
264
- <th style="text-align: center;" colspan="2" >Visual Reasoning <br>640W x 480H<br>128/128</th>
265
- <th style="text-align: center;" colspan="2" >Image Captioning<br>480W x 360H<br>0/128</th>
266
- </tr>
267
- <tr>
268
- <th>Hardware</th>
269
- <th>Number of GPUs</th>
270
- <th>Model</th>
271
- <th>Average Cost Reduction</th>
272
- <th>Latency (s)</th>
273
- <th>Queries Per Dollar</th>
274
- <th>Latency (s)th>
275
- <th>Queries Per Dollar</th>
276
- <th>Latency (s)</th>
277
- <th>Queries Per Dollar</th>
278
- </tr>
279
- </thead>
280
- <tbody>
281
- <tr>
282
- <th rowspan="3" valign="top">A100</td>
283
- <td>4</td>
284
- <td>Qwen/Qwen2.5-VL-72B-Instruct</td>
285
- <td></td>
286
- <td>6.4</td>
287
- <td>78</td>
288
- <td>4.5</td>
289
- <td>111</td>
290
- <td>4.4</td>
291
- <td>113</td>
292
- </tr>
293
- <tr>
294
- <td>2</td>
295
- <td>neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w8a8</td>
296
- <td>1.85</td>
297
- <td>7.0</td>
298
- <td>143</td>
299
- <td>4.9</td>
300
- <td>205</td>
301
- <td>4.8</td>
302
- <td>211</td>
303
- </tr>
304
- <tr>
305
- <td>1</td>
306
- <td>neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w4a16</td>
307
- <td>3.33</td>
308
- <td>9.4</td>
309
- <td>213</td>
310
- <td>5.1</td>
311
- <td>396</td>
312
- <td>4.8</td>
313
- <td>420</td>
314
- </tr>
315
- <tr>
316
- <th rowspan="3" valign="top">H100</td>
317
- <td>4</td>
318
- <td>Qwen/Qwen2.5-VL-72B-Instruct</td>
319
- <td></td>
320
- <td>4.3</td>
321
- <td>68</td>
322
- <td>3.0</td>
323
- <td>97</td>
324
- <td>2.9</td>
325
- <td>100</td>
326
- </tr>
327
- <tr>
328
- <td>2</td>
329
- <td>neuralmagic/Qwen2.5-VL-72B-Instruct-FP8-Dynamic</td>
330
- <td>1.79</td>
331
- <td>4.6</td>
332
- <td>122</td>
333
- <td>3.3</td>
334
- <td>173</td>
335
- <td>3.2</td>
336
- <td>177</td>
337
- </tr>
338
- <tr>
339
- <td>1</td>
340
- <td>neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w4a16</td>
341
- <td>5.66</td>
342
- <td>4.3</td>
343
- <td>252</td>
344
- <td>4.4</td>
345
- <td>251</td>
346
- <td>4.2</td>
347
- <td>259</td>
348
- </tr>
349
- </tbody>
350
- </table>
351
-
352
- **Use case profiles: Image Size (WxH) / prompt tokens / generation tokens
353
-
354
- **QPD: Queries per dollar, based on on-demand cost at [Lambda Labs](https://lambdalabs.com/service/gpu-cloud) (observed on 2/18/2025).
355
-
356
- ### Multi-stream asynchronous performance (measured with vLLM version 0.7.2)
357
-
358
- <table border="1" class="dataframe">
359
- <thead>
360
- <tr>
361
- <th></th>
362
- <th></th>
363
- <th></th>
364
- <th style="text-align: center;" colspan="2" >Document Visual Question Answering<br>1680W x 2240H<br>64/128</th>
365
- <th style="text-align: center;" colspan="2" >Visual Reasoning <br>640W x 480H<br>128/128</th>
366
- <th style="text-align: center;" colspan="2" >Image Captioning<br>480W x 360H<br>0/128</th>
367
- </tr>
368
- <tr>
369
- <th>Hardware</th>
370
- <th>Model</th>
371
- <th>Average Cost Reduction</th>
372
- <th>Maximum throughput (QPS)</th>
373
- <th>Queries Per Dollar</th>
374
- <th>Maximum throughput (QPS)</th>
375
- <th>Queries Per Dollar</th>
376
- <th>Maximum throughput (QPS)</th>
377
- <th>Queries Per Dollar</th>
378
- </tr>
379
- </thead>
380
- <tbody style="text-align: center">
381
- <tr>
382
- <th rowspan="3" valign="top">A100x4</th>
383
- <td>Qwen/Qwen2.5-VL-72B-Instruct</td>
384
- <td></td>
385
- <td>0.4</td>
386
- <td>180</td>
387
- <td>1.1</td>
388
- <td>539</td>
389
- <td>1.2</td>
390
- <td>595</td>
391
- </tr>
392
- <tr>
393
- <td>neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w8a8</td>
394
- <td>1.80</td>
395
- <td>0.6</td>
396
- <td>289</td>
397
- <td>2.0</td>
398
- <td>1020</td>
399
- <td>2.3</td>
400
- <td>1133</td>
401
- </tr>
402
- <tr>
403
- <td>neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w4a16</td>
404
- <td>2.75</td>
405
- <td>0.7</td>
406
- <td>341</td>
407
- <td>3.2</td>
408
- <td>1588</td>
409
- <td>4.1</td>
410
- <td>2037</td>
411
- </tr>
412
- <tr>
413
- <th rowspan="3" valign="top">H100x4</th>
414
- <td>Qwen/Qwen2.5-VL-72B-Instruct</td>
415
- <td></td>
416
- <td>0.5</td>
417
- <td>134</td>
418
- <td>1.2</td>
419
- <td>357</td>
420
- <td>1.3</td>
421
- <td>379</td>
422
- </tr>
423
- <tr>
424
- <td>neuralmagic/Qwen2.5-VL-72B-Instruct-FP8-Dynamic</td>
425
- <td>1.73</td>
426
- <td>0.9</td>
427
- <td>247</td>
428
- <td>2.2</td>
429
- <td>621</td>
430
- <td>2.4</td>
431
- <td>669</td>
432
- </tr>
433
- <tr>
434
- <td>neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w4a16</td>
435
- <td>8.27</td>
436
- <td>3.3</td>
437
- <td>913</td>
438
- <td>3.3</td>
439
- <td>898</td>
440
- <td>3.6</td>
441
- <td>991</td>
442
- </tr>
443
- </tbody>
444
- </table>
445
-
446
- **Use case profiles: Image Size (WxH) / prompt tokens / generation tokens
447
-
448
- **QPS: Queries per second.
449
-
450
- **QPD: Queries per dollar, based on on-demand cost at [Lambda Labs](https://lambdalabs.com/service/gpu-cloud) (observed on 2/18/2025).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
blobs/c4fdcf55d00ffa11e55fcc79ab5cca51d50a1d267219820ac66c36be0177fb23 DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:c4fdcf55d00ffa11e55fcc79ab5cca51d50a1d267219820ac66c36be0177fb23
3
- size 4929545176
 
 
 
 
blobs/c51a8e5862935e7465e28a6fdbe227f77d15ccc3 DELETED
@@ -1,209 +0,0 @@
1
- {
2
- "add_bos_token": false,
3
- "add_prefix_space": false,
4
- "added_tokens_decoder": {
5
- "151643": {
6
- "content": "<|endoftext|>",
7
- "lstrip": false,
8
- "normalized": false,
9
- "rstrip": false,
10
- "single_word": false,
11
- "special": true
12
- },
13
- "151644": {
14
- "content": "<|im_start|>",
15
- "lstrip": false,
16
- "normalized": false,
17
- "rstrip": false,
18
- "single_word": false,
19
- "special": true
20
- },
21
- "151645": {
22
- "content": "<|im_end|>",
23
- "lstrip": false,
24
- "normalized": false,
25
- "rstrip": false,
26
- "single_word": false,
27
- "special": true
28
- },
29
- "151646": {
30
- "content": "<|object_ref_start|>",
31
- "lstrip": false,
32
- "normalized": false,
33
- "rstrip": false,
34
- "single_word": false,
35
- "special": true
36
- },
37
- "151647": {
38
- "content": "<|object_ref_end|>",
39
- "lstrip": false,
40
- "normalized": false,
41
- "rstrip": false,
42
- "single_word": false,
43
- "special": true
44
- },
45
- "151648": {
46
- "content": "<|box_start|>",
47
- "lstrip": false,
48
- "normalized": false,
49
- "rstrip": false,
50
- "single_word": false,
51
- "special": true
52
- },
53
- "151649": {
54
- "content": "<|box_end|>",
55
- "lstrip": false,
56
- "normalized": false,
57
- "rstrip": false,
58
- "single_word": false,
59
- "special": true
60
- },
61
- "151650": {
62
- "content": "<|quad_start|>",
63
- "lstrip": false,
64
- "normalized": false,
65
- "rstrip": false,
66
- "single_word": false,
67
- "special": true
68
- },
69
- "151651": {
70
- "content": "<|quad_end|>",
71
- "lstrip": false,
72
- "normalized": false,
73
- "rstrip": false,
74
- "single_word": false,
75
- "special": true
76
- },
77
- "151652": {
78
- "content": "<|vision_start|>",
79
- "lstrip": false,
80
- "normalized": false,
81
- "rstrip": false,
82
- "single_word": false,
83
- "special": true
84
- },
85
- "151653": {
86
- "content": "<|vision_end|>",
87
- "lstrip": false,
88
- "normalized": false,
89
- "rstrip": false,
90
- "single_word": false,
91
- "special": true
92
- },
93
- "151654": {
94
- "content": "<|vision_pad|>",
95
- "lstrip": false,
96
- "normalized": false,
97
- "rstrip": false,
98
- "single_word": false,
99
- "special": true
100
- },
101
- "151655": {
102
- "content": "<|image_pad|>",
103
- "lstrip": false,
104
- "normalized": false,
105
- "rstrip": false,
106
- "single_word": false,
107
- "special": true
108
- },
109
- "151656": {
110
- "content": "<|video_pad|>",
111
- "lstrip": false,
112
- "normalized": false,
113
- "rstrip": false,
114
- "single_word": false,
115
- "special": true
116
- },
117
- "151657": {
118
- "content": "<tool_call>",
119
- "lstrip": false,
120
- "normalized": false,
121
- "rstrip": false,
122
- "single_word": false,
123
- "special": false
124
- },
125
- "151658": {
126
- "content": "</tool_call>",
127
- "lstrip": false,
128
- "normalized": false,
129
- "rstrip": false,
130
- "single_word": false,
131
- "special": false
132
- },
133
- "151659": {
134
- "content": "<|fim_prefix|>",
135
- "lstrip": false,
136
- "normalized": false,
137
- "rstrip": false,
138
- "single_word": false,
139
- "special": false
140
- },
141
- "151660": {
142
- "content": "<|fim_middle|>",
143
- "lstrip": false,
144
- "normalized": false,
145
- "rstrip": false,
146
- "single_word": false,
147
- "special": false
148
- },
149
- "151661": {
150
- "content": "<|fim_suffix|>",
151
- "lstrip": false,
152
- "normalized": false,
153
- "rstrip": false,
154
- "single_word": false,
155
- "special": false
156
- },
157
- "151662": {
158
- "content": "<|fim_pad|>",
159
- "lstrip": false,
160
- "normalized": false,
161
- "rstrip": false,
162
- "single_word": false,
163
- "special": false
164
- },
165
- "151663": {
166
- "content": "<|repo_name|>",
167
- "lstrip": false,
168
- "normalized": false,
169
- "rstrip": false,
170
- "single_word": false,
171
- "special": false
172
- },
173
- "151664": {
174
- "content": "<|file_sep|>",
175
- "lstrip": false,
176
- "normalized": false,
177
- "rstrip": false,
178
- "single_word": false,
179
- "special": false
180
- }
181
- },
182
- "additional_special_tokens": [
183
- "<|im_start|>",
184
- "<|im_end|>",
185
- "<|object_ref_start|>",
186
- "<|object_ref_end|>",
187
- "<|box_start|>",
188
- "<|box_end|>",
189
- "<|quad_start|>",
190
- "<|quad_end|>",
191
- "<|vision_start|>",
192
- "<|vision_end|>",
193
- "<|vision_pad|>",
194
- "<|image_pad|>",
195
- "<|video_pad|>"
196
- ],
197
- "bos_token": null,
198
- "chat_template": "{%- if tools %}\n {{- '<|im_start|>system\\n' }}\n {%- if messages[0]['role'] == 'system' %}\n {{- messages[0]['content'] }}\n {%- else %}\n {{- 'You are a helpful assistant.' }}\n {%- endif %}\n {{- \"\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n {%- for tool in tools %}\n {{- \"\\n\" }}\n {{- tool | tojson }}\n {%- endfor %}\n {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n {%- if messages[0]['role'] == 'system' %}\n {{- '<|im_start|>system\\n' + messages[0]['content'] + '<|im_end|>\\n' }}\n {%- else %}\n {{- '<|im_start|>system\\nYou are a helpful assistant.<|im_end|>\\n' }}\n {%- endif %}\n{%- endif %}\n{%- for message in messages %}\n {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) or (message.role == \"assistant\" and not message.tool_calls) %}\n {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n {%- elif message.role == \"assistant\" %}\n {{- '<|im_start|>' + message.role }}\n {%- if message.content %}\n {{- '\\n' + message.content }}\n {%- endif %}\n {%- for tool_call in message.tool_calls %}\n {%- if tool_call.function is defined %}\n {%- set tool_call = tool_call.function %}\n {%- endif %}\n {{- '\\n<tool_call>\\n{\"name\": \"' }}\n {{- tool_call.name }}\n {{- '\", \"arguments\": ' }}\n {{- tool_call.arguments | tojson }}\n {{- '}\\n</tool_call>' }}\n {%- endfor %}\n {{- '<|im_end|>\\n' }}\n {%- elif message.role == \"tool\" %}\n {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != \"tool\") %}\n {{- '<|im_start|>user' }}\n {%- endif %}\n {{- '\\n<tool_response>\\n' }}\n {{- message.content }}\n {{- '\\n</tool_response>' }}\n {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n {{- '<|im_end|>\\n' }}\n {%- endif %}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n",
199
- "clean_up_tokenization_spaces": false,
200
- "eos_token": "<|im_end|>",
201
- "errors": "replace",
202
- "extra_special_tokens": {},
203
- "model_max_length": 131072,
204
- "pad_token": "<|endoftext|>",
205
- "processor_class": "Qwen2_5_VLProcessor",
206
- "split_special_tokens": false,
207
- "tokenizer_class": "Qwen2Tokenizer",
208
- "unk_token": null
209
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
blobs/d17169ad645848ed1c32ef9d34529d80bbe052b3f5309d9f7973ed0fde30c09a DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:d17169ad645848ed1c32ef9d34529d80bbe052b3f5309d9f7973ed0fde30c09a
3
- size 4782749368
 
 
 
 
blobs/de3d7b2e27518f882a09cccf449e98ee4f5d72c3 DELETED
@@ -1,255 +0,0 @@
1
- {
2
- "_name_or_path": "Qwen/Qwen2.5-VL-72B-Instruct",
3
- "architectures": [
4
- "Qwen2_5_VLForConditionalGeneration"
5
- ],
6
- "attention_dropout": 0.0,
7
- "bos_token_id": 151643,
8
- "eos_token_id": 151645,
9
- "hidden_act": "silu",
10
- "hidden_size": 8192,
11
- "image_token_id": 151655,
12
- "initializer_range": 0.02,
13
- "intermediate_size": 29568,
14
- "max_position_embeddings": 128000,
15
- "max_window_layers": 80,
16
- "model_type": "qwen2_5_vl",
17
- "num_attention_heads": 64,
18
- "num_hidden_layers": 80,
19
- "num_key_value_heads": 8,
20
- "quantization_config": {
21
- "config_groups": {
22
- "group_0": {
23
- "input_activations": {
24
- "actorder": null,
25
- "block_structure": null,
26
- "dynamic": true,
27
- "group_size": null,
28
- "num_bits": 8,
29
- "observer": null,
30
- "observer_kwargs": {},
31
- "strategy": "token",
32
- "symmetric": true,
33
- "type": "float"
34
- },
35
- "output_activations": null,
36
- "targets": [
37
- "Linear"
38
- ],
39
- "weights": {
40
- "actorder": null,
41
- "block_structure": null,
42
- "dynamic": false,
43
- "group_size": null,
44
- "num_bits": 8,
45
- "observer": "minmax",
46
- "observer_kwargs": {},
47
- "strategy": "channel",
48
- "symmetric": true,
49
- "type": "float"
50
- }
51
- }
52
- },
53
- "format": "float-quantized",
54
- "global_compression_ratio": 1.323305644571225,
55
- "ignore": [
56
- "visual.blocks.0.attn.qkv",
57
- "visual.blocks.0.attn.proj",
58
- "visual.blocks.0.mlp.gate_proj",
59
- "visual.blocks.0.mlp.up_proj",
60
- "visual.blocks.0.mlp.down_proj",
61
- "visual.blocks.1.attn.qkv",
62
- "visual.blocks.1.attn.proj",
63
- "visual.blocks.1.mlp.gate_proj",
64
- "visual.blocks.1.mlp.up_proj",
65
- "visual.blocks.1.mlp.down_proj",
66
- "visual.blocks.2.attn.qkv",
67
- "visual.blocks.2.attn.proj",
68
- "visual.blocks.2.mlp.gate_proj",
69
- "visual.blocks.2.mlp.up_proj",
70
- "visual.blocks.2.mlp.down_proj",
71
- "visual.blocks.3.attn.qkv",
72
- "visual.blocks.3.attn.proj",
73
- "visual.blocks.3.mlp.gate_proj",
74
- "visual.blocks.3.mlp.up_proj",
75
- "visual.blocks.3.mlp.down_proj",
76
- "visual.blocks.4.attn.qkv",
77
- "visual.blocks.4.attn.proj",
78
- "visual.blocks.4.mlp.gate_proj",
79
- "visual.blocks.4.mlp.up_proj",
80
- "visual.blocks.4.mlp.down_proj",
81
- "visual.blocks.5.attn.qkv",
82
- "visual.blocks.5.attn.proj",
83
- "visual.blocks.5.mlp.gate_proj",
84
- "visual.blocks.5.mlp.up_proj",
85
- "visual.blocks.5.mlp.down_proj",
86
- "visual.blocks.6.attn.qkv",
87
- "visual.blocks.6.attn.proj",
88
- "visual.blocks.6.mlp.gate_proj",
89
- "visual.blocks.6.mlp.up_proj",
90
- "visual.blocks.6.mlp.down_proj",
91
- "visual.blocks.7.attn.qkv",
92
- "visual.blocks.7.attn.proj",
93
- "visual.blocks.7.mlp.gate_proj",
94
- "visual.blocks.7.mlp.up_proj",
95
- "visual.blocks.7.mlp.down_proj",
96
- "visual.blocks.8.attn.qkv",
97
- "visual.blocks.8.attn.proj",
98
- "visual.blocks.8.mlp.gate_proj",
99
- "visual.blocks.8.mlp.up_proj",
100
- "visual.blocks.8.mlp.down_proj",
101
- "visual.blocks.9.attn.qkv",
102
- "visual.blocks.9.attn.proj",
103
- "visual.blocks.9.mlp.gate_proj",
104
- "visual.blocks.9.mlp.up_proj",
105
- "visual.blocks.9.mlp.down_proj",
106
- "visual.blocks.10.attn.qkv",
107
- "visual.blocks.10.attn.proj",
108
- "visual.blocks.10.mlp.gate_proj",
109
- "visual.blocks.10.mlp.up_proj",
110
- "visual.blocks.10.mlp.down_proj",
111
- "visual.blocks.11.attn.qkv",
112
- "visual.blocks.11.attn.proj",
113
- "visual.blocks.11.mlp.gate_proj",
114
- "visual.blocks.11.mlp.up_proj",
115
- "visual.blocks.11.mlp.down_proj",
116
- "visual.blocks.12.attn.qkv",
117
- "visual.blocks.12.attn.proj",
118
- "visual.blocks.12.mlp.gate_proj",
119
- "visual.blocks.12.mlp.up_proj",
120
- "visual.blocks.12.mlp.down_proj",
121
- "visual.blocks.13.attn.qkv",
122
- "visual.blocks.13.attn.proj",
123
- "visual.blocks.13.mlp.gate_proj",
124
- "visual.blocks.13.mlp.up_proj",
125
- "visual.blocks.13.mlp.down_proj",
126
- "visual.blocks.14.attn.qkv",
127
- "visual.blocks.14.attn.proj",
128
- "visual.blocks.14.mlp.gate_proj",
129
- "visual.blocks.14.mlp.up_proj",
130
- "visual.blocks.14.mlp.down_proj",
131
- "visual.blocks.15.attn.qkv",
132
- "visual.blocks.15.attn.proj",
133
- "visual.blocks.15.mlp.gate_proj",
134
- "visual.blocks.15.mlp.up_proj",
135
- "visual.blocks.15.mlp.down_proj",
136
- "visual.blocks.16.attn.qkv",
137
- "visual.blocks.16.attn.proj",
138
- "visual.blocks.16.mlp.gate_proj",
139
- "visual.blocks.16.mlp.up_proj",
140
- "visual.blocks.16.mlp.down_proj",
141
- "visual.blocks.17.attn.qkv",
142
- "visual.blocks.17.attn.proj",
143
- "visual.blocks.17.mlp.gate_proj",
144
- "visual.blocks.17.mlp.up_proj",
145
- "visual.blocks.17.mlp.down_proj",
146
- "visual.blocks.18.attn.qkv",
147
- "visual.blocks.18.attn.proj",
148
- "visual.blocks.18.mlp.gate_proj",
149
- "visual.blocks.18.mlp.up_proj",
150
- "visual.blocks.18.mlp.down_proj",
151
- "visual.blocks.19.attn.qkv",
152
- "visual.blocks.19.attn.proj",
153
- "visual.blocks.19.mlp.gate_proj",
154
- "visual.blocks.19.mlp.up_proj",
155
- "visual.blocks.19.mlp.down_proj",
156
- "visual.blocks.20.attn.qkv",
157
- "visual.blocks.20.attn.proj",
158
- "visual.blocks.20.mlp.gate_proj",
159
- "visual.blocks.20.mlp.up_proj",
160
- "visual.blocks.20.mlp.down_proj",
161
- "visual.blocks.21.attn.qkv",
162
- "visual.blocks.21.attn.proj",
163
- "visual.blocks.21.mlp.gate_proj",
164
- "visual.blocks.21.mlp.up_proj",
165
- "visual.blocks.21.mlp.down_proj",
166
- "visual.blocks.22.attn.qkv",
167
- "visual.blocks.22.attn.proj",
168
- "visual.blocks.22.mlp.gate_proj",
169
- "visual.blocks.22.mlp.up_proj",
170
- "visual.blocks.22.mlp.down_proj",
171
- "visual.blocks.23.attn.qkv",
172
- "visual.blocks.23.attn.proj",
173
- "visual.blocks.23.mlp.gate_proj",
174
- "visual.blocks.23.mlp.up_proj",
175
- "visual.blocks.23.mlp.down_proj",
176
- "visual.blocks.24.attn.qkv",
177
- "visual.blocks.24.attn.proj",
178
- "visual.blocks.24.mlp.gate_proj",
179
- "visual.blocks.24.mlp.up_proj",
180
- "visual.blocks.24.mlp.down_proj",
181
- "visual.blocks.25.attn.qkv",
182
- "visual.blocks.25.attn.proj",
183
- "visual.blocks.25.mlp.gate_proj",
184
- "visual.blocks.25.mlp.up_proj",
185
- "visual.blocks.25.mlp.down_proj",
186
- "visual.blocks.26.attn.qkv",
187
- "visual.blocks.26.attn.proj",
188
- "visual.blocks.26.mlp.gate_proj",
189
- "visual.blocks.26.mlp.up_proj",
190
- "visual.blocks.26.mlp.down_proj",
191
- "visual.blocks.27.attn.qkv",
192
- "visual.blocks.27.attn.proj",
193
- "visual.blocks.27.mlp.gate_proj",
194
- "visual.blocks.27.mlp.up_proj",
195
- "visual.blocks.27.mlp.down_proj",
196
- "visual.blocks.28.attn.qkv",
197
- "visual.blocks.28.attn.proj",
198
- "visual.blocks.28.mlp.gate_proj",
199
- "visual.blocks.28.mlp.up_proj",
200
- "visual.blocks.28.mlp.down_proj",
201
- "visual.blocks.29.attn.qkv",
202
- "visual.blocks.29.attn.proj",
203
- "visual.blocks.29.mlp.gate_proj",
204
- "visual.blocks.29.mlp.up_proj",
205
- "visual.blocks.29.mlp.down_proj",
206
- "visual.blocks.30.attn.qkv",
207
- "visual.blocks.30.attn.proj",
208
- "visual.blocks.30.mlp.gate_proj",
209
- "visual.blocks.30.mlp.up_proj",
210
- "visual.blocks.30.mlp.down_proj",
211
- "visual.blocks.31.attn.qkv",
212
- "visual.blocks.31.attn.proj",
213
- "visual.blocks.31.mlp.gate_proj",
214
- "visual.blocks.31.mlp.up_proj",
215
- "visual.blocks.31.mlp.down_proj",
216
- "visual.merger.mlp.0",
217
- "visual.merger.mlp.2",
218
- "lm_head"
219
- ],
220
- "kv_cache_scheme": null,
221
- "quant_method": "compressed-tensors",
222
- "quantization_status": "compressed"
223
- },
224
- "rms_norm_eps": 1e-06,
225
- "rope_scaling": {
226
- "mrope_section": [
227
- 16,
228
- 24,
229
- 24
230
- ],
231
- "rope_type": "default",
232
- "type": "default"
233
- },
234
- "rope_theta": 1000000.0,
235
- "sliding_window": 32768,
236
- "tie_word_embeddings": false,
237
- "torch_dtype": "bfloat16",
238
- "transformers_version": "4.49.0.dev0",
239
- "use_cache": true,
240
- "use_sliding_window": false,
241
- "video_token_id": 151656,
242
- "vision_config": {
243
- "hidden_size": 1280,
244
- "in_chans": 3,
245
- "intermediate_size": 3456,
246
- "model_type": "qwen2_5_vl",
247
- "out_hidden_size": 8192,
248
- "spatial_patch_size": 14,
249
- "tokens_per_second": 2
250
- },
251
- "vision_end_token_id": 151653,
252
- "vision_start_token_id": 151652,
253
- "vision_token_id": 151654,
254
- "vocab_size": 152064
255
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
blobs/ee397dd8b35e26aea29863b307013b1bc2071c8c DELETED
@@ -1,13 +0,0 @@
1
- {
2
- "bos_token_id": 151643,
3
- "do_sample": true,
4
- "eos_token_id": [
5
- 151645,
6
- 151643
7
- ],
8
- "pad_token_id": 151643,
9
- "repetition_penalty": 1.05,
10
- "top_k": 1,
11
- "top_p": 0.001,
12
- "transformers_version": "4.49.0.dev0"
13
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
blobs/f3611e6a6ce31c332c9fcbaa2d5975ad981a6aa05470b66df74dab8811f7cb0a DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:f3611e6a6ce31c332c9fcbaa2d5975ad981a6aa05470b66df74dab8811f7cb0a
3
- size 4096149840
 
 
 
 
chat_template.json DELETED
@@ -1,3 +0,0 @@
1
- {
2
- "chat_template": "{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n{% endif %}<|im_start|>{{ message['role'] }}\n{% if message['content'] is string %}{{ message['content'] }}<|im_end|>\n{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}"
3
- }
 
 
 
 
config.json DELETED
@@ -1,255 +0,0 @@
1
- {
2
- "_name_or_path": "Qwen/Qwen2.5-VL-72B-Instruct",
3
- "architectures": [
4
- "Qwen2_5_VLForConditionalGeneration"
5
- ],
6
- "attention_dropout": 0.0,
7
- "bos_token_id": 151643,
8
- "eos_token_id": 151645,
9
- "hidden_act": "silu",
10
- "hidden_size": 8192,
11
- "image_token_id": 151655,
12
- "initializer_range": 0.02,
13
- "intermediate_size": 29568,
14
- "max_position_embeddings": 128000,
15
- "max_window_layers": 80,
16
- "model_type": "qwen2_5_vl",
17
- "num_attention_heads": 64,
18
- "num_hidden_layers": 80,
19
- "num_key_value_heads": 8,
20
- "quantization_config": {
21
- "config_groups": {
22
- "group_0": {
23
- "input_activations": {
24
- "actorder": null,
25
- "block_structure": null,
26
- "dynamic": true,
27
- "group_size": null,
28
- "num_bits": 8,
29
- "observer": null,
30
- "observer_kwargs": {},
31
- "strategy": "token",
32
- "symmetric": true,
33
- "type": "float"
34
- },
35
- "output_activations": null,
36
- "targets": [
37
- "Linear"
38
- ],
39
- "weights": {
40
- "actorder": null,
41
- "block_structure": null,
42
- "dynamic": false,
43
- "group_size": null,
44
- "num_bits": 8,
45
- "observer": "minmax",
46
- "observer_kwargs": {},
47
- "strategy": "channel",
48
- "symmetric": true,
49
- "type": "float"
50
- }
51
- }
52
- },
53
- "format": "float-quantized",
54
- "global_compression_ratio": 1.323305644571225,
55
- "ignore": [
56
- "visual.blocks.0.attn.qkv",
57
- "visual.blocks.0.attn.proj",
58
- "visual.blocks.0.mlp.gate_proj",
59
- "visual.blocks.0.mlp.up_proj",
60
- "visual.blocks.0.mlp.down_proj",
61
- "visual.blocks.1.attn.qkv",
62
- "visual.blocks.1.attn.proj",
63
- "visual.blocks.1.mlp.gate_proj",
64
- "visual.blocks.1.mlp.up_proj",
65
- "visual.blocks.1.mlp.down_proj",
66
- "visual.blocks.2.attn.qkv",
67
- "visual.blocks.2.attn.proj",
68
- "visual.blocks.2.mlp.gate_proj",
69
- "visual.blocks.2.mlp.up_proj",
70
- "visual.blocks.2.mlp.down_proj",
71
- "visual.blocks.3.attn.qkv",
72
- "visual.blocks.3.attn.proj",
73
- "visual.blocks.3.mlp.gate_proj",
74
- "visual.blocks.3.mlp.up_proj",
75
- "visual.blocks.3.mlp.down_proj",
76
- "visual.blocks.4.attn.qkv",
77
- "visual.blocks.4.attn.proj",
78
- "visual.blocks.4.mlp.gate_proj",
79
- "visual.blocks.4.mlp.up_proj",
80
- "visual.blocks.4.mlp.down_proj",
81
- "visual.blocks.5.attn.qkv",
82
- "visual.blocks.5.attn.proj",
83
- "visual.blocks.5.mlp.gate_proj",
84
- "visual.blocks.5.mlp.up_proj",
85
- "visual.blocks.5.mlp.down_proj",
86
- "visual.blocks.6.attn.qkv",
87
- "visual.blocks.6.attn.proj",
88
- "visual.blocks.6.mlp.gate_proj",
89
- "visual.blocks.6.mlp.up_proj",
90
- "visual.blocks.6.mlp.down_proj",
91
- "visual.blocks.7.attn.qkv",
92
- "visual.blocks.7.attn.proj",
93
- "visual.blocks.7.mlp.gate_proj",
94
- "visual.blocks.7.mlp.up_proj",
95
- "visual.blocks.7.mlp.down_proj",
96
- "visual.blocks.8.attn.qkv",
97
- "visual.blocks.8.attn.proj",
98
- "visual.blocks.8.mlp.gate_proj",
99
- "visual.blocks.8.mlp.up_proj",
100
- "visual.blocks.8.mlp.down_proj",
101
- "visual.blocks.9.attn.qkv",
102
- "visual.blocks.9.attn.proj",
103
- "visual.blocks.9.mlp.gate_proj",
104
- "visual.blocks.9.mlp.up_proj",
105
- "visual.blocks.9.mlp.down_proj",
106
- "visual.blocks.10.attn.qkv",
107
- "visual.blocks.10.attn.proj",
108
- "visual.blocks.10.mlp.gate_proj",
109
- "visual.blocks.10.mlp.up_proj",
110
- "visual.blocks.10.mlp.down_proj",
111
- "visual.blocks.11.attn.qkv",
112
- "visual.blocks.11.attn.proj",
113
- "visual.blocks.11.mlp.gate_proj",
114
- "visual.blocks.11.mlp.up_proj",
115
- "visual.blocks.11.mlp.down_proj",
116
- "visual.blocks.12.attn.qkv",
117
- "visual.blocks.12.attn.proj",
118
- "visual.blocks.12.mlp.gate_proj",
119
- "visual.blocks.12.mlp.up_proj",
120
- "visual.blocks.12.mlp.down_proj",
121
- "visual.blocks.13.attn.qkv",
122
- "visual.blocks.13.attn.proj",
123
- "visual.blocks.13.mlp.gate_proj",
124
- "visual.blocks.13.mlp.up_proj",
125
- "visual.blocks.13.mlp.down_proj",
126
- "visual.blocks.14.attn.qkv",
127
- "visual.blocks.14.attn.proj",
128
- "visual.blocks.14.mlp.gate_proj",
129
- "visual.blocks.14.mlp.up_proj",
130
- "visual.blocks.14.mlp.down_proj",
131
- "visual.blocks.15.attn.qkv",
132
- "visual.blocks.15.attn.proj",
133
- "visual.blocks.15.mlp.gate_proj",
134
- "visual.blocks.15.mlp.up_proj",
135
- "visual.blocks.15.mlp.down_proj",
136
- "visual.blocks.16.attn.qkv",
137
- "visual.blocks.16.attn.proj",
138
- "visual.blocks.16.mlp.gate_proj",
139
- "visual.blocks.16.mlp.up_proj",
140
- "visual.blocks.16.mlp.down_proj",
141
- "visual.blocks.17.attn.qkv",
142
- "visual.blocks.17.attn.proj",
143
- "visual.blocks.17.mlp.gate_proj",
144
- "visual.blocks.17.mlp.up_proj",
145
- "visual.blocks.17.mlp.down_proj",
146
- "visual.blocks.18.attn.qkv",
147
- "visual.blocks.18.attn.proj",
148
- "visual.blocks.18.mlp.gate_proj",
149
- "visual.blocks.18.mlp.up_proj",
150
- "visual.blocks.18.mlp.down_proj",
151
- "visual.blocks.19.attn.qkv",
152
- "visual.blocks.19.attn.proj",
153
- "visual.blocks.19.mlp.gate_proj",
154
- "visual.blocks.19.mlp.up_proj",
155
- "visual.blocks.19.mlp.down_proj",
156
- "visual.blocks.20.attn.qkv",
157
- "visual.blocks.20.attn.proj",
158
- "visual.blocks.20.mlp.gate_proj",
159
- "visual.blocks.20.mlp.up_proj",
160
- "visual.blocks.20.mlp.down_proj",
161
- "visual.blocks.21.attn.qkv",
162
- "visual.blocks.21.attn.proj",
163
- "visual.blocks.21.mlp.gate_proj",
164
- "visual.blocks.21.mlp.up_proj",
165
- "visual.blocks.21.mlp.down_proj",
166
- "visual.blocks.22.attn.qkv",
167
- "visual.blocks.22.attn.proj",
168
- "visual.blocks.22.mlp.gate_proj",
169
- "visual.blocks.22.mlp.up_proj",
170
- "visual.blocks.22.mlp.down_proj",
171
- "visual.blocks.23.attn.qkv",
172
- "visual.blocks.23.attn.proj",
173
- "visual.blocks.23.mlp.gate_proj",
174
- "visual.blocks.23.mlp.up_proj",
175
- "visual.blocks.23.mlp.down_proj",
176
- "visual.blocks.24.attn.qkv",
177
- "visual.blocks.24.attn.proj",
178
- "visual.blocks.24.mlp.gate_proj",
179
- "visual.blocks.24.mlp.up_proj",
180
- "visual.blocks.24.mlp.down_proj",
181
- "visual.blocks.25.attn.qkv",
182
- "visual.blocks.25.attn.proj",
183
- "visual.blocks.25.mlp.gate_proj",
184
- "visual.blocks.25.mlp.up_proj",
185
- "visual.blocks.25.mlp.down_proj",
186
- "visual.blocks.26.attn.qkv",
187
- "visual.blocks.26.attn.proj",
188
- "visual.blocks.26.mlp.gate_proj",
189
- "visual.blocks.26.mlp.up_proj",
190
- "visual.blocks.26.mlp.down_proj",
191
- "visual.blocks.27.attn.qkv",
192
- "visual.blocks.27.attn.proj",
193
- "visual.blocks.27.mlp.gate_proj",
194
- "visual.blocks.27.mlp.up_proj",
195
- "visual.blocks.27.mlp.down_proj",
196
- "visual.blocks.28.attn.qkv",
197
- "visual.blocks.28.attn.proj",
198
- "visual.blocks.28.mlp.gate_proj",
199
- "visual.blocks.28.mlp.up_proj",
200
- "visual.blocks.28.mlp.down_proj",
201
- "visual.blocks.29.attn.qkv",
202
- "visual.blocks.29.attn.proj",
203
- "visual.blocks.29.mlp.gate_proj",
204
- "visual.blocks.29.mlp.up_proj",
205
- "visual.blocks.29.mlp.down_proj",
206
- "visual.blocks.30.attn.qkv",
207
- "visual.blocks.30.attn.proj",
208
- "visual.blocks.30.mlp.gate_proj",
209
- "visual.blocks.30.mlp.up_proj",
210
- "visual.blocks.30.mlp.down_proj",
211
- "visual.blocks.31.attn.qkv",
212
- "visual.blocks.31.attn.proj",
213
- "visual.blocks.31.mlp.gate_proj",
214
- "visual.blocks.31.mlp.up_proj",
215
- "visual.blocks.31.mlp.down_proj",
216
- "visual.merger.mlp.0",
217
- "visual.merger.mlp.2",
218
- "lm_head"
219
- ],
220
- "kv_cache_scheme": null,
221
- "quant_method": "compressed-tensors",
222
- "quantization_status": "compressed"
223
- },
224
- "rms_norm_eps": 1e-06,
225
- "rope_scaling": {
226
- "mrope_section": [
227
- 16,
228
- 24,
229
- 24
230
- ],
231
- "rope_type": "default",
232
- "type": "default"
233
- },
234
- "rope_theta": 1000000.0,
235
- "sliding_window": 32768,
236
- "tie_word_embeddings": false,
237
- "torch_dtype": "bfloat16",
238
- "transformers_version": "4.49.0.dev0",
239
- "use_cache": true,
240
- "use_sliding_window": false,
241
- "video_token_id": 151656,
242
- "vision_config": {
243
- "hidden_size": 1280,
244
- "in_chans": 3,
245
- "intermediate_size": 3456,
246
- "model_type": "qwen2_5_vl",
247
- "out_hidden_size": 8192,
248
- "spatial_patch_size": 14,
249
- "tokens_per_second": 2
250
- },
251
- "vision_end_token_id": 151653,
252
- "vision_start_token_id": 151652,
253
- "vision_token_id": 151654,
254
- "vocab_size": 152064
255
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
generation_config.json DELETED
@@ -1,13 +0,0 @@
1
- {
2
- "bos_token_id": 151643,
3
- "do_sample": true,
4
- "eos_token_id": [
5
- 151645,
6
- 151643
7
- ],
8
- "pad_token_id": 151643,
9
- "repetition_penalty": 1.05,
10
- "top_k": 1,
11
- "top_p": 0.001,
12
- "transformers_version": "4.49.0.dev0"
13
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
merges.txt DELETED
The diff for this file is too large to render. See raw diff
 
model-00001-of-00016.safetensors DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:c4fdcf55d00ffa11e55fcc79ab5cca51d50a1d267219820ac66c36be0177fb23
3
- size 4929545176
 
 
 
 
model-00002-of-00016.safetensors DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:66dcd62e9496e9b8c456f6c95df848034fe648f4414ff1088a35f7a6efa44156
3
- size 4873985928
 
 
 
 
model-00003-of-00016.safetensors DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:4f76e418f06f6b9b7f8a524f6405b1baee70ca7418a9a660f7537c490263ae35
3
- size 4782749304
 
 
 
 
model-00004-of-00016.safetensors DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:09782507221803a2287fcacb4ef166317fee2e62839ad43caa4c2f9ec461f77c
3
- size 4873986032
 
 
 
 
model-00005-of-00016.safetensors DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:ab1e3dfb60f12733be7aae9274996f14814cc4680a707e1a201a0b627b4373ea
3
- size 4782749368
 
 
 
 
model-00006-of-00016.safetensors DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:39f8a745a63bdd2552d0aaa25728108ccd66718611bb8f1b1d1998a86e5054a1
3
- size 4873986032
 
 
 
 
model-00007-of-00016.safetensors DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:2f452321174378d32090e7c94cffa3b43a4539e2f407f936b12a35639122005a
3
- size 4782749368
 
 
 
 
model-00008-of-00016.safetensors DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:87afe66c34acb2c5a7b118db74939afe8f47613f442eca4ed0f38a02e4e93fed
3
- size 4873986032
 
 
 
 
model-00009-of-00016.safetensors DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:4a79addd10e3e66dcc4b53822ed86a7e31197ca3a54eaacc2a8140921d158f47
3
- size 4782749368
 
 
 
 
model-00010-of-00016.safetensors DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:83154d2dd5e7064947716c0930013e17943c378f80343c5f6067e8afdc291c1f
3
- size 4873986032
 
 
 
 
model-00011-of-00016.safetensors DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:48dd3dd98f76920c997a634e4f390f4ab11e629c7d76d752ef94844477bea183
3
- size 4782749368
 
 
 
 
model-00012-of-00016.safetensors DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:3168a6cbc214a1e83a04235e676fad12952ef36010442e02e007701ea23037e4
3
- size 4873986032
 
 
 
 
model-00013-of-00016.safetensors DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:98d1b9f92f0154fc8f7cbfe0c6d7d3955b088cf2c520e1a41c2f157c07c5b6bf
3
- size 4782749368
 
 
 
 
model-00014-of-00016.safetensors DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:46d25976bd3a6cb3055a1376a5a3ffec0f350d6a34a9749cbdd0495409e5e744
3
- size 4873986032