reece124 commited on
Commit
09b2bf2
·
verified ·
1 Parent(s): 48153fe

Upload folder using huggingface_hub

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. README.md +411 -0
  2. config.bak.json +69 -0
  3. config.json +51 -0
  4. configuration_opencua.py +37 -0
  5. generation_config.json +4 -0
  6. model-1-of-64.safetensors +3 -0
  7. model-10-of-64.safetensors +3 -0
  8. model-11-of-64.safetensors +3 -0
  9. model-12-of-64.safetensors +3 -0
  10. model-13-of-64.safetensors +3 -0
  11. model-14-of-64.safetensors +3 -0
  12. model-15-of-64.safetensors +3 -0
  13. model-16-of-64.safetensors +3 -0
  14. model-17-of-64.safetensors +3 -0
  15. model-18-of-64.safetensors +3 -0
  16. model-19-of-64.safetensors +3 -0
  17. model-2-of-64.safetensors +3 -0
  18. model-20-of-64.safetensors +3 -0
  19. model-21-of-64.safetensors +3 -0
  20. model-22-of-64.safetensors +3 -0
  21. model-23-of-64.safetensors +3 -0
  22. model-24-of-64.safetensors +3 -0
  23. model-25-of-64.safetensors +3 -0
  24. model-26-of-64.safetensors +3 -0
  25. model-27-of-64.safetensors +3 -0
  26. model-28-of-64.safetensors +3 -0
  27. model-29-of-64.safetensors +3 -0
  28. model-3-of-64.safetensors +3 -0
  29. model-30-of-64.safetensors +3 -0
  30. model-31-of-64.safetensors +3 -0
  31. model-32-of-64.safetensors +3 -0
  32. model-33-of-64.safetensors +3 -0
  33. model-34-of-64.safetensors +3 -0
  34. model-35-of-64.safetensors +3 -0
  35. model-36-of-64.safetensors +3 -0
  36. model-37-of-64.safetensors +3 -0
  37. model-38-of-64.safetensors +3 -0
  38. model-39-of-64.safetensors +3 -0
  39. model-4-of-64.safetensors +3 -0
  40. model-40-of-64.safetensors +3 -0
  41. model-41-of-64.safetensors +3 -0
  42. model-42-of-64.safetensors +3 -0
  43. model-43-of-64.safetensors +3 -0
  44. model-44-of-64.safetensors +3 -0
  45. model-45-of-64.safetensors +3 -0
  46. model-46-of-64.safetensors +3 -0
  47. model-47-of-64.safetensors +3 -0
  48. model-48-of-64.safetensors +3 -0
  49. model-49-of-64.safetensors +3 -0
  50. model-5-of-64.safetensors +3 -0
README.md ADDED
@@ -0,0 +1,411 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model:
3
+ - Qwen/Qwen2.5-VL-32B-Instruct
4
+ datasets:
5
+ - xlangai/AgentNet
6
+ - xlangai/aguvis-stage1
7
+ - xlangai/aguvis-stage2
8
+ - osunlp/UGround-V1-Data
9
+ language:
10
+ - en
11
+ license: mit
12
+ metrics:
13
+ - code_eval
14
+ - accuracy
15
+ pipeline_tag: image-text-to-text
16
+ tags:
17
+ - VLM
18
+ - Computer-Use-Agent
19
+ - OS-Agent
20
+ - GUI
21
+ - Grounding
22
+ library_name: transformers
23
+ ---
24
+
25
+ <h1 style="
26
+ font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Helvetica,Arial,sans-serif;
27
+ font-size:48px;
28
+ font-weight:700;
29
+ line-height:1.25;
30
+ text-align:center;
31
+ margin:0 0 24px;">
32
+ OpenCUA: Open Foundations for Computer-Use Agents
33
+ </h1>
34
+
35
+ <div style="
36
+ display:flex;
37
+ justify-content:center;
38
+ gap:12px;
39
+ flex-wrap:wrap;
40
+ margin-bottom:28px;">
41
+
42
+ <a href="https://opencua.xlang.ai/" style="
43
+ display:inline-block;
44
+ padding:8px 24px;
45
+ background:#2b2b2b;
46
+ color:#ffffff;
47
+ border-radius:36px;
48
+ text-decoration:none;
49
+ font-weight:600;
50
+ font-size:16px;">
51
+ 🌐 Website
52
+ </a>
53
+
54
+ <a href="https://arxiv.org/abs/2508.09123" style="
55
+ display:inline-block;
56
+ padding:8px 24px;
57
+ background:#2b2b2b;
58
+ color:#ffffff;
59
+ border-radius:36px;
60
+ text-decoration:none;
61
+ font-weight:600;
62
+ font-size:16px;">
63
+ 📝 Paper
64
+ </a>
65
+
66
+ <a href="https://github.com/xlang-ai/OpenCUA" style="
67
+ display:inline-block;
68
+ padding:8px 24px;
69
+ background:#2b2b2b;
70
+ color:#ffffff;
71
+ border-radius:36px;
72
+ text-decoration:none;
73
+ font-weight:600;
74
+ font-size:16px;">
75
+ 💻 Code
76
+ </a>
77
+ </div>
78
+
79
+ <div style="max-width:900px;margin:0 auto;">
80
+
81
+ # Introduction
82
+ <div style="
83
+ max-width: 880px; /* 可按需调节整体宽度 */
84
+ margin: 0 auto; /* 居中容器 */
85
+ text-align: justify; /* 关键:两端对齐 */
86
+ text-justify: inter-word; /* 优化英文对齐效果 */
87
+ line-height: 1.6;">
88
+
89
+ OpenCUA models (OpenCUA-7B and OpenCUA-32B) are end-to-end computer-use foundation models than can produce executable actions in the computer environments. They are based on the weights of Qwen2.5-VL-7B-Instruction and Qwen2.5-VL-32B-Instruction.
90
+ They demonstrate superior performance across CUA benchmarks. In particular, <b>OpenCUA-32B</b> achieves an average success rate of **34.8%** on [OSWorld-Verified](https://os-world.github.io/),
91
+ establishing a new state-of-the-art (SOTA) among open-source models and surpassing OpenAI CUA (GPT-4o). Both models also have strong grounding performance, OpenCUA-32B achieves 59.6% on [OSWorld-G](https://osworld-grounding.github.io/) and 55.3% on [Screenspot-Pro](https://arxiv.org/abs/2504.07981).
92
+ </div>
93
+
94
+ ### Key Features
95
+
96
+ - **Superior Computer-Use Capablity**: Able to execute multi-step computer-use actions with effective planning and reasoning
97
+ - **Multi-OS Support**: Trained on demonstrations across Ubuntu, Windows, and macOS
98
+ - **Visual Grounding**: Strong GUI element recognition and spatial reasoning capabilities
99
+ - **Multi-Image Context**: Processes up to 3 screenshot history for better context understanding
100
+ - **Reflective Reasoning**: Enhanced with reflective long Chain-of-Thought that identifies errors and provides corrective reasoning
101
+
102
+
103
+ # Performance
104
+
105
+ ### Online Agent Evaluation
106
+ OpenCUA models achieves strong performance on **[OSWorld-Verified](https://os-world.github.io/)**.
107
+ OPENCUA-32B achieves the best performance among all open-source models with an average success rate of 34.8%, outperforming prior baselines by large margins.
108
+ It also closes the gap to proprietary Claude models.
109
+ <div align="center">
110
+
111
+ | **Model** | **15 Steps** | **50 Steps** | **100 Steps** |
112
+ |-------------------------------|:--------:|:--------:|:---------:|
113
+ | **Proprietary** | | | |
114
+ | OpenAI CUA | 26.0 | 31.3 | 31.4 |
115
+ | Seed 1.5-VL | 27.9 | — | 34.1 |
116
+ | Claude 3.7 Sonnet | 27.1 | 35.8 | 35.9 |
117
+ | Claude 4 Sonnet | 31.2 | 43.9 | 41.5 |
118
+ | **Open-Source** | | | |
119
+ | Qwen 2.5-VL-32B-Instruct | 3.0 | — | 3.9 |
120
+ | Qwen 2.5-VL-72B-Instruct | 4.4 | — | 5.0 |
121
+ | Kimi-VL-A3B | 9.7 | — | 10.3 |
122
+ | UI-TARS-72B-DPO | 24.0 | 25.8 | 27.1 |
123
+ | UI-TARS-1.5-7B | 24.5 | 27.3 | 27.4 |
124
+ | OpenCUA-7B *(Ours)* | 24.3 | 27.9 | 26.6 |
125
+ | **OpenCUA-32B *(Ours)*** | **29.7** | **34.1** | **34.8** |
126
+ </div>
127
+
128
+ *OpenCUA scores are the mean of 3 independent runs.*
129
+
130
+ ### GUI Grounding Performance
131
+ <div align="center">
132
+
133
+ | **Model** | **OSWorld-G** | **ScreenSpot-V2** | **ScreenSpot-Pro** |
134
+ |-------|-----------|---------------|----------------|
135
+ | Qwen2.5-VL-7B | 31.4 | 88.8 | 27.6 |
136
+ | Qwen2.5-VL-32B | 46.5 | 87.0 | 39.4 |
137
+ | UI-TARS-72B | 57.1 | 90.3 | 38.1 |
138
+ | **OpenCUA-A3B** | 48.6 | 91.4 | 28.5 |
139
+ | **OpenCUA-Qwen2-7B** | 45.7 | 88.5 | 23.7 |
140
+ | **OpenCUA-7B** | 55.3 | 92.3 | 50.0 |
141
+ | **OpenCUA-32B** | **59.6** | **93.4** | **55.3** |
142
+ </div>
143
+
144
+
145
+ ### AgentNetBench (Offline Evaluation)
146
+ <div align="center">
147
+
148
+ | **Model** | **Coordinate Actions** | **Content Actions** | **Function Actions** | **Average** |
149
+ |-------|-------------------|-----------------|------------------|---------|
150
+ | Qwen2.5-VL-7B | 50.7 | 40.8 | 3.1 | 48.0 |
151
+ | Qwen2.5-VL-32B | 66.6 | 47.2 | 41.5 | 64.8 |
152
+ | Qwen2.5-VL-72B | 67.2 | 52.6 | 50.5 | 67.0 |
153
+ | OpenAI CUA | 71.7 | 57.3 | **80.0** | 73.1 |
154
+ | **OpenCUA-7B** | 79.0 | 62.0 | 44.3 | 75.2 |
155
+ | **OpenCUA-32B** | **81.9** | 66.1 | 55.7 | **79.1** |
156
+ </div>
157
+
158
+ # 🚀 Quick Start
159
+ <div style="border-left: 6px solid #f28c28; background: #fff8e6; padding: 12px 16px; margin: 16px 0;">
160
+ <strong>⚠️ Important for Qwen-based Models (OpenCUA-7B, OpenCUA-32B):</strong>
161
+
162
+ To align with our training infrastructure, we have modified the model in two places:
163
+ <ul style="margin-top: 8px;">
164
+ <li>1. Multimodal Rotary Position Embedding (M-RoPE) has been replaced with 1D RoPE</strong>.</li>
165
+ <li>2. Using the same Tokenizer and ChatTemplate as Kimi-VL.</li>
166
+ <li>Do not use the default transformers and vllm classes to load the model. Tokenizer and Chat Template should be aligned if training the models.</li>
167
+ </ul>
168
+ </div>
169
+
170
+
171
+ ## Installation & Download
172
+
173
+ First, install the required transformers dependencies:
174
+
175
+ ```bash
176
+ conda create -n opencua python=3.10
177
+ conda activate opencua
178
+ pip install -r requirement.txt
179
+ ```
180
+
181
+ Download the model weight from huggingface:
182
+ ```bash
183
+ from huggingface_hub import snapshot_download
184
+ snapshot_download(
185
+ repo_id="xlangai/OpenCUA-32B",
186
+ local_dir="OpenCUA-32B",
187
+ local_dir_use_symlinks=False
188
+ )
189
+ ```
190
+
191
+ ## 🎯 GUI Grounding
192
+
193
+ The following code demonstrates how to use OpenCUA models for GUI grounding tasks:
194
+
195
+ ```python
196
+ import base64
197
+ import torch
198
+ from transformers import AutoTokenizer, AutoModel, AutoImageProcessor
199
+ from PIL import Image
200
+ import json
201
+
202
+ def encode_image(image_path: str) -> str:
203
+ """Encode image to base64 string for model input."""
204
+ with open(image_path, "rb") as f:
205
+ return base64.b64encode(f.read()).decode()
206
+
207
+ def load_opencua_model(model_path: str):
208
+ """Load OpenCUA model, tokenizer, and image processor."""
209
+ tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
210
+ model = AutoModel.from_pretrained(
211
+ model_path,
212
+ torch_dtype="auto",
213
+ device_map="auto",
214
+ trust_remote_code=True
215
+ )
216
+ image_processor = AutoImageProcessor.from_pretrained(model_path, trust_remote_code=True)
217
+
218
+ return model, tokenizer, image_processor
219
+
220
+ def create_grounding_messages(image_path: str, instruction: str):
221
+ """Create chat messages for GUI grounding task."""
222
+ system_prompt = (
223
+ "You are a GUI agent. You are given a task and a screenshot of the screen. "
224
+ "You need to perform a series of pyautogui actions to complete the task."
225
+ )
226
+
227
+ messages = [
228
+ {"role": "system", "content": system_prompt},
229
+ {
230
+ "role": "user",
231
+ "content": [
232
+ {"type": "image", "image": f"data:image/png;base64,{encode_image(image_path)}"},
233
+ {"type": "text", "text": instruction},
234
+ ],
235
+ },
236
+ ]
237
+ return messages
238
+
239
+ def run_inference(model, tokenizer, image_processor, messages, image_path):
240
+ """Run inference on the model."""
241
+ # Prepare text input
242
+ input_ids = tokenizer.apply_chat_template(
243
+ messages, tokenize=True, add_generation_prompt=True
244
+ )
245
+ input_ids = torch.tensor([input_ids]).to(model.device)
246
+
247
+ # Prepare image input
248
+ image = Image.open(image_path).convert('RGB')
249
+ image_info = image_processor.preprocess(images=[image])
250
+ pixel_values = torch.tensor(image_info['pixel_values']).to(
251
+ dtype=torch.bfloat16, device=model.device
252
+ )
253
+ grid_thws = torch.tensor(image_info['image_grid_thw'])
254
+
255
+ # Generate response
256
+ with torch.no_grad():
257
+ generated_ids = model.generate(
258
+ input_ids,
259
+ pixel_values=pixel_values,
260
+ grid_thws=grid_thws,
261
+ max_new_tokens=512,
262
+ temperature=0
263
+ )
264
+
265
+ # Decode output
266
+ prompt_len = input_ids.shape[1]
267
+ generated_ids = generated_ids[:, prompt_len:]
268
+ output_text = tokenizer.batch_decode(
269
+ generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
270
+ )[0]
271
+
272
+ return output_text
273
+
274
+ # Example usage
275
+ model_path = "xlangai/OpenCUA-32B" # or other model variants
276
+ image_path = "screenshot.png"
277
+ instruction = "Click on the submit button"
278
+
279
+ # Load model
280
+ model, tokenizer, image_processor = load_opencua_model(model_path)
281
+
282
+ # Create messages and run inference
283
+ messages = create_grounding_messages(image_path, instruction)
284
+ result = run_inference(model, tokenizer, image_processor, messages, image_path)
285
+
286
+ print("Model output:", result)
287
+ ```
288
+
289
+ <div style="border-left: 6px solid #9ca3af; background: #f5f5f5; padding: 12px 16px; margin: 16px 0;">
290
+ <em>Expected result: ```python
291
+ pyautogui.click(x=1432, y=344)
292
+ ```</em>
293
+ </div>
294
+
295
+ ## 🖥️ Computer Use Agent
296
+ **[OpenCUAAgent](https://github.com/xlang-ai/OSWorld/blob/main/mm_agents/opencua_agent.py)** is developed in the [OSWorld](https://github.com/xlang-ai/OSWorld) environment based on OpenCUA models. It iteratively perceives the environment via screenshots, produces reflective long CoT as inner monologue, and predicts the next action to be executed. OpenCUAAgent uses 3 images in total and L2 CoT format in default.
297
+
298
+ Command for running OpenCUA-7B and OpenCUA-32B in OSWorld:
299
+ ```
300
+ python run_multienv_opencua.py \
301
+ --headless \
302
+ --observation_type screenshot \
303
+ --model OpenCUA-32B \
304
+ --result_dir ./results --test_all_meta_path evaluation_examples/test_all_no_gdrive.json \
305
+ --max_steps 100 \
306
+ --num_envs 30 \
307
+ --coordinate_type qwen25
308
+ ```
309
+ <div style="border-left: 6px solid #9ca3af; background: #f5f5f5; padding: 12px 16px; margin: 16px 0;">
310
+ <em>Currently we only supports huggingface inference. We are implementing the vLLM supports of OpenCUA models. Please stay tuned.</em>
311
+ </div>
312
+
313
+ ## Important Notes on Coordinate Systems
314
+ <div style="border-left: 6px solid #9ca3af; background: #f5f5f5; padding: 12px 16px; margin: 16px 0;">
315
+ <ul style="margin: 0;">
316
+ <li><strong><code>xlangai/OpenCUA-A3B</code></strong> – Relative coordinates <em>(not supported in this code)</em></li>
317
+ <li><strong><code>xlangai/OpenCUA-Qwen2-7B</code></strong> – Relative coordinates</li>
318
+ <li><strong><code>xlangai/OpenCUA-7B</code></strong> – Absolute coordinates</li>
319
+ <li><strong><code>xlangai/OpenCUA-32B</code></strong> – Absolute coordinates</li>
320
+ </ul>
321
+ </div>
322
+
323
+ **OpenCUA models use different coordinate systems depending on the base model:**
324
+
325
+ - **OpenCUA-Qwen2-7B**: Outputs **relative coordinates** (0.0 to 1.0 range)
326
+ ```python
327
+ # Example output: pyautogui.click(x=0.5, y=0.3)
328
+ # x=0.5 means 50% from left edge, y=0.3 means 30% from top edge
329
+
330
+ # Convert to absolute coordinates:
331
+ def qwen2_relative_to_absolute(rel_x, rel_y, original_width, original_height):
332
+ abs_x = int(rel_x * original_width)
333
+ abs_y = int(rel_y * original_height)
334
+ return abs_x, abs_y
335
+ ```
336
+
337
+ - **OpenCUA-7B and OpenCUA-32B** (Qwen2.5-based): Output **absolute coordinates** after smart resize
338
+ ```python
339
+ # Example output: pyautogui.click(x=960, y=324)
340
+ # These are coordinates on the smart-resized image, not the original image
341
+
342
+ # Convert to original image coordinates:
343
+ # Please refer to the smart_resize function in: https://github.com/huggingface/transformers/blob/67ddc82fbc7e52c6f42a395b4a6d278c55b77a39/src/transformers/models/qwen2_vl/image_processing_qwen2_vl.py#L55
344
+ def qwen25_smart_resize_to_absolute(model_x, model_y, original_width, original_height):
345
+ # First, calculate the smart-resized dimensions
346
+ resized_height, resized_width = smart_resize(original_height, original_width, factor = 28, min_pixels = 3136, max_pixels = 12845056)
347
+
348
+ # Convert model output to relative coordinates on original image
349
+ rel_x = model_x / resized_width
350
+ rel_y = model_y / resized_height
351
+
352
+ # Then convert to absolute coordinates on original image
353
+ abs_x = int(rel_x * original_width)
354
+ abs_y = int(rel_y * original_height)
355
+ return abs_x, abs_y
356
+ ```
357
+
358
+ <div style="border-left: 6px solid #9ca3af; background: #f5f5f5; padding: 12px 16px; margin: 16px 0;">
359
+ <strong>Understanding Smart Resize for Qwen2.5-based Models:</strong>
360
+ <p style="margin: 8px 0 0;">
361
+ The Qwen2.5-VL models use a “smart resize” preprocessing that maintains aspect ratio while fitting within pixel constraints.
362
+ For coordinate conversion, you need the smart resize function from the
363
+ <a href="https://github.com/QwenLM/Qwen2.5-VL/blob/d2240f11656bfe404b9ba56db4e51cd09f522ff1/qwen-vl-utils/src/qwen_vl_utils/vision_process.py#L60">
364
+ official Qwen2.5-VL implementation</a>.
365
+ </p>
366
+ </div>
367
+
368
+
369
+ # TODO
370
+ ## vLLM Support
371
+ We are actively working with the vLLM team to add support for OpenCUA models.
372
+
373
+ **Workaround:** For now, please use the standard transformers library as shown in the examples above. We will update this section once vLLM support becomes available.
374
+
375
+ ## Training Code
376
+ OpenCUA models are developed based on the training infrastructure of Kimi Team. We are developting the training pipeline based on the open-source infrastructure as well.
377
+
378
+ ## License
379
+
380
+ This project is licensed under the MIT License - see the LICENSE file in the root folder for details.
381
+
382
+ ## Research Use and Disclaimer
383
+
384
+ OpenCUA models are intended for **research and educational purposes only**.
385
+
386
+ ### Prohibited Uses
387
+ - The model may **not** be used for any purpose or activity that violates applicable laws or regulations in any jurisdiction
388
+ - Use for illegal, unethical, or harmful activities is strictly prohibited
389
+
390
+ ### Disclaimer
391
+ - The authors, contributors, and copyright holders are **not responsible** for any illegal, unethical, or harmful use of the Software, nor for any direct or indirect damages resulting from such use
392
+ - Use of the "OpenCUA" name, logo, or trademarks does **not** imply any endorsement or affiliation unless separate written permission is obtained
393
+ - Users are solely responsible for ensuring their use complies with applicable laws and regulations
394
+
395
+ ## Citation
396
+
397
+ If you use OpenCUA models in your research, please cite our work:
398
+
399
+ ```bibtex
400
+ @misc{wang2025opencuaopenfoundationscomputeruse,
401
+ title={OpenCUA: Open Foundations for Computer-Use Agents},
402
+ author={Xinyuan Wang and Bowen Wang and Dunjie Lu and Junlin Yang and Tianbao Xie and Junli Wang and Jiaqi Deng and Xiaole Guo and Yiheng Xu and Chen Henry Wu and Zhennan Shen and Zhuokai Li and Ryan Li and Xiaochuan Li and Junda Chen and Boyuan Zheng and Peihang Li and Fangyu Lei and Ruisheng Cao and Yeqiao Fu and Dongchan Shin and Martin Shin and Jiarui Hu and Yuyan Wang and Jixuan Chen and Yuxiao Ye and Danyang Zhang and Dikang Du and Hao Hu and Huarong Chen and Zaida Zhou and Haotian Yao and Ziwei Chen and Qizheng Gu and Yipu Wang and Heng Wang and Diyi Yang and Victor Zhong and Flood Sung and Y. Charles and Zhilin Yang and Tao Yu},
403
+ year={2025},
404
+ eprint={2508.09123},
405
+ archivePrefix={arXiv},
406
+ primaryClass={cs.AI},
407
+ url={https://arxiv.org/abs/2508.09123},
408
+ }
409
+ ```
410
+
411
+ </div>
config.bak.json ADDED
@@ -0,0 +1,69 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "OpenCUAForConditionalGeneration"
4
+ ],
5
+ "auto_map": {
6
+ "AutoConfig": "configuration_opencua.OpenCUAConfig",
7
+ "AutoModel": "modeling_opencua.OpenCUAForConditionalGeneration",
8
+ "AutoModelForCausalLM": "modeling_opencua.OpenCUAForConditionalGeneration"
9
+ },
10
+ "ignore_index": -100,
11
+ "media_placeholder_token_id": 151664,
12
+ "model_type": "opencua",
13
+ "pad_token_id": 0,
14
+ "text_config": {
15
+ "bos_token_id": 151643,
16
+ "eos_token_id": 151644,
17
+ "head_dim": 128,
18
+ "hidden_act": "silu",
19
+ "hidden_size": 5120,
20
+ "initializer_range": 0.02,
21
+ "intermediate_size": 27648,
22
+ "k_proj_bias": true,
23
+ "max_length": 20,
24
+ "min_length": 0,
25
+ "model_type": "qwen2",
26
+ "num_attention_heads": 40,
27
+ "num_beam_groups": 1,
28
+ "num_beams": 1,
29
+ "num_hidden_layers": 64,
30
+ "num_key_value_heads": 8,
31
+ "pad_token_id": 152063,
32
+ "pretraining_sequence_length": 131072,
33
+ "q_proj_bias": true,
34
+ "rms_norm_eps": 1e-05,
35
+ "rope_theta": 1000000.0,
36
+ "tie_word_embeddings": false,
37
+ "torch_dtype": "bfloat16",
38
+ "use_bfloat16": false,
39
+ "use_cache": true,
40
+ "v_proj_bias": true,
41
+ "vocab_size": 152064
42
+ },
43
+ "tie_word_embeddings": false,
44
+ "torch_dtype": "bfloat16",
45
+ "transformers_version": "4.48.3",
46
+ "vision_config": {
47
+ "depth": 32,
48
+ "fullatt_block_indexes": [
49
+ 7,
50
+ 15,
51
+ 23,
52
+ 31
53
+ ],
54
+ "hidden_act": "silu",
55
+ "hidden_size": 1280,
56
+ "num_heads": 16,
57
+ "in_chans": 3,
58
+ "intermediate_size": 3456,
59
+
60
+ "patch_size": 14,
61
+ "spatial_merge_size": 2,
62
+ "spatial_patch_size": 14,
63
+ "temporal_patch_size": 2,
64
+ "out_hidden_size": 5120,
65
+ "tokens_per_second": 2,
66
+ "window_size": 112
67
+ },
68
+ "vocab_size": 152064
69
+ }
config.json ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Qwen2_5_VLForConditionalGeneration"
4
+ ],
5
+ "attention_dropout": 0.0,
6
+ "eos_token_id": 151645,
7
+ "hidden_act": "silu",
8
+ "hidden_size": 5120,
9
+ "image_token_id": 151655,
10
+ "initializer_range": 0.02,
11
+ "intermediate_size": 27648,
12
+ "max_position_embeddings": 128000,
13
+ "max_window_layers": 64,
14
+ "model_type": "qwen2_5_vl",
15
+ "num_attention_heads": 40,
16
+ "num_hidden_layers": 64,
17
+ "num_key_value_heads": 8,
18
+ "pad_token_id": 151643,
19
+ "rms_norm_eps": 1e-06,
20
+ "rope_scaling": {
21
+ "mrope_section": [
22
+ 16,
23
+ 24,
24
+ 24
25
+ ],
26
+ "rope_type": "default",
27
+ "type": "default"
28
+ },
29
+ "rope_theta": 1000000.0,
30
+ "sliding_window": 32768,
31
+ "tie_word_embeddings": false,
32
+ "torch_dtype": "bfloat16",
33
+ "transformers_version": "4.49.0",
34
+ "use_cache": true,
35
+ "use_sliding_window": false,
36
+ "video_token_id": 151656,
37
+ "vision_config": {
38
+ "hidden_size": 1280,
39
+ "in_chans": 3,
40
+ "intermediate_size": 3456,
41
+ "model_type": "qwen2_5_vl",
42
+ "out_hidden_size": 5120,
43
+ "spatial_patch_size": 14,
44
+ "tokens_per_second": 2,
45
+ "torch_dtype": "bfloat16"
46
+ },
47
+ "vision_end_token_id": 151653,
48
+ "vision_start_token_id": 151652,
49
+ "vision_token_id": 151654,
50
+ "vocab_size": 152064
51
+ }
configuration_opencua.py ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from transformers.configuration_utils import PretrainedConfig
2
+ from transformers.models.qwen2_5_vl.configuration_qwen2_5_vl import Qwen2_5_VLVisionConfig
3
+ from transformers.models.qwen2.configuration_qwen2 import Qwen2Config
4
+
5
+
6
+ class OpenCUAConfig(PretrainedConfig):
7
+ """OpenCUA-2.5-32B model configuration.
8
+
9
+ Args:
10
+ vision_config: Configuration for the vision model.Qwen2_5_VLVisionConfig
11
+ text_config: Configuration for the text model. Qwen2Config
12
+ pad_token_id: The token ID to use for padding.
13
+ """
14
+
15
+ model_type = "opencua"
16
+
17
+ def __init__(
18
+ self,
19
+ vision_config: dict | Qwen2_5_VLVisionConfig | None = None,
20
+ text_config: dict | Qwen2Config | None = None,
21
+ ignore_index: int = -100,
22
+ media_placeholder_token_id: int = 151664,
23
+ pad_token_id: int = 0,
24
+ **kwargs
25
+ ):
26
+ if isinstance(vision_config, dict):
27
+ vision_config = Qwen2_5_VLVisionConfig(**vision_config)
28
+ self.vision_config = vision_config
29
+
30
+ if isinstance(text_config, dict):
31
+ text_config = Qwen2Config(**text_config)
32
+ self.text_config = text_config
33
+
34
+ self.ignore_index = ignore_index
35
+ self.media_placeholder_token_id = media_placeholder_token_id
36
+
37
+ super().__init__(pad_token_id=pad_token_id, **kwargs)
generation_config.json ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ {
2
+ "max_length": 32768,
3
+ "eos_token_id": 151644
4
+ }
model-1-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0f93f8fdb8948cb1533461a48dbcc53ff3f49334fb4a9f39fba89b030b2671f2
3
+ size 3910073936
model-10-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0dea6a376c59a96a85dbf170bee7709de7591fff8fe3dab573a189f003b8efbf
3
+ size 975212080
model-11-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7305f64fbbd5fed1d919db106f720ea7ad4fdc0a3fbcedd53641bb3aa5cc0f31
3
+ size 975212096
model-12-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fa8097a66699924bbeeaef401c4494bc589b2f03656b631cd294722e1be0e56c
3
+ size 975212096
model-13-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f62d3f60e4e9f5ced3999c7b59791d938f0624c6439d053f1010b3758833d692
3
+ size 975212096
model-14-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:414abeb9ebc8b8d1556cc30654cdd89c29244fa598667d60b4c60acdc05febf1
3
+ size 975212096
model-15-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:07e791bc6e5def800381c0f875e64fe44701ad1d223871929293cb402745abf2
3
+ size 975212096
model-16-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dcfeec3a6e3d3761344d25bc7e64b470497f39b142abe76a27f0be328fa5a51b
3
+ size 975212096
model-17-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5d862948c63a9f309203fc804a6d57249dbf937cfd23cd470f4787ab1b6f408e
3
+ size 975212096
model-18-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0e593c5f9733724fe9e4e97737477425c7ebc4fe4ffc91848b28d2bb1372e725
3
+ size 975212096
model-19-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2fd8ae7e12435ab7a71a933e7d7b4cb542f1d87d9dbba786ea5995b43164fa7c
3
+ size 975212096
model-2-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ba11af2777844ed3e665df561e3505e8a3ed16a0c0f7c5167d30f9b08d0524ff
3
+ size 975212080
model-20-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:20582ca438c35b1818cd372f6602b03bc98d606efb987022939982f9c21d8950
3
+ size 975212096
model-21-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5a66d3be5340ce35964b0217a11ae461332bd1b61461714d7844a695e294bc79
3
+ size 975212096
model-22-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:05a397d86b3124248deb7da026325e94dfe5f9fc2c1afc393aec4101f404ae27
3
+ size 975212096
model-23-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6a5a54d6f0094dff6c34de92caf1e962e36176585769148026efc51305b0060f
3
+ size 975212096
model-24-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3adb06cbc432f29a0144373f44bada870e4b55a9e06d47593cdd69ad427bd7cc
3
+ size 975212096
model-25-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6a9084a804f924021d66a35dc2c3fc35109d2ad7e99fbfdc0106ebe85602eac7
3
+ size 975212096
model-26-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3563166a42ae3c8332ea79ef95aa17597d82bcb52a54695b3281e7e000b80af6
3
+ size 975212096
model-27-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6b29bfc99c4cad7ca5e9b976734051b5b30181371e1a5e007e9dbde27f19e858
3
+ size 975212096
model-28-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:be52480d723a9b68bf1942cb074a5855456b8059ec5d22f99d12d0effc5b4547
3
+ size 975212096
model-29-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3fb36bcb0a2acf028f3e83149d8954ec5f4b5c011b1ca538689e6dda034c298b
3
+ size 975212096
model-3-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f4a858641c4db4486f6142aaf5f7e6e2dba1fd14a7640c31ce9f86289f6cc9ec
3
+ size 975212080
model-30-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:39cf268f7c72fc00bfa8b2a2ea6ab72243e46c88597e3c4aedb768288b534b05
3
+ size 975212096
model-31-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4f54de1fae911dc809f8ca311ff948facdc4c613fc659468aadb6af99b58b125
3
+ size 975212096
model-32-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:937716b987d5994fb46faecc35622ceb0fa18283aac2cbb506c4b9d3f1e3fcae
3
+ size 975212096
model-33-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:08d4297513cb07588a018c0df47fd2849e30e64ebe61675b576dfef2a0a5c831
3
+ size 975212096
model-34-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6651b5a783a8de35e606fe884b29c5a13ba237123972acbdae99fbe44614af57
3
+ size 975212096
model-35-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bfeaf19d699c85bff1aa97b7a9106d2cecc4a6ecb0d6958250ee4b389c3b87f1
3
+ size 975212096
model-36-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:439ea1bf04cc07f00d1e0a8ededd645e89e008fc8eec89c8213a82df7bc93442
3
+ size 975212096
model-37-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:893ef546ef828989c6839b6b0ab02e6e6257bb4eef4ee9fd1deaa46202d62aa4
3
+ size 975212096
model-38-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:447ecde3b04b45b54f6e24cfcd4dfa74cf68023c0f5a22eaaa230380d844dfdf
3
+ size 975212096
model-39-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:51186e7ff2c99616c75a85a28fbb37351e26c5dc2949d23d4373a3467910e935
3
+ size 975212096
model-4-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0d13cc0d8d8f37544d3bf0dbf039f0f94ef1114dbc53a2bb6679cf8c180b77bc
3
+ size 975212080
model-40-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9d1cbb025e2acb4527d4cae146bf3fc53200b5f714828ed452a7085cd40e5064
3
+ size 975212096
model-41-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:51d77fa16c2beed95ba7f5ad37a42e1258acda22640631f0ca0978db1b222672
3
+ size 975212096
model-42-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4e4f70cf44323490fc83788b6ebe670e247851d9c17a86dd1845e7377bf9691b
3
+ size 975212096
model-43-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dcc0ef3ad628eb5c377f26caa4d1340c4b5e941cac9fdfad0e8826b8205a7520
3
+ size 975212096
model-44-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a832b22641ce8b082cbf7f2abb3a6d12cee21211465ec1f6a6945e1b01ca3a08
3
+ size 975212096
model-45-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0f87e97668db5ff3af25b363da785365c5e8c061b0a69a899e148be7951200e3
3
+ size 975212096
model-46-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:942391c6100fbfdad57e548f7d4e9789486a04ee016d6f3edea08d08cb0fa7c2
3
+ size 975212096
model-47-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a04f304e9310861fa3bf4410475f60d8a02fac1ee27bfa4ffd93459f0340860d
3
+ size 975212096
model-48-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fac24ae7395d74a349439772072bbc0bd900d68a8c68f42ee4fa7e5745731a7c
3
+ size 975212096
model-49-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0b944ee340bf8ad7916ba6428ec6479c9d0bda3060fcb2b6a0fdacb6058c7023
3
+ size 975212096
model-5-of-64.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d380966b60109b38e6a060423f347d9c90c84c1be995c1dfa534f412cdee9a65
3
+ size 975212080