Upload model files

README.md

@@ -0,0 +1,495 @@
+---
+base_model:
+- Qwen/Qwen2.5-VL-7B-Instruct
+datasets:
+- xlangai/AgentNet
+- xlangai/aguvis-stage1
+- smolagents/aguvis-stage-2
+- osunlp/UGround-V1-Data
+language:
+- en
+license: mit
+metrics:
+- accuracy
+- code_eval
+pipeline_tag: image-text-to-text
+library_name: transformers
+tags:
+- VLM
+- Computer-Use-Agent
+- OS-Agent
+- GUI
+- Grounding
+---
+
+<h1 style="
+  font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Helvetica,Arial,sans-serif;
+  font-size:48px;
+  font-weight:700;
+  line-height:1.25;
+  text-align:center;
+  margin:0 0 24px;">
+  OpenCUA: Open Foundations for Computer-Use Agents
+</h1>
+
+<div style="
+  display:flex;
+  justify-content:center;
+  gap:12px;
+  flex-wrap:wrap;
+  margin-bottom:28px;">
+  
+  <a href="https://opencua.xlang.ai/" style="
+     display:inline-block;
+     padding:8px 24px;
+     background:#2b2b2b;
+     color:#ffffff;
+     border-radius:36px;
+     text-decoration:none;
+     font-weight:600;
+     font-size:16px;">
+    🌐 Website
+  </a>
+
+  <a href="https://arxiv.org/abs/2508.09123" style="
+     display:inline-block;
+     padding:8px 24px;
+     background:#2b2b2b;
+     color:#ffffff;
+     border-radius:36px;
+     text-decoration:none;
+     font-weight:600;
+     font-size:16px;">
+    📝 Paper
+  </a>
+
+  <a href="https://github.com/xlang-ai/OpenCUA" style="
+     display:inline-block;
+     padding:8px 24px;
+     background:#2b2b2b;
+     color:#ffffff;
+     border-radius:36px;
+     text-decoration:none;
+     font-weight:600;
+     font-size:16px;">
+    💻 Code
+  </a>
+</div>
+
+<div style="max-width:900px;margin:0 auto;">
+
+#  Introduction
+<div style="
+  max-width: 880px;              /* 可按需调节整体宽度 */
+  margin: 0 auto;               /* 居中容器 */
+  text-align: justify;          /* 关键：两端对齐 */
+  text-justify: inter-word;     /* 优化英文对齐效果 */
+  line-height: 1.6;">
+  
+OpenCUA models (OpenCUA-7B and OpenCUA-32B) are end-to-end computer-use foundation models than can produce executable actions in the computer environments.  They are based on the weights of Qwen2.5-VL-7B-Instruction and Qwen2.5-VL-32B-Instruction. 
+They demonstrate superior performance across CUA benchmarks. In particular, <b>OpenCUA-32B</b> achieves an average success rate of **34.8%** on [OSWorld-Verified](https://os-world.github.io/), 
+establishing a new state-of-the-art (SOTA) among open-source models and surpassing OpenAI CUA (GPT-4o). Both models also have strong grounding performance, OpenCUA-32B achieves 59.6% on [OSWorld-G](https://osworld-grounding.github.io/) and 55.3% on [Screenspot-Pro](https://arxiv.org/abs/2504.07981).
+</div>
+
+### Key Features
+
+- **Superior Computer-Use Capablity**: Able to execute multi-step computer-use actions with effective planning and reasoning
+- **Multi-OS Support**: Trained on demonstrations across Ubuntu, Windows, and macOS
+- **Visual Grounding**: Strong GUI element recognition and spatial reasoning capabilities
+- **Multi-Image Context**: Processes up to 3 screenshot history for better context understanding
+- **Reflective Reasoning**: Enhanced with reflective long Chain-of-Thought that identifies errors and provides corrective reasoning
+
+
+# Performance
+
+### Online Agent Evaluation
+OpenCUA models achieves strong performance on **[OSWorld-Verified](https://os-world.github.io/)**. 
+OPENCUA-32B achieves the best performance among all open-source models with an average success rate of 34.8%, outperforming prior baselines by large margins. 
+It also closes the gap to proprietary Claude models.
+<div align="center">
+
+| **Model**                        | **15 Steps** | **50 Steps** | **100 Steps** |
+|-------------------------------|:--------:|:--------:|:---------:|
+| **Proprietary**               |          |          |           |
+| OpenAI CUA                    | 26.0     | 31.3     | 31.4      |
+| Seed 1.5-VL                   | 27.9     | —        | 34.1      |
+| Claude 3.7 Sonnet             | 27.1     | 35.8     | 35.9      |
+| Claude 4 Sonnet               | 31.2     | 43.9     | 41.5      |
+| **Open-Source**               |          |          |           |
+| Qwen 2.5-VL-32B-Instruct       | 3.0      | —        | 3.9       |
+| Qwen 2.5-VL-72B-Instruct       | 4.4      | —        | 5.0       |
+| Kimi-VL-A3B                   | 9.7      | —        | 10.3      |
+| UI-TARS-72B-DPO               | 24.0     | 25.8     | 27.1      |
+| UI-TARS-1.5-7B                | 24.5     | 27.3     | 27.4      |
+| OpenCUA-7B *(Ours)*           | 24.3     | 27.9     | 26.6      |
+| **OpenCUA-32B *(Ours)***      | **29.7** | **34.1** | **34.8**  |
+</div>
+
+*OpenCUA scores are the mean of 3 independent runs.*
+
+### GUI Grounding Performance
+<div align="center">
+
+| **Model** | **OSWorld-G** | **ScreenSpot-V2** | **ScreenSpot-Pro** |
+|-------|-----------|---------------|----------------|
+| Qwen2.5-VL-7B | 31.4 | 88.8 | 27.6 |  
+| Qwen2.5-VL-32B | 46.5 | 87.0 | 39.4 |
+| UI-TARS-72B | 57.1 | 90.3 | 38.1 |
+| **OpenCUA-A3B** | 48.6 | 91.4 | 28.5 |
+| **OpenCUA-Qwen2-7B** | 45.7 | 88.5 | 23.7 |
+| **OpenCUA-7B** | 55.3 | 92.3 | 50.0 |
+| **OpenCUA-32B** | **59.6** | **93.4** | **55.3** |
+</div>
+
+
+### AgentNetBench (Offline Evaluation)
+<div align="center">
+
+| **Model** | **Coordinate Actions** | **Content Actions** | **Function Actions** | **Average** |
+|-------|-------------------|-----------------|------------------|---------|
+| Qwen2.5-VL-7B | 50.7 | 40.8 | 3.1 | 48.0 |
+| Qwen2.5-VL-32B | 66.6 | 47.2 | 41.5 | 64.8 |
+| Qwen2.5-VL-72B | 67.2 | 52.6 | 50.5 | 67.0 |
+| OpenAI CUA          | 71.7 | 57.3 | **80.0** | 73.1 |
+| **OpenCUA-7B**  | 79.0 | 62.0 | 44.3 | 75.2 |
+| **OpenCUA-32B** | **81.9** | 66.1 | 55.7 | **79.1** |
+</div>
+
+#  🚀 Quick Start
+<div style="border-left: 6px solid #f28c28; background: #fff8e6; padding: 12px 16px; margin: 16px 0;">
+  <strong>⚠️ Important for Qwen-based Models (OpenCUA-7B, OpenCUA-32B):</strong>
+  
+  To align with our training infrastructure, we have modified the model in two places:
+  <ul style="margin-top: 8px;">
+    <li>1. Multimodal Rotary Position Embedding (M-RoPE) has been replaced with 1D RoPE</strong>.</li>
+    <li>2. Using the same Tokenizer and ChatTemplate as Kimi-VL.</li>
+    <li>Do not use the default transformers and vllm classes to load the model. Tokenizer and Chat Template should be aligned if training the models.</li>
+  </ul>
+</div>
+
+
+## Installation & Download
+
+First, install the required transformers dependencies:
+
+```bash
+conda create -n opencua python=3.10
+conda activate opencua
+pip install -r requirement.txt
+```
+
+Download the model weight from huggingface:
+```bash
+from huggingface_hub import snapshot_download
+snapshot_download(
+    repo_id="xlangai/OpenCUA-7B",
+    local_dir="OpenCUA-7B",                
+    local_dir_use_symlinks=False  
+)
+```
+
+## 🎯 GUI Grounding 
+
+The following code demonstrates how to use OpenCUA models for GUI grounding tasks:
+
+```python
+import base64
+import torch
+from transformers import AutoTokenizer, AutoModel, AutoImageProcessor
+from PIL import Image
+import json
+
+def encode_image(image_path: str) -> str:
+    """Encode image to base64 string for model input."""
+    with open(image_path, "rb") as f:
+        return base64.b64encode(f.read()).decode()
+
+def load_opencua_model(model_path: str):
+    """Load OpenCUA model, tokenizer, and image processor."""
+    tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
+    model = AutoModel.from_pretrained(
+        model_path, 
+        torch_dtype="auto", 
+        device_map="auto", 
+        trust_remote_code=True
+    )
+    image_processor = AutoImageProcessor.from_pretrained(model_path, trust_remote_code=True)
+    
+    return model, tokenizer, image_processor
+
+def create_grounding_messages(image_path: str, instruction: str):
+    """Create chat messages for GUI grounding task."""
+    system_prompt = (
+        "You are a GUI agent. You are given a task and a screenshot of the screen. "
+        "You need to perform a series of pyautogui actions to complete the task."
+    )
+    
+    messages = [
+        {"role": "system", "content": system_prompt},
+        {
+            "role": "user",
+            "content": [
+                {"type": "image", "image": f"data:image/png;base64,{encode_image(image_path)}"},
+                {"type": "text", "text": instruction},
+            ],
+        },
+    ]
+    return messages
+
+def run_inference(model, tokenizer, image_processor, messages, image_path):
+    """Run inference on the model."""
+    # Prepare text input
+    input_ids = tokenizer.apply_chat_template(
+        messages, tokenize=True, add_generation_prompt=True
+    )
+    input_ids = torch.tensor([input_ids]).to(model.device)
+    
+    # Prepare image input  
+    image = Image.open(image_path).convert('RGB')
+    image_info = image_processor.preprocess(images=[image])
+    pixel_values = torch.tensor(image_info['pixel_values']).to(
+        dtype=torch.bfloat16, device=model.device
+    )
+    grid_thws = torch.tensor(image_info['image_grid_thw'])
+    
+    # Generate response
+    with torch.no_grad():
+        generated_ids = model.generate(
+            input_ids,
+            pixel_values=pixel_values,
+            grid_thws=grid_thws,
+            max_new_tokens=512,
+            temperature=0
+        )
+    
+    # Decode output
+    prompt_len = input_ids.shape[1]
+    generated_ids = generated_ids[:, prompt_len:]
+    output_text = tokenizer.batch_decode(
+        generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
+    )[0]
+    
+    return output_text
+
+# Example usage
+model_path = "OpenCUA/OpenCUA-7B"  # or other model variants
+image_path = "screenshot.png"
+instruction = "Click on the submit button"
+
+# Load model
+model, tokenizer, image_processor = load_opencua_model(model_path)
+
+# Create messages and run inference
+messages = create_grounding_messages(image_path, instruction)
+result = run_inference(model, tokenizer, image_processor, messages, image_path)
+
+print("Model output:", result)
+```
+
+<div style="border-left: 6px solid #9ca3af; background: #f5f5f5; padding: 12px 16px; margin: 16px 0;">
+  <em>Expected result:</em> ```python
+pyautogui.click(x=1443, y=343)
+```
+</div>
+
+You can also run the five grounding examples in [OpenCUA/model/inference/huggingface_inference.py](https://github.com/xlang-ai/OpenCUA/blob/main/model/inference/huggingface_inference.py):
+``` 
+cd ./model/inference/
+python huggingface_inference.py
+```
+
+## 🖥️ Computer Use Agent
+**[OpenCUAAgent](https://github.com/xlang-ai/OSWorld/blob/main/mm_agents/opencua_agent.py)** is developed in the [OSWorld](https://github.com/xlang-ai/OSWorld) environment based on OpenCUA models. It iteratively perceives the environment via screenshots, produces reflective long CoT as inner monologue, and predicts the next action to be executed. OpenCUAAgent uses 3 images in total and L2 CoT format in default.
+
+Command for running OpenCUA-7B and OpenCUA-32B in OSWorld:
+```
+    python run_multienv_opencua.py \
+        --headless \
+        --observation_type screenshot \
+        --model OpenCUA-32B \
+        --result_dir ./results --test_all_meta_path evaluation_examples/test_all_no_gdrive.json \
+        --max_steps 100 \
+        --num_envs 30  \
+        --coordinate_type qwen25
+```
+<div style="border-left: 6px solid #9ca3af; background: #f5f5f5; padding: 12px 16px; margin: 16px 0;">
+  <em>Currently we only supports huggingface inference. We are implementing the vLLM supports of OpenCUA models. Please stay tuned.</em>
+</div>
+
+---
+
+# AgentNet Dataset - Large-Scale Computer-Use Dataset
+
+<div align="center">
+  <img src="https://cdn-uploads.huggingface.co/production/uploads/67b327cdd4665a0448eef7d5/dw5k183ucDSB2SZuS5f2V.png" width="400" alt="AgentNet Dataset Domain Distribution">
+</div>
+
+AgentNet is the first large-scale desktop computer-use agent trajectory dataset, containing 22.6K human-annotated computer-use tasks across Windows, macOS, and Ubuntu systems. 
+
+👉 **[AgentNet Huggingface Dataset](https://huggingface.co/datasets/xlangai/AgentNet)**
+
+Download the dataset here：
+```
+pip install -U huggingface_hub
+huggingface-cli download xlangai/AgentNet --repo-type dataset --local-dir ./AgentNet
+```
+
+Collecting computer-use agent training data requires 3 steps:
+- Demonstrate human computer-use task via [AgentNetTool](https://agentnet-tool.xlang.ai/);
+- Preprocess the demonstration using [Action Reduction & State-Action Matching](./data/data-processor);
+- For each step, [synthesize reflective long CoT](./data/cot-generator)
+
+
+## 1 AgentNetTool – Annotation & Verification Tool
+<div align="center">
+  <img src="https://cdn-uploads.huggingface.co/production/uploads/67b327cdd4665a0448eef7d5/ETjCOoIRR7f1YZCJ2kfiW.png" width="700" alt="AgentNet Tool">
+</div>
+
+
+Our **AgentNetTool** is a cross-platform GUI recorder that runs unobtrusively on annotators’ machines. It captures synchronized **screen video**, **mouse/keyboard events**, and **accessibility trees**, then provides an in-browser UI for reviewing, trimming, and submitting demonstrations. AgentNet Tool is available on Windows, macOS and Ubuntu. 
+
+👉 **[AgentNetTool Document](https://agentnet-tool.xlang.ai/)**
+
+
+
+## 2 DataProcessor – Action Reduction & State–Action Matching
+Raw demonstrations can contain thousands of low-level events that are too dense for model training.  
+The **DataProcessor** module (`./data/data-process/`) performs two key steps:
+
+1.  **Action Reduction** — merges granular signals into concise, semantically meaningful PyAutoGUI actions (e.g., collapsing mouse moves → click, coalescing scrolls, grouping key-press sequences into text or hotkeys).  
+2.  **State–Action Matching** — aligns every reduced action with the *last visually distinct frame* **before** the action begins, avoiding future-information leakage and yielding compact state–action pairs.
+
+These processed trajectories underlie all downstream training and evaluation.
+
+---
+
+## 3 CoTGenerator – Synthesizing Reflective Long Chain-of-Thought Inner Monologue
+To boost robustness and interpretability, we augment each trajectory with **reflective long Chain-of-Thought (CoT) reasoning**.  
+The **CoTGenerator** pipeline (`./data/cot-generator/`) synthesizes step-level reflections that:
+
+*   reflect on the previous action,
+*   explain *why* an action is chosen given the current observation and history,  
+*   note potential alternative actions, and  
+*   forecast the expected next state.
+
+Empirically, models trained with these rich CoTs scale better with data and generalize across unseen applications.
+
+
+# Evaluation
+
+<div align="center">
+  <img src="https://cdn-uploads.huggingface.co/production/uploads/67b327cdd4665a0448eef7d5/emy1QCJwQj9KqHkVmtNH2.png" width="800" alt="AgentNetBench">
+</div>
+
+
+**AgentNetBench** (`./AgentNetBench/`) provides a realistic offline evaluator for OS agent trajectories. It compares model-predicted low-level actions (click, moveTo, write, press, scroll, terminate, etc.) against ground-truth human actions and reports detailed metrics.
+
+👉 See **[AgentNetBench/README.md](./evaluation/agentnetbench/README.md)** for usage instructions.
+
+# TODO
+## vLLM Support
+We are actively working with the vLLM team to add support for OpenCUA models.
+
+**Workaround:** For now, please use the standard transformers library as shown in the examples above. We will update this section once vLLM support becomes available.
+
+## Training Code
+OpenCUA models are developed based on the training infrastructure of Kimi Team. We are developting the training pipeline based on the open-source infrastructure as well.
+
+# Acknowledge
+<p>
+We thank Su Yu, Caiming Xiong, Binyuan Hui, and the anonymous reviewers for their insightful discussions and valuable feedback. 
+We are grateful to Moonshot AI for providing training infrastructure and annotated data. 
+We also sincerely appreciate Calvin, Ziwei Chen, Jin Zhang, Ze Li, Zhengtao Wang, Yanxu Chen, and Qizheng Gu from the Kimi Team for their strong infrastructure support and helpful guidance. 
+The development of our tool is based on the open-source projects-<a href="https://github.com/TheDuckAI/DuckTrack" target="_blank">DuckTrack</a> and <a href="https://github.com/OpenAdaptAI/OpenAdapt" target="_blank">OpenAdapt</a>. 
+We are very grateful to their commitment to the open source community. Finally, we extend our deepest thanks to all annotators for their tremendous effort and contributions to this project.
+</p>
+
+# License
+
+This project is licensed under the MIT License - see the LICENSE file in the root folder for details.
+
+## Research Use and Disclaimer
+
+OpenCUA models are intended for **research and educational purposes only**. 
+
+### Prohibited Uses
+- The model may **not** be used for any purpose or activity that violates applicable laws or regulations in any jurisdiction
+- Use for illegal, unethical, or harmful activities is strictly prohibited
+
+### Disclaimer
+- The authors, contributors, and copyright holders are **not responsible** for any illegal, unethical, or harmful use of the Software, nor for any direct or indirect damages resulting from such use
+- Use of the "OpenCUA" name, logo, or trademarks does **not** imply any endorsement or affiliation unless separate written permission is obtained
+- Users are solely responsible for ensuring their use complies with applicable laws and regulations
+
+## Important Notes on Coordinate Systems
+<div style="border-left: 6px solid #9ca3af; background: #f5f5f5; padding: 12px 16px; margin: 16px 0;">
+  <ul style="margin: 0;">
+    <li><strong><code>OpenCUA/OpenCUA-A3B</code></strong> – Relative coordinates <em>(not supported in this code)</em></li>
+    <li><strong><code>OpenCUA/OpenCUA-Qwen2-7B</code></strong> – Relative coordinates</li>
+    <li><strong><code>OpenCUA/OpenCUA-7B</code></strong> – Absolute coordinates</li>
+    <li><strong><code>OpenCUA/OpenCUA-32B</code></strong> – Absolute coordinates</li>
+  </ul>
+</div>
+
+**OpenCUA models use different coordinate systems depending on the base model:**
+
+- **OpenCUA-Qwen2-7B**: Outputs **relative coordinates** (0.0 to 1.0 range)
+  ```python
+  # Example output: pyautogui.click(x=0.5, y=0.3)
+  # x=0.5 means 50% from left edge, y=0.3 means 30% from top edge
+  
+  # Convert to absolute coordinates:
+  def qwen2_relative_to_absolute(rel_x, rel_y, original_width, original_height):
+      abs_x = int(rel_x * original_width)
+      abs_y = int(rel_y * original_height)
+      return abs_x, abs_y
+  ```
+
+- **OpenCUA-7B and OpenCUA-32B** (Qwen2.5-based): Output **absolute coordinates** after smart resize
+  ```python
+  # Example output: pyautogui.click(x=960, y=324)  
+  # These are coordinates on the smart-resized image, not the original image
+  
+  # Convert to original image coordinates:
+  # Please refer to the smart_resize function in: https://github.com/huggingface/transformers/blob/67ddc82fbc7e52c6f42a395b4a6d278c55b77a39/src/transformers/models/qwen2_vl/image_processing_qwen2_vl.py#L55
+  def qwen25_smart_resize_to_absolute(model_x, model_y, original_width, original_height):
+      # First, calculate the smart-resized dimensions
+      resized_height, resized_width = smart_resize(original_height, original_width, factor = 28, min_pixels = 3136, max_pixels = 12845056)
+      
+      # Convert model output to relative coordinates on original image
+      rel_x = model_x / resized_width
+      rel_y = model_y / resized_height
+      
+      # Then convert to absolute coordinates on original image
+      abs_x = int(rel_x * original_width)
+      abs_y = int(rel_y * original_height)
+      return abs_x, abs_y
+  ```
+
+<div style="border-left: 6px solid #9ca3af; background: #f5f5f5; padding: 12px 16px; margin: 16px 0;">
+  <strong>Understanding Smart Resize for Qwen2.5-based Models:</strong>
+  <p style="margin: 8px 0 0;">
+    The Qwen2.5-VL models use a “smart resize” preprocessing that maintains aspect ratio while fitting within pixel constraints.
+    For coordinate conversion, you need the smart resize function from the
+    <a href="https://github.com/QwenLM/Qwen2.5-VL/blob/d2240f11656bfe404b9ba56db4e51cd09f522ff1/qwen-vl-utils/src/qwen_vl_utils/vision_process.py#L60">
+      official Qwen2.5-VL implementation</a>.
+  </p>
+</div>
+
+## Citation
+
+If you use OpenCUA models in your research, please cite our work:
+
+```bibtex
+@misc{wang2025opencuaopenfoundationscomputeruse,
+      title={OpenCUA: Open Foundations for Computer-Use Agents}, 
+      author={Xinyuan Wang and Bowen Wang and Dunjie Lu and Junlin Yang and Tianbao Xie and Junli Wang and Jiaqi Deng and Xiaole Guo and Yiheng Xu and Chen Henry Wu and Zhennan Shen and Zhuokai Li and Ryan Li and Xiaochuan Li and Junda Chen and Boyuan Zheng and Peihang Li and Fangyu Lei and Ruisheng Cao and Yeqiao Fu and Dongchan Shin and Martin Shin and Jiarui Hu and Yuyan Wang and Jixuan Chen and Yuxiao Ye and Danyang Zhang and Dikang Du and Hao Hu and Huarong Chen and Zaida Zhou and Haotian Yao and Ziwei Chen and Qizheng Gu and Yipu Wang and Heng Wang and Diyi Yang and Victor Zhong and Flood Sung and Y. Charles and Zhilin Yang and Tao Yu},
+      year={2025},
+      eprint={2508.09123},
+      archivePrefix={arXiv},
+      primaryClass={cs.AI},
+      url={https://arxiv.org/abs/2508.09123}, 
+}
+```
+
+</div>

config.bak.json

@@ -0,0 +1,69 @@
+{
+  "architectures": [
+    "OpenCUAForConditionalGeneration"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_opencua.OpenCUAConfig",
+    "AutoModel": "modeling_opencua.OpenCUAForConditionalGeneration",
+    "AutoModelForCausalLM": "modeling_opencua.OpenCUAForConditionalGeneration"
+  },
+  "ignore_index": -100,
+  "media_placeholder_token_id": 151664,
+  "model_type": "opencua",
+  "pad_token_id": 0,
+  "text_config": {
+    "bos_token_id": 151643,
+    "eos_token_id": 151644,
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 3584,
+    "initializer_range": 0.02,
+    "intermediate_size": 18944,
+    "k_proj_bias": true,
+    "max_length": 20,
+    "min_length": 0,
+    "model_type": "qwen2",
+    "num_attention_heads": 28,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_hidden_layers": 28,
+    "num_key_value_heads": 4,
+    "pad_token_id": 152063,
+    "pretraining_sequence_length": 128000,
+    "q_proj_bias": true,
+    "rms_norm_eps": 1e-05,
+    "rope_theta": 1000000.0,
+    "tie_word_embeddings": false,
+    "torch_dtype": "bfloat16",
+    "use_bfloat16": false,
+    "use_cache": true,
+    "v_proj_bias": true,
+    "vocab_size": 152064
+  },
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.48.3",
+  "vision_config": {
+    "depth": 32,
+    "fullatt_block_indexes": [
+      7,
+      15,
+      23,
+      31
+    ],
+    "hidden_act": "silu",
+    "hidden_size": 1280,
+    "num_heads": 16,
+    "in_chans": 3,
+    "intermediate_size": 3420,
+    
+    "patch_size": 14,
+    "spatial_merge_size": 2,
+    "spatial_patch_size": 14,
+    "temporal_patch_size": 2,
+    "out_hidden_size": 3584,
+    "tokens_per_second": 2,
+    "window_size": 112
+  },
+  "vocab_size": 152064
+}

config.json

@@ -0,0 +1,55 @@
+{
+  "architectures": [
+    "Qwen2_5_VLForConditionalGeneration"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 151643,
+  "eos_token_id": 151645,
+  "vision_start_token_id": 151652,
+  "vision_end_token_id": 151653,
+  "vision_token_id": 151654,
+  "image_token_id": 151655,
+  "video_token_id": 151656,
+  "hidden_act": "silu",
+  "hidden_size": 3584,
+  "initializer_range": 0.02,
+  "intermediate_size": 18944,
+  "max_position_embeddings": 128000,
+  "max_window_layers": 28,
+  "model_type": "qwen2_5_vl",
+  "num_attention_heads": 28,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 4,
+  "rms_norm_eps": 1e-06,
+  "rope_theta": 1000000.0,
+  "sliding_window": 32768,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.41.2",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vision_config": {
+    "depth": 32,
+    "hidden_act": "silu",
+    "hidden_size": 1280,
+    "intermediate_size": 3420,
+    "num_heads": 16,
+    "in_chans": 3,
+    "out_hidden_size": 3584,
+    "patch_size": 14,
+    "spatial_merge_size": 2,
+    "spatial_patch_size": 14,
+    "window_size": 112,
+    "fullatt_block_indexes": [
+      7,
+      15,
+      23,
+      31
+    ],
+    "tokens_per_second": 2,
+    "temporal_patch_size": 2
+  },
+  "rope_scaling": {
+    "type": "default"},
+  "vocab_size": 152064
+}

configuration_opencua.py

@@ -0,0 +1,38 @@
+from transformers.configuration_utils import PretrainedConfig
+from transformers.models.qwen2_5_vl.configuration_qwen2_5_vl import Qwen2_5_VLVisionConfig
+from transformers.models.qwen2.configuration_qwen2 import Qwen2Config
+
+
+class OpenCUAConfig(PretrainedConfig):
+    """OpenCUA-2.5-7B model configuration.
+
+    Args:
+        vision_config: Configuration for the vision model.Qwen2_5_VLVisionConfig
+        text_config: Configuration for the text model. Qwen2Config
+        pad_token_id: The token ID to use for padding.
+    """
+
+    model_type = "opencua"
+
+    def __init__(
+        self,
+        vision_config: dict | Qwen2_5_VLVisionConfig | None = None,
+        text_config: dict | Qwen2Config | None = None,
+        ignore_index: int = -100,
+        media_placeholder_token_id: int = 151664,
+        pad_token_id: int = 0,
+        **kwargs
+    ):
+        if isinstance(vision_config, dict):
+            vision_config = Qwen2_5_VLVisionConfig(**vision_config)
+        self.vision_config = vision_config
+
+        if isinstance(text_config, dict):
+            text_config = Qwen2Config(**text_config)
+        self.text_config = text_config
+
+        self.ignore_index = ignore_index
+        self.media_placeholder_token_id = media_placeholder_token_id
+
+        super().__init__(pad_token_id=pad_token_id, **kwargs)
+        

generation_config.json

@@ -0,0 +1,4 @@
+{
+  "max_length": 32768,
+  "eos_token_id": 151644
+}

model-1-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:81b20357bf0a19193a26806b4fbcc0443c882fc694e637e02ebcd5670d0fd4eb
+size 2909254736

model-10-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f0b033c515d92acb396d8bc9978cdcad561a27bca2dc17c7202838f3652d35b
+size 466117168

model-11-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4043300169c14f8a1b1a1b9aead1234394d2d4b61db5d1c77222a90519a9dce8
+size 466117176

model-12-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23a0a37b8918a4c062cd7ed95834f7e2ed6d3801885c8ef4a66c0b681462ff19
+size 466117176

model-13-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:708cae96ee482105895212a1856e0c5a71bbe40471540d34e2d4cbe7fd0ab76a
+size 466117176

model-14-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23490bd2d6389fc8ee01a0f808c865a3b1c6c927e7d21e844975a50041cff79c
+size 466117176

model-15-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:104eee18d818525d3776e1d2bd5fe9c0e5d0d3298b8ab4c59382cd96222b0fc6
+size 466117176

model-16-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e2f3cf3ec4d780acaae3d06139653af043502ad9200560a3dcf41a1ab589145a
+size 466117176

model-17-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a9a7f1e09274bb6a0452c5d2c9280c99b268ceff6bfb33ec2c3b9838e4b0be43
+size 466117176

model-18-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:112aaae346d51b8c75a2ceccaa53d78edbefb6ec4270a805ba38d4d4f5e83f64
+size 466117176

model-19-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2f17abaea5d9a632a63670de0241737d0fecb8b596c99ecf1207f501d900862e
+size 466117176

model-2-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7c45530684019875a72e9cdaff5152f2193bb95c9d227e95880e3c1352ca2d89
+size 466117168

model-20-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ab2ddeedf0122fbbb61da61334a4c2b05527170e86bd989492682c923d20e176
+size 466117176

model-21-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:201213a71cd90492d7c16ef1a7bf1d7c509f3ba8de3878a3e0ad31aa337b9fd3
+size 466117176

model-22-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f9d9a20bc4ab5f20f7cc0235308ca43c0e75b63d6a216828d9f0653a2d61fd09
+size 466117176

model-23-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3e4c6f5b2de2d06d38a36ae83f4bac8f1ecb02b9728189fd20b1faa7220a7060
+size 466117176

model-24-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fa21ebc5c563ee16e07e89c04bea75beb33054232f527e91c1dcb2ab23b1e7c2
+size 466117176

model-25-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f2f82e9856bebb354084dee10c1ea079e04a6e8d75c55f4635046933de332dc2
+size 466117176

model-26-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cc325bb0e58fd55dc98618c8535e840eef815d1afb50b5c13d7a53e6f5075d1f
+size 466117176

model-27-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff43cd1fe2bc5582aafcb76bd7369145df01d2950a43f30f40a08a619ff04808
+size 466117176

model-28-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff0f9999c077caa7920159984ae0116d844c8b1ae8e18215472c8cac8f86fd61
+size 1556119320

model-3-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f102b49039d07ec5a410585b6a97cf749af04958188f25746b1292abb627b2f5
+size 466117168

model-4-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:58d2b72b6e77c062cd798886a6b55a4a5fae73839c28608d6c9cf6f2fce7b483
+size 466117168

model-5-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:288cbf94ad3f543a2a36b0fc4a8f1da7709379016bf55f18d77f58cf076ba154
+size 466117168

model-6-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:663b8b73ea760cf04c68a696aa56bb31f553c72434c182176a49ed797acacbd7
+size 466117168

model-7-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:25286a5c4ff9d4adb178f85c842e54328d8ca919ad5102832cc137eebcc77916
+size 466117168

model-8-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eda0f72734081d6c2c57e76cf9c1f191fd87d0cf117f5a95130928932df79f61
+size 466117168

model-9-of-28.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c7b5d2e1dc2bf66e500b986a1be8eb5ba672e38f2ca201ef01e474b415314c8a
+size 466117168

model.args.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d2489a260658ec0235d9c92a059b58d9f188c5e29034572f61c02e346223508e
+size 80428

model.safetensors.index.json

@@ -0,0 +1,764 @@
+{
+  "metadata": {
+    "total_size": 16584336896
+  },
+  "weight_map": {
+    "model.layers.0.self_attn.q_proj.weight": "model-1-of-28.safetensors",
+    "model.layers.0.self_attn.q_proj.bias": "model-1-of-28.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-1-of-28.safetensors",
+    "model.layers.0.self_attn.k_proj.bias": "model-1-of-28.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-1-of-28.safetensors",
+    "model.layers.0.self_attn.v_proj.bias": "model-1-of-28.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-1-of-28.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-1-of-28.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-1-of-28.safetensors",
+    "model.embed_tokens.weight": "model-1-of-28.safetensors",
+    "visual.blocks.0.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.0.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.0.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.0.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.0.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.0.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.0.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.0.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.0.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.0.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.0.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.0.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.1.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.1.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.1.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.1.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.1.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.1.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.1.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.1.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.1.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.1.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.1.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.1.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.10.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.10.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.10.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.10.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.10.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.10.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.10.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.10.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.10.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.10.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.10.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.10.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.11.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.11.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.11.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.11.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.11.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.11.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.11.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.11.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.11.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.11.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.11.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.11.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.12.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.12.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.12.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.12.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.12.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.12.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.12.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.12.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.12.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.12.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.12.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.12.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.13.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.13.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.13.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.13.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.13.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.13.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.13.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.13.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.13.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.13.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.13.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.13.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.14.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.14.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.14.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.14.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.14.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.14.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.14.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.14.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.14.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.14.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.14.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.14.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.15.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.15.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.15.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.15.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.15.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.15.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.15.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.15.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.15.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.15.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.15.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.15.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.16.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.16.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.16.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.16.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.16.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.16.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.16.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.16.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.16.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.16.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.16.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.16.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.17.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.17.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.17.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.17.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.17.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.17.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.17.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.17.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.17.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.17.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.17.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.17.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.18.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.18.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.18.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.18.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.18.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.18.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.18.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.18.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.18.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.18.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.18.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.18.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.19.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.19.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.19.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.19.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.19.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.19.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.19.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.19.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.19.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.19.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.19.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.19.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.2.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.2.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.2.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.2.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.2.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.2.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.2.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.2.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.2.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.2.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.2.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.2.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.20.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.20.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.20.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.20.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.20.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.20.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.20.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.20.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.20.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.20.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.20.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.20.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.21.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.21.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.21.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.21.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.21.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.21.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.21.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.21.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.21.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.21.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.21.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.21.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.22.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.22.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.22.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.22.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.22.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.22.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.22.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.22.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.22.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.22.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.22.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.22.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.23.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.23.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.23.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.23.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.23.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.23.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.23.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.23.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.23.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.23.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.23.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.23.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.24.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.24.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.24.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.24.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.24.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.24.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.24.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.24.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.24.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.24.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.24.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.24.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.25.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.25.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.25.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.25.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.25.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.25.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.25.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.25.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.25.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.25.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.25.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.25.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.26.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.26.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.26.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.26.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.26.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.26.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.26.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.26.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.26.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.26.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.26.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.26.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.27.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.27.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.27.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.27.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.27.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.27.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.27.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.27.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.27.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.27.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.27.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.27.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.28.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.28.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.28.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.28.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.28.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.28.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.28.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.28.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.28.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.28.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.28.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.28.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.29.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.29.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.29.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.29.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.29.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.29.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.29.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.29.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.29.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.29.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.29.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.29.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.3.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.3.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.3.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.3.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.3.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.3.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.3.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.3.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.3.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.3.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.3.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.3.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.30.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.30.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.30.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.30.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.30.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.30.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.30.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.30.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.30.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.30.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.30.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.30.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.31.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.31.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.31.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.31.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.31.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.31.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.31.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.31.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.31.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.31.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.31.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.31.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.4.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.4.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.4.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.4.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.4.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.4.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.4.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.4.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.4.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.4.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.4.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.4.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.5.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.5.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.5.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.5.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.5.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.5.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.5.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.5.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.5.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.5.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.5.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.5.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.6.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.6.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.6.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.6.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.6.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.6.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.6.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.6.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.6.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.6.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.6.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.6.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.7.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.7.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.7.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.7.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.7.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.7.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.7.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.7.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.7.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.7.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.7.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.7.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.8.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.8.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.8.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.8.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.8.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.8.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.8.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.8.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.8.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.8.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.8.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.8.norm2.weight": "model-1-of-28.safetensors",
+    "visual.blocks.9.attn.proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.9.attn.proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.9.attn.qkv.bias": "model-1-of-28.safetensors",
+    "visual.blocks.9.attn.qkv.weight": "model-1-of-28.safetensors",
+    "visual.blocks.9.mlp.down_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.9.mlp.down_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.9.mlp.gate_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.9.mlp.gate_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.9.mlp.up_proj.bias": "model-1-of-28.safetensors",
+    "visual.blocks.9.mlp.up_proj.weight": "model-1-of-28.safetensors",
+    "visual.blocks.9.norm1.weight": "model-1-of-28.safetensors",
+    "visual.blocks.9.norm2.weight": "model-1-of-28.safetensors",
+    "visual.merger.ln_q.weight": "model-1-of-28.safetensors",
+    "visual.merger.mlp.0.bias": "model-1-of-28.safetensors",
+    "visual.merger.mlp.0.weight": "model-1-of-28.safetensors",
+    "visual.merger.mlp.2.bias": "model-1-of-28.safetensors",
+    "visual.merger.mlp.2.weight": "model-1-of-28.safetensors",
+    "visual.patch_embed.proj.weight": "model-1-of-28.safetensors",
+    "model.layers.0.self_attn.rotary_emb.inv_freq": "model-1-of-28.safetensors",
+    "model.layers.27.self_attn.q_proj.weight": "model-28-of-28.safetensors",
+    "model.layers.27.self_attn.q_proj.bias": "model-28-of-28.safetensors",
+    "model.layers.27.self_attn.k_proj.weight": "model-28-of-28.safetensors",
+    "model.layers.27.self_attn.k_proj.bias": "model-28-of-28.safetensors",
+    "model.layers.27.self_attn.v_proj.weight": "model-28-of-28.safetensors",
+    "model.layers.27.self_attn.v_proj.bias": "model-28-of-28.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-28-of-28.safetensors",
+    "model.layers.27.mlp.gate_proj.weight": "model-28-of-28.safetensors",
+    "model.layers.27.mlp.down_proj.weight": "model-28-of-28.safetensors",
+    "model.layers.27.mlp.up_proj.weight": "model-28-of-28.safetensors",
+    "model.layers.27.input_layernorm.weight": "model-28-of-28.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-28-of-28.safetensors",
+    "model.norm.weight": "model-28-of-28.safetensors",
+    "lm_head.weight": "model-28-of-28.safetensors",
+    "model.layers.27.self_attn.rotary_emb.inv_freq": "model-28-of-28.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-22-of-28.safetensors",
+    "model.layers.21.self_attn.q_proj.bias": "model-22-of-28.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-22-of-28.safetensors",
+    "model.layers.21.self_attn.k_proj.bias": "model-22-of-28.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-22-of-28.safetensors",
+    "model.layers.21.self_attn.v_proj.bias": "model-22-of-28.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-22-of-28.safetensors",
+    "model.layers.21.mlp.gate_proj.weight": "model-22-of-28.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-22-of-28.safetensors",
+    "model.layers.21.mlp.up_proj.weight": "model-22-of-28.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-22-of-28.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-22-of-28.safetensors",
+    "model.layers.21.self_attn.rotary_emb.inv_freq": "model-22-of-28.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-20-of-28.safetensors",
+    "model.layers.19.self_attn.q_proj.bias": "model-20-of-28.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-20-of-28.safetensors",
+    "model.layers.19.self_attn.k_proj.bias": "model-20-of-28.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-20-of-28.safetensors",
+    "model.layers.19.self_attn.v_proj.bias": "model-20-of-28.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-20-of-28.safetensors",
+    "model.layers.19.mlp.gate_proj.weight": "model-20-of-28.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-20-of-28.safetensors",
+    "model.layers.19.mlp.up_proj.weight": "model-20-of-28.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-20-of-28.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-20-of-28.safetensors",
+    "model.layers.19.self_attn.rotary_emb.inv_freq": "model-20-of-28.safetensors",
+    "model.layers.25.self_attn.q_proj.weight": "model-26-of-28.safetensors",
+    "model.layers.25.self_attn.q_proj.bias": "model-26-of-28.safetensors",
+    "model.layers.25.self_attn.k_proj.weight": "model-26-of-28.safetensors",
+    "model.layers.25.self_attn.k_proj.bias": "model-26-of-28.safetensors",
+    "model.layers.25.self_attn.v_proj.weight": "model-26-of-28.safetensors",
+    "model.layers.25.self_attn.v_proj.bias": "model-26-of-28.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-26-of-28.safetensors",
+    "model.layers.25.mlp.gate_proj.weight": "model-26-of-28.safetensors",
+    "model.layers.25.mlp.down_proj.weight": "model-26-of-28.safetensors",
+    "model.layers.25.mlp.up_proj.weight": "model-26-of-28.safetensors",
+    "model.layers.25.input_layernorm.weight": "model-26-of-28.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-26-of-28.safetensors",
+    "model.layers.25.self_attn.rotary_emb.inv_freq": "model-26-of-28.safetensors",
+    "model.layers.24.self_attn.q_proj.weight": "model-25-of-28.safetensors",
+    "model.layers.24.self_attn.q_proj.bias": "model-25-of-28.safetensors",
+    "model.layers.24.self_attn.k_proj.weight": "model-25-of-28.safetensors",
+    "model.layers.24.self_attn.k_proj.bias": "model-25-of-28.safetensors",
+    "model.layers.24.self_attn.v_proj.weight": "model-25-of-28.safetensors",
+    "model.layers.24.self_attn.v_proj.bias": "model-25-of-28.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-25-of-28.safetensors",
+    "model.layers.24.mlp.gate_proj.weight": "model-25-of-28.safetensors",
+    "model.layers.24.mlp.down_proj.weight": "model-25-of-28.safetensors",
+    "model.layers.24.mlp.up_proj.weight": "model-25-of-28.safetensors",
+    "model.layers.24.input_layernorm.weight": "model-25-of-28.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-25-of-28.safetensors",
+    "model.layers.24.self_attn.rotary_emb.inv_freq": "model-25-of-28.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-14-of-28.safetensors",
+    "model.layers.13.self_attn.q_proj.bias": "model-14-of-28.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-14-of-28.safetensors",
+    "model.layers.13.self_attn.k_proj.bias": "model-14-of-28.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-14-of-28.safetensors",
+    "model.layers.13.self_attn.v_proj.bias": "model-14-of-28.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-14-of-28.safetensors",
+    "model.layers.13.mlp.gate_proj.weight": "model-14-of-28.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-14-of-28.safetensors",
+    "model.layers.13.mlp.up_proj.weight": "model-14-of-28.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-14-of-28.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-14-of-28.safetensors",
+    "model.layers.13.self_attn.rotary_emb.inv_freq": "model-14-of-28.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-16-of-28.safetensors",
+    "model.layers.15.self_attn.q_proj.bias": "model-16-of-28.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-16-of-28.safetensors",
+    "model.layers.15.self_attn.k_proj.bias": "model-16-of-28.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-16-of-28.safetensors",
+    "model.layers.15.self_attn.v_proj.bias": "model-16-of-28.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-16-of-28.safetensors",
+    "model.layers.15.mlp.gate_proj.weight": "model-16-of-28.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-16-of-28.safetensors",
+    "model.layers.15.mlp.up_proj.weight": "model-16-of-28.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-16-of-28.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-16-of-28.safetensors",
+    "model.layers.15.self_attn.rotary_emb.inv_freq": "model-16-of-28.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-18-of-28.safetensors",
+    "model.layers.17.self_attn.q_proj.bias": "model-18-of-28.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-18-of-28.safetensors",
+    "model.layers.17.self_attn.k_proj.bias": "model-18-of-28.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-18-of-28.safetensors",
+    "model.layers.17.self_attn.v_proj.bias": "model-18-of-28.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-18-of-28.safetensors",
+    "model.layers.17.mlp.gate_proj.weight": "model-18-of-28.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-18-of-28.safetensors",
+    "model.layers.17.mlp.up_proj.weight": "model-18-of-28.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-18-of-28.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-18-of-28.safetensors",
+    "model.layers.17.self_attn.rotary_emb.inv_freq": "model-18-of-28.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-21-of-28.safetensors",
+    "model.layers.20.self_attn.q_proj.bias": "model-21-of-28.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-21-of-28.safetensors",
+    "model.layers.20.self_attn.k_proj.bias": "model-21-of-28.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-21-of-28.safetensors",
+    "model.layers.20.self_attn.v_proj.bias": "model-21-of-28.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-21-of-28.safetensors",
+    "model.layers.20.mlp.gate_proj.weight": "model-21-of-28.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-21-of-28.safetensors",
+    "model.layers.20.mlp.up_proj.weight": "model-21-of-28.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-21-of-28.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-21-of-28.safetensors",
+    "model.layers.20.self_attn.rotary_emb.inv_freq": "model-21-of-28.safetensors",
+    "model.layers.26.self_attn.q_proj.weight": "model-27-of-28.safetensors",
+    "model.layers.26.self_attn.q_proj.bias": "model-27-of-28.safetensors",
+    "model.layers.26.self_attn.k_proj.weight": "model-27-of-28.safetensors",
+    "model.layers.26.self_attn.k_proj.bias": "model-27-of-28.safetensors",
+    "model.layers.26.self_attn.v_proj.weight": "model-27-of-28.safetensors",
+    "model.layers.26.self_attn.v_proj.bias": "model-27-of-28.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-27-of-28.safetensors",
+    "model.layers.26.mlp.gate_proj.weight": "model-27-of-28.safetensors",
+    "model.layers.26.mlp.down_proj.weight": "model-27-of-28.safetensors",
+    "model.layers.26.mlp.up_proj.weight": "model-27-of-28.safetensors",
+    "model.layers.26.input_layernorm.weight": "model-27-of-28.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-27-of-28.safetensors",
+    "model.layers.26.self_attn.rotary_emb.inv_freq": "model-27-of-28.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-9-of-28.safetensors",
+    "model.layers.8.self_attn.q_proj.bias": "model-9-of-28.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-9-of-28.safetensors",
+    "model.layers.8.self_attn.k_proj.bias": "model-9-of-28.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-9-of-28.safetensors",
+    "model.layers.8.self_attn.v_proj.bias": "model-9-of-28.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-9-of-28.safetensors",
+    "model.layers.8.mlp.gate_proj.weight": "model-9-of-28.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-9-of-28.safetensors",
+    "model.layers.8.mlp.up_proj.weight": "model-9-of-28.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-9-of-28.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-9-of-28.safetensors",
+    "model.layers.8.self_attn.rotary_emb.inv_freq": "model-9-of-28.safetensors",
+    "model.layers.22.self_attn.q_proj.weight": "model-23-of-28.safetensors",
+    "model.layers.22.self_attn.q_proj.bias": "model-23-of-28.safetensors",
+    "model.layers.22.self_attn.k_proj.weight": "model-23-of-28.safetensors",
+    "model.layers.22.self_attn.k_proj.bias": "model-23-of-28.safetensors",
+    "model.layers.22.self_attn.v_proj.weight": "model-23-of-28.safetensors",
+    "model.layers.22.self_attn.v_proj.bias": "model-23-of-28.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-23-of-28.safetensors",
+    "model.layers.22.mlp.gate_proj.weight": "model-23-of-28.safetensors",
+    "model.layers.22.mlp.down_proj.weight": "model-23-of-28.safetensors",
+    "model.layers.22.mlp.up_proj.weight": "model-23-of-28.safetensors",
+    "model.layers.22.input_layernorm.weight": "model-23-of-28.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-23-of-28.safetensors",
+    "model.layers.22.self_attn.rotary_emb.inv_freq": "model-23-of-28.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-17-of-28.safetensors",
+    "model.layers.16.self_attn.q_proj.bias": "model-17-of-28.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-17-of-28.safetensors",
+    "model.layers.16.self_attn.k_proj.bias": "model-17-of-28.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-17-of-28.safetensors",
+    "model.layers.16.self_attn.v_proj.bias": "model-17-of-28.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-17-of-28.safetensors",
+    "model.layers.16.mlp.gate_proj.weight": "model-17-of-28.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-17-of-28.safetensors",
+    "model.layers.16.mlp.up_proj.weight": "model-17-of-28.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-17-of-28.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-17-of-28.safetensors",
+    "model.layers.16.self_attn.rotary_emb.inv_freq": "model-17-of-28.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-19-of-28.safetensors",
+    "model.layers.18.self_attn.q_proj.bias": "model-19-of-28.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-19-of-28.safetensors",
+    "model.layers.18.self_attn.k_proj.bias": "model-19-of-28.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-19-of-28.safetensors",
+    "model.layers.18.self_attn.v_proj.bias": "model-19-of-28.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-19-of-28.safetensors",
+    "model.layers.18.mlp.gate_proj.weight": "model-19-of-28.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-19-of-28.safetensors",
+    "model.layers.18.mlp.up_proj.weight": "model-19-of-28.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-19-of-28.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-19-of-28.safetensors",
+    "model.layers.18.self_attn.rotary_emb.inv_freq": "model-19-of-28.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-13-of-28.safetensors",
+    "model.layers.12.self_attn.q_proj.bias": "model-13-of-28.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-13-of-28.safetensors",
+    "model.layers.12.self_attn.k_proj.bias": "model-13-of-28.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-13-of-28.safetensors",
+    "model.layers.12.self_attn.v_proj.bias": "model-13-of-28.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-13-of-28.safetensors",
+    "model.layers.12.mlp.gate_proj.weight": "model-13-of-28.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-13-of-28.safetensors",
+    "model.layers.12.mlp.up_proj.weight": "model-13-of-28.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-13-of-28.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-13-of-28.safetensors",
+    "model.layers.12.self_attn.rotary_emb.inv_freq": "model-13-of-28.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-11-of-28.safetensors",
+    "model.layers.10.self_attn.q_proj.bias": "model-11-of-28.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-11-of-28.safetensors",
+    "model.layers.10.self_attn.k_proj.bias": "model-11-of-28.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-11-of-28.safetensors",
+    "model.layers.10.self_attn.v_proj.bias": "model-11-of-28.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-11-of-28.safetensors",
+    "model.layers.10.mlp.gate_proj.weight": "model-11-of-28.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-11-of-28.safetensors",
+    "model.layers.10.mlp.up_proj.weight": "model-11-of-28.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-11-of-28.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-11-of-28.safetensors",
+    "model.layers.10.self_attn.rotary_emb.inv_freq": "model-11-of-28.safetensors",
+    "model.layers.23.self_attn.q_proj.weight": "model-24-of-28.safetensors",
+    "model.layers.23.self_attn.q_proj.bias": "model-24-of-28.safetensors",
+    "model.layers.23.self_attn.k_proj.weight": "model-24-of-28.safetensors",
+    "model.layers.23.self_attn.k_proj.bias": "model-24-of-28.safetensors",
+    "model.layers.23.self_attn.v_proj.weight": "model-24-of-28.safetensors",
+    "model.layers.23.self_attn.v_proj.bias": "model-24-of-28.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-24-of-28.safetensors",
+    "model.layers.23.mlp.gate_proj.weight": "model-24-of-28.safetensors",
+    "model.layers.23.mlp.down_proj.weight": "model-24-of-28.safetensors",
+    "model.layers.23.mlp.up_proj.weight": "model-24-of-28.safetensors",
+    "model.layers.23.input_layernorm.weight": "model-24-of-28.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-24-of-28.safetensors",
+    "model.layers.23.self_attn.rotary_emb.inv_freq": "model-24-of-28.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-12-of-28.safetensors",
+    "model.layers.11.self_attn.q_proj.bias": "model-12-of-28.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-12-of-28.safetensors",
+    "model.layers.11.self_attn.k_proj.bias": "model-12-of-28.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-12-of-28.safetensors",
+    "model.layers.11.self_attn.v_proj.bias": "model-12-of-28.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-12-of-28.safetensors",
+    "model.layers.11.mlp.gate_proj.weight": "model-12-of-28.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-12-of-28.safetensors",
+    "model.layers.11.mlp.up_proj.weight": "model-12-of-28.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-12-of-28.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-12-of-28.safetensors",
+    "model.layers.11.self_attn.rotary_emb.inv_freq": "model-12-of-28.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-4-of-28.safetensors",
+    "model.layers.3.self_attn.q_proj.bias": "model-4-of-28.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-4-of-28.safetensors",
+    "model.layers.3.self_attn.k_proj.bias": "model-4-of-28.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-4-of-28.safetensors",
+    "model.layers.3.self_attn.v_proj.bias": "model-4-of-28.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-4-of-28.safetensors",
+    "model.layers.3.mlp.gate_proj.weight": "model-4-of-28.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-4-of-28.safetensors",
+    "model.layers.3.mlp.up_proj.weight": "model-4-of-28.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-4-of-28.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-4-of-28.safetensors",
+    "model.layers.3.self_attn.rotary_emb.inv_freq": "model-4-of-28.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-15-of-28.safetensors",
+    "model.layers.14.self_attn.q_proj.bias": "model-15-of-28.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-15-of-28.safetensors",
+    "model.layers.14.self_attn.k_proj.bias": "model-15-of-28.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-15-of-28.safetensors",
+    "model.layers.14.self_attn.v_proj.bias": "model-15-of-28.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-15-of-28.safetensors",
+    "model.layers.14.mlp.gate_proj.weight": "model-15-of-28.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-15-of-28.safetensors",
+    "model.layers.14.mlp.up_proj.weight": "model-15-of-28.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-15-of-28.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-15-of-28.safetensors",
+    "model.layers.14.self_attn.rotary_emb.inv_freq": "model-15-of-28.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-7-of-28.safetensors",
+    "model.layers.6.self_attn.q_proj.bias": "model-7-of-28.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-7-of-28.safetensors",
+    "model.layers.6.self_attn.k_proj.bias": "model-7-of-28.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-7-of-28.safetensors",
+    "model.layers.6.self_attn.v_proj.bias": "model-7-of-28.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-7-of-28.safetensors",
+    "model.layers.6.mlp.gate_proj.weight": "model-7-of-28.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-7-of-28.safetensors",
+    "model.layers.6.mlp.up_proj.weight": "model-7-of-28.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-7-of-28.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-7-of-28.safetensors",
+    "model.layers.6.self_attn.rotary_emb.inv_freq": "model-7-of-28.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-6-of-28.safetensors",
+    "model.layers.5.self_attn.q_proj.bias": "model-6-of-28.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-6-of-28.safetensors",
+    "model.layers.5.self_attn.k_proj.bias": "model-6-of-28.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-6-of-28.safetensors",
+    "model.layers.5.self_attn.v_proj.bias": "model-6-of-28.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-6-of-28.safetensors",
+    "model.layers.5.mlp.gate_proj.weight": "model-6-of-28.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-6-of-28.safetensors",
+    "model.layers.5.mlp.up_proj.weight": "model-6-of-28.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-6-of-28.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-6-of-28.safetensors",
+    "model.layers.5.self_attn.rotary_emb.inv_freq": "model-6-of-28.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-5-of-28.safetensors",
+    "model.layers.4.self_attn.q_proj.bias": "model-5-of-28.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-5-of-28.safetensors",
+    "model.layers.4.self_attn.k_proj.bias": "model-5-of-28.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-5-of-28.safetensors",
+    "model.layers.4.self_attn.v_proj.bias": "model-5-of-28.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-5-of-28.safetensors",
+    "model.layers.4.mlp.gate_proj.weight": "model-5-of-28.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-5-of-28.safetensors",
+    "model.layers.4.mlp.up_proj.weight": "model-5-of-28.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-5-of-28.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-5-of-28.safetensors",
+    "model.layers.4.self_attn.rotary_emb.inv_freq": "model-5-of-28.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-3-of-28.safetensors",
+    "model.layers.2.self_attn.q_proj.bias": "model-3-of-28.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-3-of-28.safetensors",
+    "model.layers.2.self_attn.k_proj.bias": "model-3-of-28.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-3-of-28.safetensors",
+    "model.layers.2.self_attn.v_proj.bias": "model-3-of-28.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-3-of-28.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-3-of-28.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-3-of-28.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-3-of-28.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-3-of-28.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-3-of-28.safetensors",
+    "model.layers.2.self_attn.rotary_emb.inv_freq": "model-3-of-28.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-8-of-28.safetensors",
+    "model.layers.7.self_attn.q_proj.bias": "model-8-of-28.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-8-of-28.safetensors",
+    "model.layers.7.self_attn.k_proj.bias": "model-8-of-28.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-8-of-28.safetensors",
+    "model.layers.7.self_attn.v_proj.bias": "model-8-of-28.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-8-of-28.safetensors",
+    "model.layers.7.mlp.gate_proj.weight": "model-8-of-28.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-8-of-28.safetensors",
+    "model.layers.7.mlp.up_proj.weight": "model-8-of-28.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-8-of-28.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-8-of-28.safetensors",
+    "model.layers.7.self_attn.rotary_emb.inv_freq": "model-8-of-28.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-10-of-28.safetensors",
+    "model.layers.9.self_attn.q_proj.bias": "model-10-of-28.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-10-of-28.safetensors",
+    "model.layers.9.self_attn.k_proj.bias": "model-10-of-28.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-10-of-28.safetensors",
+    "model.layers.9.self_attn.v_proj.bias": "model-10-of-28.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-10-of-28.safetensors",
+    "model.layers.9.mlp.gate_proj.weight": "model-10-of-28.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-10-of-28.safetensors",
+    "model.layers.9.mlp.up_proj.weight": "model-10-of-28.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-10-of-28.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-10-of-28.safetensors",
+    "model.layers.9.self_attn.rotary_emb.inv_freq": "model-10-of-28.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-2-of-28.safetensors",
+    "model.layers.1.self_attn.q_proj.bias": "model-2-of-28.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-2-of-28.safetensors",
+    "model.layers.1.self_attn.k_proj.bias": "model-2-of-28.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-2-of-28.safetensors",
+    "model.layers.1.self_attn.v_proj.bias": "model-2-of-28.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-2-of-28.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-2-of-28.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-2-of-28.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-2-of-28.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-2-of-28.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-2-of-28.safetensors",
+    "model.layers.1.self_attn.rotary_emb.inv_freq": "model-2-of-28.safetensors"
+  }
+}

modeling_opencua.py

@@ -0,0 +1,449 @@
+# ------------------------------------------------------------------------------
+# OpenCUA‑7B Model
+#
+# This implementation is adapted from the Qwen2‑VL reference code in
+# Hugging Face Transformers v4.53.0:
+#   https://github.com/huggingface/transformers/tree/v4.53.0/src/transformers/models/qwen2_5_vl
+#
+# Checkpoint used for weight initialisation:
+#   "Qwen/Qwen2.5-VL-7B-Instruct"  –  https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct
+#
+# Key modifications
+# -----------------
+# • Replaced Multimodal Rotary Position Embedding (M‑RoPE) with 1‑D RoPE for
+#   compatibility with OpenCUA training settings.
+# • Wrapped vision encoder and language model into a single
+#   `OpenCUAForConditionalGeneration` class.
+# • Simplified weight initialisation — this file targets inference / fine‑tuning,
+#   not training from scratch.
+#
+# Copyright (c) 2025 XLANG Lab, The University of Hong Kong
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the “Software”), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+# 
+# ------------------------------------------------------------------------------
+# Prohibited Uses & Additional Disclaimer
+# ---------------------------------------
+# • The Software may **not** be used for any purpose or activity that violates
+#   applicable laws or regulations in any jurisdiction.
+# • The authors, contributors, and copyright holders are **not responsible**
+#   for any illegal, unethical, or harmful use of the Software, nor for any
+#   direct or indirect damages resulting from such use.
+# • Use of the “OpenCUA” name, logo, or trademarks does **not** imply any
+#   endorsement or affiliation unless a separate written permission is obtained.
+
+import torch
+import torch.nn as nn
+from transformers.cache_utils import Cache
+from transformers.modeling_utils import PreTrainedModel
+from transformers.models.llava.modeling_llava import LlavaCausalLMOutputWithPast
+
+from .configuration_opencua import OpenCUAConfig
+from transformers.models.qwen2_5_vl.modeling_qwen2_5_vl import Qwen2_5_VisionTransformerPretrainedModel
+from transformers.models.qwen2.modeling_qwen2 import Qwen2ForCausalLM
+
+
+class OpenCUAPreTrainedModel(PreTrainedModel):
+    config_class = OpenCUAConfig
+    base_model_prefix = "model"
+    _no_split_modules = ["Qwen2_5_VisionTransformerPretrainedModel"]
+    _skip_keys_device_placement = "past_key_values"
+    _supports_flash_attn_2 = True
+
+    def _init_weights(self, module):
+        # important: this ported version of Llava isn't meant for training from scratch - only
+        # inference and fine-tuning - so the proper init weights code has been removed - the original codebase
+        # https://github.com/haotian-liu/LLaVA/tree/main/llava should serve for that purpose
+        std = (
+            self.config.initializer_range
+            if hasattr(self.config, "initializer_range")
+            else self.config.text_config.initializer_range
+        )
+
+        if hasattr(module, "class_embedding"):
+            module.class_embedding.data.normal_(mean=0.0, std=std)
+
+        if isinstance(module, (nn.Linear, nn.Conv2d)):
+            module.weight.data.normal_(mean=0.0, std=std)
+            if module.bias is not None:
+                module.bias.data.zero_()
+        elif isinstance(module, nn.Embedding):
+            module.weight.data.normal_(mean=0.0, std=std)
+            if module.padding_idx is not None:
+                module.weight.data[module.padding_idx].zero_()
+
+    @property
+    def _supports_sdpa(self):
+        """
+        Retrieve language_model's attribute to check whether the model supports
+        SDPA or not.
+        """
+        return self.language_model._supports_sdpa
+
+
+class OpenCUAForConditionalGeneration(OpenCUAPreTrainedModel):
+
+    def __init__(self, config: OpenCUAConfig):
+        super().__init__(config)
+        self.vision_tower = Qwen2_5_VisionTransformerPretrainedModel(config.vision_config)
+        self.language_model = Qwen2ForCausalLM(config.text_config)
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.language_model.get_input_embeddings()
+
+    def set_input_embeddings(self, value):
+        self.language_model.set_input_embeddings(value)
+
+    def get_output_embeddings(self):
+        return self.language_model.get_output_embeddings()
+
+    def set_output_embeddings(self, new_embeddings):
+        self.language_model.set_output_embeddings(new_embeddings)
+
+    def set_decoder(self, decoder):
+        self.language_model.set_decoder(decoder)
+
+    def get_decoder(self):
+        return self.language_model.get_decoder()
+
+    def tie_weights(self):
+        return self.language_model.tie_weights()
+
+    def resize_token_embeddings(self, new_num_tokens: int | None = None, pad_to_multiple_of=None) -> nn.Embedding:
+        model_embeds = self.language_model.resize_token_embeddings(
+            new_num_tokens, pad_to_multiple_of)
+        # update vocab size
+        self.config.text_config.vocab_size = model_embeds.num_embeddings
+        self.vocab_size = model_embeds.num_embeddings
+        return model_embeds
+
+    def _merge_input_ids_with_image_features(
+            self,
+            image_features: torch.Tensor,
+            feature_lengths: list[int],
+            inputs_embeds: torch.Tensor,
+            input_ids: torch.Tensor,
+            attention_mask: torch.Tensor,
+            labels: torch.Tensor | None = None):
+        """
+        Args:
+            image_features (:obj:`torch.Tensor` of shape :obj:`(num_image_tokens, embed_dim)`):
+                The image features to merge with the input embeddings.
+            feature_lengths: the length of image feature.
+            inputs_embeds (:obj:`torch.Tensor` of shape :obj:`(batch_size, sequence_length, embed_dim)`):
+                The input embeddings.
+            input_ids (:obj:`torch.Tensor` of shape :obj:`(batch_size, sequence_length)`):
+                The input ids.
+            attention_mask (:obj:`torch.Tensor` of shape :obj:`(batch_size, sequence_length)`):
+                The attention mask.
+            labels (:obj:`torch.Tensor` of shape :obj:`(batch_size, sequence_length)`, *optional*):
+                The labels.
+        """
+
+        image_token_index: int = self.config.media_placeholder_token_id
+        pad_token_id: int = self.config.pad_token_id
+        ignore_index: int = self.config.ignore_index
+
+        _, embed_dim = image_features.shape
+
+        batch_size, sequence_length = input_ids.shape
+        left_padding = not torch.sum(
+            input_ids[:, -1] == torch.tensor(pad_token_id))
+
+        # 1. Create a mask to know where special image tokens are
+        _token_occupation_table = torch.ones_like(input_ids.flatten())
+        _token_occupation_table[input_ids.flatten() == image_token_index] = \
+            torch.tensor(feature_lengths,
+                         dtype=torch.long, device=input_ids.device)
+        _token_occupation_table = _token_occupation_table.reshape(
+            input_ids.shape)
+
+        max_embed_dim = _token_occupation_table.sum(-1).max().item()
+        assert max_embed_dim >= sequence_length, (
+            f"The maximum embedding dimension ({max_embed_dim}) is less than the sequence length ({sequence_length})"
+        )
+        batch_indices, non_image_indices = torch.where(input_ids != image_token_index)
+
+        # 2. Compute the positions where text should be written
+        # Calculate new positions for text tokens in merged image-text sequence.
+        new_token_positions = torch.cumsum(_token_occupation_table, -1) - 1
+        nb_image_pad = max_embed_dim - 1 - new_token_positions[:, -1]
+        if left_padding:
+            new_token_positions += nb_image_pad[:, None]  # offset for left padding
+        text_to_overwrite = new_token_positions[batch_indices, non_image_indices]
+
+        # 3. Create the full embedding, already padded to the maximum position
+        final_embedding = torch.zeros(
+            batch_size, max_embed_dim, embed_dim, dtype=inputs_embeds.dtype, device=inputs_embeds.device
+        )
+        final_attention_mask = torch.zeros(
+            batch_size, max_embed_dim, dtype=attention_mask.dtype, device=inputs_embeds.device
+        )
+        if labels is not None:
+            final_labels = torch.full(
+                (batch_size, max_embed_dim), ignore_index, dtype=input_ids.dtype, device=input_ids.device
+            )
+        # In case the Vision model or the Language model has been offloaded to CPU, we need to manually
+        # set the corresponding tensors into their correct target device.
+        target_device = inputs_embeds.device
+        batch_indices, non_image_indices, text_to_overwrite = (
+            batch_indices.to(target_device),
+            non_image_indices.to(target_device),
+            text_to_overwrite.to(target_device),
+        )
+        attention_mask = attention_mask.to(target_device)
+
+        # 4. Fill the embeddings based on the mask.
+        final_embedding[batch_indices, text_to_overwrite] = inputs_embeds[batch_indices, non_image_indices]
+        final_attention_mask[batch_indices, text_to_overwrite] = attention_mask[batch_indices, non_image_indices]
+        if labels is not None:
+            final_labels[batch_indices, text_to_overwrite] = labels[batch_indices, non_image_indices]
+
+        # 5. Fill the embeddings corresponding to the images. Anything that is not `text_positions` needs filling (#29835)
+        image_to_overwrite = torch.full(
+            (batch_size, max_embed_dim), True, dtype=torch.bool, device=inputs_embeds.device
+        )
+        image_to_overwrite[batch_indices, text_to_overwrite] = False
+        image_to_overwrite &= image_to_overwrite.cumsum(-1) - 1 >= nb_image_pad[:, None].to(target_device)
+
+        if image_to_overwrite.sum() != image_features.shape[:-1].numel():
+            raise ValueError(
+                f"The input provided to the model are wrong. The number of image tokens is {image_to_overwrite.sum()} while"
+                f" the number of image features given to the model is {image_features.shape[:-1].numel()}. "
+                "This prevents correct indexing and breaks batch generation."
+            )
+
+        final_embedding[image_to_overwrite] = image_features.contiguous().reshape(-1, embed_dim).to(target_device)
+        final_attention_mask |= image_to_overwrite
+        position_ids = (final_attention_mask.cumsum(-1) - 1).masked_fill_((final_attention_mask == 0), 1)
+
+        # 6. Mask out the embedding at padding positions, as we later use the past_key_value value to determine the non-attended tokens.
+        batch_indices, pad_indices = torch.where(input_ids == pad_token_id)
+        indices_to_mask = new_token_positions[batch_indices, pad_indices]
+
+        final_embedding[batch_indices, indices_to_mask] = 0
+
+        if labels is None:
+            final_labels = None
+
+        return final_embedding, final_attention_mask, final_labels, position_ids
+
+    def _extract_image_features(self,
+                                pixel_values: torch.FloatTensor | list[torch.FloatTensor],
+                                grid_thws: torch.FloatTensor,
+                                ):
+        """
+        Args:
+            pixel_values (:obj:`torch.FloatTensor` of shape :obj:`(sum_num_image_tokens, channels)`):
+                The pixel values of the images processed by image processor.
+            grid_thws: (B,3)
+
+        Returns:
+            selected_image_feature (:obj:`torch.FloatTensor` of shape :obj:`(num_image_tokens, embed_dim)`):
+                The selected image features to use as input to the projector head.
+            
+        """
+
+        assert len(grid_thws.shape)==2 and grid_thws.shape[1]==3, f"grid_thws must be a 2D tensor with shape (batched, 3), but got {grid_thws.shape}"
+        if isinstance(pixel_values, list):
+            pixel_values = torch.cat(pixel_values, dim=0)
+        image_features_ = self.vision_tower(pixel_values, grid_thw=grid_thws)
+        image_features_list = []
+        start_idx = 0
+        for i, grid_thw in enumerate(grid_thws):
+            end_idx = start_idx + (grid_thw[0] * grid_thw[1] * grid_thw[2]) // 4
+            image_features_list.append(image_features_[start_idx:end_idx, :])
+            start_idx = end_idx
+
+        selected_image_feature = torch.cat(image_features_list, dim=0)
+        feature_lengths = [x.size(0) for x in image_features_list]
+        return selected_image_feature, feature_lengths
+
+    def forward(
+        self,
+        input_ids: torch.LongTensor | None = None,
+        pixel_values: torch.FloatTensor | list[torch.FloatTensor] | None = None,
+        grid_thws: torch.Tensor = None,
+        attention_mask: torch.Tensor | None = None,
+        position_ids: torch.LongTensor | None = None,
+        past_key_values: list[torch.FloatTensor] | None = None,
+        inputs_embeds: torch.FloatTensor | None = None,
+        labels: torch.LongTensor | None = None,
+        use_cache: bool | None = None,
+        output_attentions: bool | None = None,
+        output_hidden_states: bool | None = None,
+        return_dict: bool | None = None,
+    ) -> tuple | LlavaCausalLMOutputWithPast:
+        r"""
+        Args:
+            labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
+                Labels for computing the masked language modeling loss. Indices should either be in `[0, ...,
+                config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored
+                (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.
+
+        ```"""
+
+        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+        if inputs_embeds is None:
+            # 1. Extra the input embeddings
+            inputs_embeds = self.get_input_embeddings()(input_ids)
+            # 2. Merge text and images
+            if pixel_values is not None and len(pixel_values) > 0 and input_ids.shape[1] != 1:
+                image_feature, feature_lengths = self._extract_image_features(
+                    pixel_values, grid_thws)
+
+                inputs_embeds = inputs_embeds.to(image_feature.dtype)  # num_tokens, embed_dim
+                inputs_embeds, attention_mask, labels, position_ids = \
+                    self._merge_input_ids_with_image_features(image_feature, feature_lengths, inputs_embeds, input_ids, attention_mask, labels
+                    )
+            # In case input_ids.shape[1] == 1 & pixel_values==None & past_key_values != None, we are in the case of
+            # generation with cache
+            elif past_key_values is not None and pixel_values is not None and input_ids.shape[1] == 1:
+                # Retrieve the first layer to inspect the logits and mask out the hidden states
+                # that are set to 0
+                first_layer_past_key_value = past_key_values[0][0][:, :, :, 0]
+
+                # Sum all dimensions of head_dim (-2) to avoid random errors such as: https://github.com/huggingface/transformers/pull/28032#issuecomment-1863691941
+                batch_index, non_attended_tokens = torch.where(first_layer_past_key_value.float().sum(-2) == 0)
+
+                # Get the target length
+                target_length = input_ids.shape[1]
+                past_length = first_layer_past_key_value.shape[-1]
+
+                extended_attention_mask = torch.ones(
+                    (attention_mask.shape[0], past_length),
+                    dtype=attention_mask.dtype,
+                    device=attention_mask.device,
+                )
+
+                # Filter out only the tokens that can be un-attended, this can happen
+                # if one uses Llava + Fused modules where the cache on the
+                # first iteration is already big enough, or if one passes custom cache
+                valid_indices = non_attended_tokens < extended_attention_mask.size(-1)
+                new_batch_index = batch_index[valid_indices]
+                new_non_attended_tokens = non_attended_tokens[valid_indices]
+
+                # Zero-out the places where we don't need to attend
+                extended_attention_mask[new_batch_index, new_non_attended_tokens] = 0
+
+                attention_mask = torch.cat((extended_attention_mask, attention_mask[:, -target_length:]), dim=1)
+                position_ids = torch.sum(attention_mask, dim=1).unsqueeze(-1) - 1
+
+        outputs = self.language_model(
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+
+        logits = outputs[0]
+
+        loss = None
+        if labels is not None:
+            # Shift so that tokens < n predict n
+            if attention_mask is not None:
+                shift_attention_mask = attention_mask[..., 1:]
+                shift_logits = logits[..., :-1, :][shift_attention_mask.to(logits.device) != 0].contiguous()
+                shift_labels = labels[..., 1:][shift_attention_mask.to(labels.device) != 0].contiguous()
+            else:
+                shift_logits = logits[..., :-1, :].contiguous()
+                shift_labels = labels[..., 1:].contiguous()
+            # Flatten the tokens
+            loss_fct = nn.CrossEntropyLoss()
+            loss = loss_fct(
+                shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1).to(shift_logits.device)
+            )
+
+        if not return_dict:
+            output = (logits,) + outputs[1:]
+            return (loss,) + output if loss is not None else output
+
+        return LlavaCausalLMOutputWithPast(
+            loss=loss,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+    def prepare_inputs_for_generation(
+        self, input_ids, past_key_values=None, inputs_embeds=None, pixel_values=None, grid_thws=None, attention_mask=None, **kwargs
+    ):
+        if past_key_values is not None:
+            if isinstance(past_key_values, Cache):
+                cache_length = past_key_values.get_seq_length()
+                past_length = past_key_values.seen_tokens
+            else:
+                cache_length = past_length = past_key_values[0][0].shape[2]
+
+            # Keep only the unprocessed tokens:
+            # 1 - If the length of the attention_mask exceeds the length of input_ids, then we are in a setting where
+            # some of the inputs are exclusively passed as part of the cache (e.g. when passing input_embeds as
+            # input)
+            if attention_mask is not None and attention_mask.shape[1] > input_ids.shape[1]:
+                input_ids = input_ids[:, -(attention_mask.shape[1] - past_length) :]
+            # 2 - If the past_length is smaller than input_ids', then input_ids holds all input tokens. We can discard
+            # input_ids based on the past_length.
+            elif past_length < input_ids.shape[1]:
+                input_ids = input_ids[:, past_length:]
+            # 3 - Otherwise (past_length >= input_ids.shape[1]), let's assume input_ids only has unprocessed tokens.
+            elif self.config.media_placeholder_token_id in input_ids:
+                input_ids = input_ids[:, input_ids.shape[1] - 1 :]
+            # If the cache has seen more tokens than it can hold, then the cache has a size limit. Let's discard the
+            # older attention values, as their corresponding values are not part of the input.
+            if cache_length < past_length and attention_mask is not None:
+                attention_mask = attention_mask[:, -(cache_length + input_ids.shape[1]) :]
+
+        position_ids = kwargs.get("position_ids", None)
+        if attention_mask is not None and position_ids is None:
+            # create position_ids on the fly for batch generation
+            position_ids = attention_mask.long().cumsum(-1) - 1
+            position_ids.masked_fill_(attention_mask == 0, 1)
+            if past_key_values:
+                position_ids = position_ids[:, -input_ids.shape[1] :]
+
+        # if `inputs_embeds` are passed, we only want to use them in the 1st generation step
+        if inputs_embeds is not None and past_key_values is None:
+            model_inputs = {"inputs_embeds": inputs_embeds}
+        else:
+            model_inputs = {"input_ids": input_ids}
+
+        model_inputs.update(
+            {
+                "position_ids": position_ids,
+                "past_key_values": past_key_values,
+                "use_cache": kwargs.get("use_cache"),
+                "attention_mask": attention_mask,
+                "pixel_values": pixel_values,
+                "grid_thws": grid_thws,
+            }
+        )
+        return model_inputs
+
+    def _reorder_cache(self, *args, **kwargs):
+        return self.language_model._reorder_cache(*args, **kwargs)

preprocessor_config.json

@@ -0,0 +1,18 @@
+{
+  "min_pixels": 3136,
+  "max_pixels": 12845056,
+  "patch_size": 14,
+  "temporal_patch_size": 2,
+  "merge_size": 2,
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "image_processor_type": "Qwen2VLImageProcessor"
+}

processing_opencua.py

@@ -0,0 +1,26 @@
+# processing_opencua.py
+from transformers import Qwen2_5_VLProcessor, AutoTokenizer, AutoImageProcessor
+
+class OpenCUAProcessor(Qwen2_5_VLProcessor):
+    # 用字符串就行，但我们会在 from_pretrained 里手动加载，避免字符串反射
+    tokenizer_class = "TikTokenV3"
+
+    @classmethod
+    def from_pretrained(cls, pretrained_model_name_or_path, **kwargs):
+        # 确保 remote code 可用
+        trust_remote_code = kwargs.get("trust_remote_code", False)
+
+        # 1) 手动加载 tokenizer（会按模型目录里的 tokenizer_config.json -> TikTokenV3 + tokenization_opencua.py）
+        tokenizer = AutoTokenizer.from_pretrained(
+            pretrained_model_name_or_path,
+            trust_remote_code=trust_remote_code,
+        )
+
+        # 2) 手动加载图像处理器（保持 Qwen2VLImageProcessor）
+        image_processor = AutoImageProcessor.from_pretrained(
+            pretrained_model_name_or_path,
+            trust_remote_code=trust_remote_code,
+        )
+
+        # 3) 构造并返回 Qwen2.5-VL 的 Processor 实例
+        return cls(image_processor=image_processor, tokenizer=tokenizer)

processor_config.json

@@ -0,0 +1,4 @@
+{
+  "processor_class": "processing_opencua.OpenCUAProcessor",
+  "image_processor_type": "Qwen2VLImageProcessor"
+}

tiktoken.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b2b1b8dfb5cc5f024bafc373121c6aba3f66f9a5a0269e243470a1de16a33186
+size 2561218

tokenization_opencua.py

@@ -0,0 +1,379 @@
+import os
+import tiktoken
+
+from logging import getLogger
+from pathlib import Path
+from typing import (
+    cast,
+    Tuple,
+    Dict,
+    Iterator,
+    List,
+    Union,
+    Optional,
+)
+from shutil import copyfile
+from tiktoken.load import load_tiktoken_bpe
+from tokenizers import AddedToken
+from transformers.tokenization_utils import PreTrainedTokenizer
+from transformers.models.gpt2.tokenization_gpt2 import bytes_to_unicode
+
+# 导入Qwen2Tokenizer用于继承
+try:
+    from transformers.models.qwen2.tokenization_qwen2 import Qwen2Tokenizer
+    QWEN2_AVAILABLE = True
+except ImportError:
+    QWEN2_AVAILABLE = False
+    Qwen2Tokenizer = PreTrainedTokenizer
+
+
+logger = getLogger(__name__)
+VOCAB_FILES_NAMES = {"vocab_file": "tiktoken.model"}
+
+class TikTokenTokenizer(PreTrainedTokenizer):
+    """
+    Tokenizing and encoding/decoding text using the Tiktoken tokenizer. See megatron/tokenizer/tiktoken_tokenizer.py.
+
+    This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
+    this superclass for more information regarding those methods.
+
+    Args:
+        vocab_file (`str`):
+            The path to the Tiktoken model file.
+        bos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `"<|begin_of_text|>",`):
+            The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
+        eos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `"<|end_of_text|>"`):
+            The end of sequence token.
+        unk_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `"<|reserved_special_token_249|>"`):
+            The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
+            token instead. The second to last item in special_tokens.
+        pad_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `"<|reserved_special_token_250|>"`):
+            The token used for padding, for example when batching sequences of different lengths.
+        additional_special_tokens (list of `str`, *optional*):
+            A tuple or a list of additional tokens, which will be marked as `special`, meaning that they will be
+            skipped when decoding if `skip_special_tokens` is set to `True`.
+    """
+
+    vocab_files_names = VOCAB_FILES_NAMES
+
+    model_input_names = ["input_ids", "attention_mask"]
+
+    special_tokens: Dict[str, int]
+
+    num_reserved_special_tokens = 256
+
+    pat_str = "|".join(
+        [
+            r"""[\p{Han}]+""",
+            r"""[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}&&[^\p{Han}]]*[\p{Ll}\p{Lm}\p{Lo}\p{M}&&[^\p{Han}]]+(?i:'s|'t|'re|'ve|'m|'ll|'d)?""",
+            r"""[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}&&[^\p{Han}]]+[\p{Ll}\p{Lm}\p{Lo}\p{M}&&[^\p{Han}]]*(?i:'s|'t|'re|'ve|'m|'ll|'d)?""",
+            r"""\p{N}{1,3}""",
+            r""" ?[^\s\p{L}\p{N}]+[\r\n]*""",
+            r"""\s*[\r\n]+""",
+            r"""\s+(?!\S)""",
+            r"""\s+""",
+        ]
+    )
+
+    def __init__(
+        self,
+        vocab_file,
+        bos_token: Union[str, AddedToken]="[BOS]",
+        eos_token: Union[str, AddedToken]="[EOS]",
+        unk_token: Union[str, AddedToken, None]=None,
+        pad_token: Union[str, AddedToken, None]=None,
+        additional_special_tokens: List[str]=None,
+        added_tokens_decoder: Optional[dict] = None,
+        **kwargs,
+    ):
+        assert os.path.isfile(vocab_file), vocab_file
+
+        if additional_special_tokens is None:
+            # dumping mode
+            used_special_tokens  = [
+                "<|im_end|>", 
+                "<|im_user|>", 
+                "<|im_assistant|>", 
+                "<|reserved_token_0|>",
+                "<|start_header_id|>", 
+                "<|end_header_id|>", 
+                "<|reserved_token_1|>", 
+                "[EOT]",  
+                "<|im_system|>", 
+                "<|reserved_token_2|>",
+                "<|reserved_token_3|>",
+                "<|reserved_token_4|>",
+                "<|reserved_token_5|>",
+                "<|reserved_token_6|>", 
+                "<|reserved_token_7|>",
+                "<|im_middle|>", 
+                "<|media_begin|>",
+                "<|media_content|>", 
+                "<|media_end|>", 
+                "<|media_placeholder|>",
+                # 添加标准Qwen2.5-VL需要的token
+                "<|vision_start|>",
+                "<|vision_end|>", 
+                "<|image_pad|>",
+                "<|video_pad|>",
+            ] 
+            used_reserved_tokens = 12  # 原来8个 + 新增4个vision相关token
+            last_reserved_token_id = self.num_reserved_special_tokens - 4 - len(used_special_tokens)  + used_reserved_tokens - 1
+            additional_special_tokens = used_special_tokens + [
+                f"<|reserved_token_{i}|>"
+                for i in range(used_reserved_tokens, last_reserved_token_id + 1)
+            ]
+            # num_reserved_special_tokens = additional_special_tokens + BOS + EOS + unk_token + pad_token
+            assert len(additional_special_tokens) + 4 == self.num_reserved_special_tokens, f"additional_special_tokens num: {len(additional_special_tokens)} is not correct"
+            # we assume that the instance is under initialization and unk_token and pad_token should be automatically inferred
+            if unk_token is not None:
+                raise ValueError("unk_token should not be set in dumping mode when additional_special_tokens is None")
+            if pad_token is not None:
+                raise ValueError("pad_token should not be set in dumping mode when additional_special_tokens is None")
+            # last two reserved tokens
+            unk_token = f"[UNK]"
+            pad_token = f"[PAD]"
+
+            logger.info(f"adding unk_token: {unk_token} and pad_token: {pad_token}")
+            self.additional_special_tokens = additional_special_tokens
+            special_tokens = [str(bos_token), str(eos_token)] + additional_special_tokens + [str(unk_token), str(pad_token)]
+
+            self.vocab_file = vocab_file
+            mergeable_ranks = load_tiktoken_bpe(vocab_file)
+            num_base_tokens = len(mergeable_ranks)
+            self.special_tokens = {
+                token: num_base_tokens + i for i, token in enumerate(special_tokens)
+            }
+        else:
+            self.additional_special_tokens = additional_special_tokens
+            special_tokens_mapping = {
+            i: added_tokens_decoder[i].content for i in added_tokens_decoder
+        }
+
+            self.vocab_file = vocab_file
+            mergeable_ranks = load_tiktoken_bpe(vocab_file)
+            num_base_tokens = len(mergeable_ranks)
+            self.special_tokens = {
+                special_tokens_mapping.get(i, f"<|reserved_token_{i}|>"): i
+                for i in range(
+                    num_base_tokens, num_base_tokens + self.num_reserved_special_tokens + 2
+                )
+            }
+
+
+
+        self.model = tiktoken.Encoding(
+            name=Path(vocab_file).name,
+            pat_str=self.pat_str,
+            mergeable_ranks=mergeable_ranks,
+            special_tokens=self.special_tokens,
+        )
+        logger.info(f"Reloaded tiktoken model from {vocab_file}")
+
+        self.n_words: int = self.model.n_vocab
+        # BOS / EOS token IDs
+        self.bos_id: int = self.special_tokens[str(bos_token)]
+        self.eos_id: int = self.special_tokens[str(eos_token)]
+        
+        logger.info(
+            f"#words: {self.n_words} - BOS ID: {self.bos_id} - EOS ID: {self.eos_id}"
+        )
+
+        self.pad_id: int = self.special_tokens[str(pad_token)]
+        self.unk_id: int = self.special_tokens[str(unk_token)]
+        self.byte_encoder = bytes_to_unicode()
+        self.byte_decoder = {v: k for k, v in self.byte_encoder.items()}
+
+        self.decoder = {}
+        for i in range(self.n_words):
+            # Taken from https://gist.github.com/xenova/a452a6474428de0182b17605a98631ee
+            decoding = ''.join([
+                self.byte_encoder[ord(char)] for char in
+                self.model.decode_single_token_bytes(i).decode('latin-1')
+            ])
+            self.decoder[i] = decoding
+
+        self.encoder = {}
+        for i in range(self.n_words):
+            if i in self.decoder:
+                self.encoder[self.decoder[i]] = i
+
+        super().__init__(
+            bos_token=bos_token,
+            eos_token=eos_token,
+            unk_token=unk_token,
+            pad_token=pad_token,
+            additional_special_tokens=self.additional_special_tokens,
+            **kwargs,
+        )
+        self.all_special_ids_set = set(self.all_special_ids)
+
+    def encode(
+        self,
+        text: str,
+        allow_special_tokens = True,
+        **kwargs
+    ) -> List[int]:
+        """
+        Encodes a string into a list of token IDs.
+
+        Args:
+            text (str): The input string to be encoded.
+
+        Returns:
+            list[int]: A list of token IDs.
+        """
+        # If there are other args, we should call super().encode because there are a lot of code
+        # to handle those args. supper().encode finally will call _tokenize and _convert_token_to_id.
+        # NOTE: our encode method is not compatible with the super().encode method,
+        #   e.g. split_special_tokens' default is True in our encode method.
+        if len(kwargs) > 0:
+            logger.warning( f"Calling super().encode with {kwargs}" )
+            return super().encode(text, **kwargs)
+
+        assert type(text) is str
+
+        # The tiktoken tokenizer can handle <=400k chars without
+        # pyo3_runtime.PanicException.
+        TIKTOKEN_MAX_ENCODE_CHARS = 400_000
+
+        # https://github.com/openai/tiktoken/issues/195
+        # Here we iterate over subsequences and split if we exceed the limit
+        # of max consecutive non-whitespace or whitespace characters.
+        MAX_NO_WHITESPACES_CHARS = 25_000
+
+        texts = self.pre_tokenizer_process(text)
+
+        all_substrs = []
+        for text in texts:
+            substrs = (
+                substr
+                for i in range(0, len(text), TIKTOKEN_MAX_ENCODE_CHARS)
+                for substr in self._split_whitespaces_or_nonwhitespaces(
+                    text[i: i + TIKTOKEN_MAX_ENCODE_CHARS], MAX_NO_WHITESPACES_CHARS
+                )
+            )
+            all_substrs.extend(substrs)
+
+        t: List[int] = []
+        for substr in all_substrs:
+            if allow_special_tokens:
+                t.extend(
+                    self.model.encode(
+                        substr,
+                        allowed_special="all",
+                    )
+                )
+            else:
+                t.extend(
+                    self.model.encode(
+                        substr,
+                        disallowed_special=(),
+                    )
+                )
+
+        return t
+
+    def decode(
+        self,
+        token_ids: Union[int, List[int]],
+        **kwargs
+    ) -> str:
+        """
+        Decodes a list of token IDs into a string.
+
+        Args:
+            token_ids (List[int]): The list of token IDs to be decoded.
+
+        Returns:
+            str: The decoded string.
+        """
+        # If there are other args, we should call super().decode because there are a lot of code
+        # to handle those args. supper().encode finally will call convert_tokens_to_string and _convert_id_to_token.
+        if len(kwargs) > 0:
+            return super().decode(token_ids, **kwargs)
+
+        if type(token_ids) is int:
+            token_ids = [token_ids]
+
+        return self.model.decode(cast(List[int], token_ids))
+
+    @staticmethod
+    def _split_whitespaces_or_nonwhitespaces(
+        s: str, max_consecutive_slice_len: int
+    ) -> Iterator[str]:
+        """
+        Splits the string `s` so that each substring contains no more than `max_consecutive_slice_len`
+        consecutive whitespaces or consecutive non-whitespaces.
+        """
+        current_slice_len = 0
+        current_slice_is_space = s[0].isspace() if len(s) > 0 else False
+        slice_start = 0
+
+        for i in range(len(s)):
+            is_now_space = s[i].isspace()
+
+            if current_slice_is_space ^ is_now_space:
+                current_slice_len = 1
+                current_slice_is_space = is_now_space
+            else:
+                current_slice_len += 1
+                if current_slice_len > max_consecutive_slice_len:
+                    yield s[slice_start:i]
+                    slice_start = i
+                    current_slice_len = 1
+        yield s[slice_start:]
+
+    def pre_tokenizer_process(self, text: str) -> List[str]:
+        """
+        pre-tokenizes the input text into a list of tokens.
+        This method is used to split the input text into smaller chunks for internal processing.
+        """
+        return [text]
+
+
+    """ ----- Below are the abstract methods required by PreTrainedTokenizer ----- """
+    @property
+    def vocab_size(self) -> int:
+        return self.n_words
+
+    def get_vocab(self) -> Dict[str, int]:
+        return self.encoder
+
+    def _tokenize(self, text: str, **kwargs) -> List[str]:
+        return [
+            self.decoder[t]
+            for t in self.encode(text)
+        ]
+
+    def _convert_token_to_id(self, token: str) -> int:
+        return self.encoder.get(token, self.unk_id)
+
+    def _convert_id_to_token(self, index: int) -> str:
+        return self.decoder.get(index)
+
+    @staticmethod
+    def clean_up_tokenization(out_string: str) -> str:
+        return out_string
+
+    def convert_tokens_to_string(self, tokens: List[str]) -> str:
+        text = ''.join(tokens)
+        text = bytearray([self.byte_decoder[c] for c in text]).decode('utf-8', 'replace')
+        return text
+
+    def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
+        if not os.path.isdir(save_directory):
+            raise ValueError(f"vocabulary path ({save_directory}) should be a directory")
+        out_vocab_file = os.path.join(
+            save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
+        )
+
+        if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file) and os.path.isfile(self.vocab_file):
+            copyfile(self.vocab_file, out_vocab_file)
+
+        return (out_vocab_file,)
+
+
+class TikTokenV3(TikTokenTokenizer):
+    num_reserved_special_tokens = 293 + 128
+    pat_str = "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"

tokenizer_config.json

@@ -0,0 +1,270 @@
+{
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "[BOS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "[EOS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|im_user|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|im_assistant|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|reserved_token_0|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|start_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|end_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|reserved_token_1|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "[EOT]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|im_system|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|reserved_token_2|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|reserved_token_3|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|reserved_token_4|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<|reserved_token_5|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151658": {
+      "content": "<|reserved_token_6|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151659": {
+      "content": "<|reserved_token_7|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151660": {
+      "content": "<|im_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151661": {
+      "content": "<|media_begin|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151662": {
+      "content": "<|media_content|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151663": {
+      "content": "<|media_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151664": {
+      "content": "<|media_placeholder|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151665": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151666": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151667": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151668": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "152062": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "152063": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+
+  },
+  "additional_special_tokens": [
+    "<|im_end|>",
+    "<|im_user|>", 
+    "<|im_assistant|>", 
+    "<|reserved_token_0|>",
+    "<|start_header_id|>", 
+    "<|end_header_id|>", 
+    "<|reserved_token_1|>", 
+    "[EOT]",  
+    "<|im_system|>", 
+    "<|reserved_token_2|>", 
+    "<|reserved_token_3|>", 
+    "<|reserved_token_4|>", 
+    "<|reserved_token_5|>",
+    "<|reserved_token_6|>",
+    "<|reserved_token_7|>",
+    "<|im_middle|>", 
+    "<|media_begin|>", 
+    "<|media_content|>", 
+    "<|media_end|>", 
+    "<|media_placeholder|>",
+    "<|vision_start|>",
+    "<|vision_end|>", 
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": "[BOS]",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "[EOS]",
+  "extra_special_tokens": {},
+  "chat_template": "{%- for message in messages -%}{%- if loop.first and messages[0]['role'] != 'system' -%}{{'<|im_system|>system<|im_middle|>You are a helpful assistant<|im_end|>'}}{%- endif -%}{%- if message['role'] == 'system' -%}{{'<|im_system|>'}}{%- endif -%}{%- if message['role'] == 'user' -%}{{'<|im_user|>'}}{%- endif -%}{%- if message['role'] == 'assistant' -%}{{'<|im_assistant|>'}}{%- endif -%}{{- message['role'] -}}{{'<|im_middle|>'}}{%- if message['content'] is string -%}{{- message['content'] + '<|im_end|>' -}}{%- else -%}{%- for content in message['content'] -%}{%- if content['type'] == 'image' or 'image' in content or 'image_url' in content -%}{{'<|media_begin|>image<|media_content|><|media_placeholder|><|media_end|>'}}{%- else -%}{{content['text']}}{%- endif -%}{%- endfor -%}{{'<|im_end|>'}}{%- endif -%}{%- endfor -%}{%- if add_generation_prompt -%}{{'<|im_assistant|>assistant<|im_middle|>'}}{%- endif -%}",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "[PAD]",
+  "tokenizer_class": "TikTokenV3",
+  "unk_token": "[UNK]",
+  "auto_map": {
+    "AutoTokenizer": [
+      "tokenization_opencua.TikTokenV3",
+      null
+    ]
+  }
+}