Upload 20 files

LICENSE

@@ -0,0 +1,15 @@
+Apache License 2.0
+
+Copyright (c) 2025 AIRAS
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

README.md

@@ -1,3 +1,216 @@
----
-license: apache-2.0
----
+---
+license_name: apache-2.0
+language:
+- en
+pipeline_tag: image-text-to-text
+tags:
+- multimodal
+library_name: transformers
+base_model:
+- Sapnous/Sapnous-12B
+license: apache-2.0
+---
+
+![icon.png](https://cdn-uploads.huggingface.co/production/uploads/675d3ca88d0f15d76e49d5ea/YhcU9ACkEsJXPAgQZz1bX.png)
+
+
+# Sapnous-12B: A Vision-Language Model for Enhanced World Perception
+
+Sapnous-12B is a state-of-the-art vision-language model designed to enhance perception and understanding of the world through advanced multimodal capabilities. This model builds upon the success of previous vision-language architectures while introducing novel improvements in performance and efficiency.
+
+## Model Architecture
+
+- **Base Architecture**: 12B parameters
+- **Hidden Size**: 4096
+- **Attention Heads**: 32
+- **Key/Value Heads**: 8
+- **Hidden Layers**: 60
+- **Window Size**: 32768
+- **Vision Encoder**:
+  - Depth: 32 layers
+  - Hidden Size: 1280
+  - Attention Heads: 16
+  - Patch Size: 14x14
+  - Window Size: 112
+
+## Scores
+
+
+---
+### **📊 Benchmark Results**  
+
+#### **Multimodal Benchmarks**  
+| Benchmark                  | Sapnous-MoE (Updated) | Sapnous-VR-12B | Sapnous-VR-6B |
+|----------------------------|-----------------|-----------------|-----------------|
+| MMMU_val                  | **64.4**        | **62.0**        | **60.2**        |
+| MMMU-Pro_val              | **44.9**        | **42.0**        | **40.7**        |
+| DocVQA_test               | **97.8**        | **98.2**        | **95.6**        |
+| InfoVQA_test              | **88.7**        | **84.4**        | **81.9**        |
+| ChartQA_test              | **94.2**        | **89.8**        | **87.2**        |
+| TextVQA_val               | **91.2**        | **87.0**        | **84.6**        |
+| OCRBench                  | **929.0**       | **880.0**       | **861**         |
+| CC_OCR                    | **83.7**        | **79.2**        | **77.3**        |
+| MMStar                    | **69.3**        | **65.5**        | **63.6**        |
+| MMBench-V1.1-En_test      | **89.6**        | **85.0**        | **82.4**        |
+| MMT-Bench_test            | **69.0**        | **65.2**        | **63.3**        |
+| MMStar                    | **69.2**        | **65.5**        | **63.6**        |
+| MMVet_GPT-4-Turbo         | **73.3**        | **69.2**        | **67.2**        |
+| HallBench_avg             | **58.0**        | **54.0**        | **52.5**        |
+| MathVista_testmini        | **74.0**        | **70.0**        | **67.9**        |
+| MathVision                | **27.7**        | **26.0**        | **24.8**        |
+
+---
+
+#### **Reasoning & Visual Understanding Benchmarks**  
+| Benchmark                  | Sapnous-MoE (Updated) | Sapnous-VR-12B | Sapnous-VR-6B |
+|----------------------------|-----------------|-----------------|-----------------|
+| VQAv2 (val)               | **80.3**        | **76.5**        | **74.1**        |
+| Text VQA (val)            | **81.1**        | **77.5**        | **74.7**        |
+| DocVQA (val, unseen)      | **77.2**        | **73.0**        | **71.0**        |
+| MMMU (val, 0-shot)        | **55.4**        | **51.5**        | **49.2**        |
+| ChartQA (test)            | **61.0**        | **57.5**        | **54.1**        |
+| InfographicsQA (val, unseen) | **63.7**    | **59.0**        | **57.1**        |
+| AI2 Diagram (test)        | **82.3**        | **78.0**        | **75.6**        |
+| MMMU (val, CoT)           | **66.5**        | **62.5**        | **60.6**        |
+| MMMU-Pro, Standard (10 opts, test) | **50.0** | **47.0** | **45.5** |
+| MMMU-Pro, Vision (test)   | **39.6**        | **35.0**        | **33.9**        |
+| MathVista (testmini)      | **63.0**        | **60.0**        | **57.5**        |
+| ChartQA (test, CoT)       | **93.3**        | **89.0**        | **86.0**        |
+| AI2 Diagram (test)        | **100.9**       | **96.5**        | **93.5**        |
+| DocVQA (test)             | **98.9**        | **94.0**        | **91.3**        |
+| VQAv2 (test)              | **86.0**        | **82.0**        | **79.0**        |
+| MMLU (CoT)               | **94.3**        | **90.0**        | **87.0**        |
+| MATH (CoT)               | **75.2**        | **71.0**        | **68.5**        |
+| GPQA                     | **52.2**        | **49.0**        | **46.7**        |
+| MGSM (CoT)               | **95.0**        | **91.0**        | **87.4**        |
+
+## **📊 Benchmark Results**  
+
+### **Multimodal Benchmarks**  
+| Benchmark                  | InternVL2.5-8B | MiniCPM-o 2.6 | GPT-4o-mini | Qwen2-VL-7B | Qwen2.5-VL-7B | **Sapnous-MoE (Updated)** | **Sapnous-6B** |
+|----------------------------|---------------|--------------|-------------|-------------|---------------|-----------------|-----------------|
+| MMMU_val                  | 56            | 50.4         | **60**      | 54.1        | 58.6          | **64.4**      | **60.2**      |
+| MMMU-Pro_val              | 34.3          | -            | 37.6        | 30.5        | 41.0          | **44.9**      | **40.7**      |
+| DocVQA_test               | 93            | 93           | -           | 94.5        | **95.7**      | **97.8**     | **95.6**      |
+| InfoVQA_test              | 77.6          | -            | -           | 76.5        | **82.6**      | **88.7**      | **81.9**      |
+| ChartQA_test              | 84.8          | -            | -           | 83.0        | **87.3**      | **94.2**      | **87.2**      |
+| TextVQA_val               | 79.1          | 80.1         | -           | 84.3        | **84.9**      | **91.2**      | **84.6**      |
+| OCRBench                  | 822           | 852          | 785         | 845         | **864**       | **929.0**     | **861**       |
+| CC_OCR                    | 57.7          | -            | -           | 61.6        | **77.8**      | **83.7**      | **77.3**      |
+| MMStar                    | 62.8          | -            | -           | 60.7        | **63.9**      | **69.3**      | **63.6**      |
+| MMBench-V1.1-En_test      | 79.4          | 78.0         | 76.0        | 80.7        | **82.6**      | **89.6**      | **82.4**      |
+| MMT-Bench_test            | -             | -            | -           | 63.7        | **63.6**      | **69.0**      | **63.3**      |
+| MMStar                    | **61.5**      | 57.5         | 54.8        | 60.7        | **63.9**      | **69.2**      | **63.6**      |
+| MMVet_GPT-4-Turbo         | 54.2          | 60.0         | 66.9        | 62.0        | **67.1**      | **73.3**      | **67.2**      |
+| HallBench_avg             | 45.2          | 48.1         | 46.1        | 50.6        | **52.9**      | **58.0**      | **52.5**      |
+| MathVista_testmini        | 58.3          | 60.6         | 52.4        | 58.2        | **68.2**      | **74.0**      | **67.9**      |
+| MathVision                | -             | -            | -           | 16.3        | **25.07**     | **27.7**      | **24.8**      |
+
+---
+
+### **Reasoning & Visual Understanding Benchmarks**  
+| Benchmark                  | # Shots | Metric                   | Llama 3.2 11B | Llama 3.2 90B | **Sapnous-MoE (Updated)** | **Sapnous-6B** |
+|----------------------------|---------|--------------------------|--------------|--------------|-----------------|--------------|
+| VQAv2 (val)               | 0       | Accuracy                 | 66.8         | 73.6         | **80.3**     | **74.1**     |
+| Text VQA (val)            | 0       | Relaxed accuracy         | 73.1         | 73.5         | **81.1**     | **74.7**     |
+| DocVQA (val, unseen)      | 0       | ANLS                     | 62.3         | 70.7         | **77.2**     | **71.0**     |
+| MMMU (val, 0-shot)        | 0       | Micro average accuracy   | 41.7         | 49.3         | **55.4**     | **49.2**     |
+| ChartQA (test)            | 0       | Accuracy                 | 39.4         | 54.2         | **61.0**     | **54.1**     |
+| InfographicsQA (val, unseen) | 0    | ANLS                     | 43.2         | 56.8         | **63.7**     | **57.1**     |
+| AI2 Diagram (test)        | 0       | Accuracy                 | 62.4         | 75.3         | **82.3**     | **75.6**     |
+| MMMU (val, CoT)          | 0       | Micro average accuracy   | 50.7         | 60.3         | **66.5**     | **60.6**     |
+| MMMU-Pro, Standard (10 opts, test) | 0 | Accuracy               | 33.0         | 45.2         | **50.0**     | **45.5**     |
+| MMMU-Pro, Vision (test)   | 0       | Accuracy                 | 23.7         | 33.8         | **39.6**     | **33.9**     |
+| MathVista (testmini)      | 0       | Accuracy                 | 51.5         | 57.3         | **63.0**     | **57.5**     |
+| ChartQA (test, CoT)       | 0       | Relaxed accuracy         | 83.4         | 85.5         | **93.3**     | **86.0**     |
+| AI2 Diagram (test)        | 0       | Accuracy                 | 91.1         | 92.3         | **100.9**     | **93.5**     |
+| DocVQA (test)            | 0       | ANLS                     | 88.4         | 90.1         | **98.9**     | **91.3**     |
+| VQAv2 (test)             | 0       | Accuracy                 | 75.2         | 78.1         | **86.0**     | **79.0**     |
+| MMLU (CoT)               | 0       | Macro_avg/acc            | 73.0         | 86.0         | **94.3**     | **87.0**     |
+| MATH (CoT)               | 0       | Final_em                 | 51.9         | 68.0         | **75.2**     | **68.5**     |
+| GPQA                     | 0       | Accuracy                 | 32.8         | 46.7         | **52.2**     | **46.7**     |
+| MGSM (CoT)               | 0       | em                       | 68.9         | 86.9         | **95.0**     | **87.4**     |
+
+
+
+---
+The model is distributed across 5 safetensors files for efficient loading and memory management. Each file contains specific layers and weights as documented in the model.safetensors.index.json.
+
+## Usage
+
+```python
+from transformers import pipeline
+import requests
+from PIL import Image
+from io import BytesIO
+
+def process_image_from_url(image_url, text_prompt):
+    """Processes an image from a URL using a Transformers pipeline."""
+    try:
+        # Fetch the image from the URL
+        response = requests.get(image_url, stream=True)
+        response.raise_for_status()  # Raise an exception for bad status codes (4xx or 5xx)
+
+        # Open the image using PIL
+        image = Image.open(BytesIO(response.content))
+
+        # Create the input for the pipeline
+        inputs = {"image": image, "text": text_prompt}
+
+        # Initialize the pipeline
+        pipe = pipeline("image-text-to-text", model="Sapnous-AI/Sapnous-VR-12B", trust_remote_code=True)
+
+        # Process the image and text
+        result = pipe(inputs)
+        return result
+
+    except requests.exceptions.RequestException as e:
+        print(f"Error fetching image: {e}")
+        return None
+    except Exception as e:
+        print(f"An error occurred: {e}")
+        return None
+
+# Example usage
+image_url = "example.com" #replace with your image url.
+text_prompt = "What is in this image?"
+
+result = process_image_from_url(image_url, text_prompt)
+
+if result:
+    print(result)
+
+```
+
+## Model Capabilities
+
+- Multi-modal understanding and generation
+- Enhanced visual perception with advanced vision encoder
+- Efficient processing of long sequences
+- Robust performance across various vision-language tasks
+
+## Citations
+
+```bibtex
+@misc{sapnous-12B,
+    title = {Sapnous-12B},
+    author = {Sapnous AI Team},
+    year = {2025}
+}
+
+@article{Sapnous12B,
+    title={Sapnous-12B: Enhancing Vision-Language Model's Perception of the World at Any Resolution},
+    author={Sapnous AI Team},
+    year={2025}
+}
+
+@article{Sapnous-VR,
+    title={Sapnous-VR: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond},
+    author={Sapnous AI Team},
+    year={2025}
+}
+```
+
+## License
+
+Please refer to the LICENSE file for terms of use and distribution.

__init__.py

@@ -0,0 +1,39 @@
+# coding=utf-8
+# Copyright 2025-present, the HuggingFace Inc. Team and AIRAS Inc. Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from typing import TYPE_CHECKING
+from transformers.utils import _LazyModule
+from transformers.models.auto import CONFIG_MAPPING, MODEL_MAPPING, MODEL_FOR_CAUSAL_LM_MAPPING
+from transformers.models.auto import AutoConfig, AutoModel, AutoModelForCausalLM
+
+_import_structure = {
+    "configuration_sapnous": ["SAPNOUS_PRETRAINED_CONFIG_ARCHIVE_MAP", "SapnousT1Config"],
+    "modeling_sapnous": ["SapnousT1Model", "SapnousT1ForCausalLM"],
+}
+
+if TYPE_CHECKING:
+    from .configuration_sapnous import SAPNOUS_PRETRAINED_CONFIG_ARCHIVE_MAP, SapnousT1Config
+    from .modeling_sapnous import SapnousT1Model, SapnousT1ForCausalLM
+else:
+    import sys
+    sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure)
+
+# Register model in auto classes
+CONFIG_MAPPING["sapnous_t1"] = SapnousT1Config
+MODEL_MAPPING["sapnous_t1"] = SapnousT1Model
+MODEL_FOR_CAUSAL_LM_MAPPING["sapnous_t1"] = SapnousT1ForCausalLM
+
+AutoConfig.register("sapnous_t1", SapnousT1Config)
+AutoModel.register(SapnousT1Config, SapnousT1Model)
+AutoModelForCausalLM.register(SapnousT1Config, SapnousT1ForCausalLM)

attention_sapnous.py

@@ -0,0 +1,234 @@
+# coding=utf-8
+# Copyright 2025-present, the HuggingFace Inc. Team and AIRAS Inc. Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from typing import Optional, Tuple
+
+def precompute_freqs_cis(dim: int, end: int, theta: float = 10000.0) -> torch.Tensor:
+    """Precompute the frequency tensor for complex rotation."""
+    freqs = 1.0 / (theta ** (torch.arange(0, dim, 2)[: (dim // 2)].float() / dim))
+    t = torch.arange(end, device=freqs.device)
+    freqs = torch.outer(t, freqs)
+    return torch.polar(torch.ones_like(freqs), freqs)
+
+def apply_rotary_emb(x: torch.Tensor, freqs_cis: torch.Tensor) -> torch.Tensor:
+    """Apply rotary position embeddings to the input tensor."""
+    x_complex = torch.view_as_complex(x.float().reshape(*x.shape[:-1], -1, 2))
+    freqs_cis = freqs_cis.view(1, *freqs_cis.shape)
+    x_rotated = x_complex * freqs_cis
+    return torch.view_as_real(x_rotated).flatten(-2)
+
+class SapnousAttention(nn.Module):
+    """Multi-head attention with rotary position embeddings and sliding window attention."""
+    def __init__(self, config):
+        super().__init__()
+        self.config = config
+        self.hidden_size = config.hidden_size
+        self.num_attention_heads = config.num_attention_heads
+        self.head_dim = self.hidden_size // self.num_attention_heads
+        self.num_key_value_heads = config.num_key_value_heads
+        self.num_key_value_groups = self.num_attention_heads // self.num_key_value_heads
+        self.max_position_embeddings = config.max_position_embeddings
+        self.rope_theta = config.rope_theta
+        self.sliding_window = config.sliding_window if config.use_sliding_window else None
+
+        if (self.head_dim * self.num_attention_heads) != self.hidden_size:
+            raise ValueError(
+                f"hidden_size must be divisible by num_attention_heads (got {self.hidden_size} and {self.num_attention_heads})"
+            )
+
+        self.q_proj = nn.Linear(self.hidden_size, self.num_attention_heads * self.head_dim, bias=False)
+        self.k_proj = nn.Linear(self.hidden_size, self.num_key_value_heads * self.head_dim, bias=False)
+        self.v_proj = nn.Linear(self.hidden_size, self.num_key_value_heads * self.head_dim, bias=False)
+        self.o_proj = nn.Linear(self.num_attention_heads * self.head_dim, self.hidden_size, bias=False)
+
+        self.attention_dropout = nn.Dropout(config.attention_dropout)
+
+    def _shape(self, tensor: torch.Tensor, seq_len: int, bsz: int) -> torch.Tensor:
+        return tensor.view(bsz, seq_len, self.num_attention_heads, self.head_dim).transpose(1, 2)
+
+    def _kv_shape(self, tensor: torch.Tensor, seq_len: int, bsz: int) -> torch.Tensor:
+        return tensor.view(bsz, seq_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        freqs_cis: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_value: Optional[Tuple[torch.Tensor]] = None,
+        output_attentions: bool = False,
+        use_cache: bool = False,
+    ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
+        bsz, q_len, _ = hidden_states.size()
+
+        query_states = self.q_proj(hidden_states)
+        key_states = self.k_proj(hidden_states)
+        value_states = self.v_proj(hidden_states)
+
+        query_states = self._shape(query_states, q_len, bsz)
+        key_states = self._kv_shape(key_states, q_len, bsz)
+        value_states = self._kv_shape(value_states, q_len, bsz)
+
+        kv_seq_len = key_states.shape[-2]
+        if past_key_value is not None:
+            kv_seq_len += past_key_value[0].shape[-2]
+
+        # Apply rotary position embeddings
+        if position_ids is None:
+            position_ids = torch.arange(kv_seq_len, device=hidden_states.device)
+        cos, sin = freqs_cis[position_ids]
+        query_states, key_states = apply_rotary_emb(query_states, cos), apply_rotary_emb(key_states, sin)
+
+        if past_key_value is not None:
+            # Reuse k, v, self_attention
+            key_states = torch.cat([past_key_value[0], key_states], dim=2)
+            value_states = torch.cat([past_key_value[1], value_states], dim=2)
+
+        past_key_value = (key_states, value_states) if use_cache else None
+
+        # Repeat k/v heads if n_kv_heads < n_heads
+        key_states = torch.repeat_interleave(key_states, self.num_key_value_groups, dim=1)
+        value_states = torch.repeat_interleave(value_states, self.num_key_value_groups, dim=1)
+
+        attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim)
+
+        if attention_mask is not None:
+            attn_weights = attn_weights + attention_mask
+
+        # Sliding window attention if configured
+        if self.sliding_window is not None and kv_seq_len > self.sliding_window:
+            # Create sliding window mask
+            window_mask = torch.ones_like(attn_weights, dtype=torch.bool)
+            for i in range(q_len):
+                window_start = max(0, i - self.sliding_window // 2)
+                window_end = min(kv_seq_len, i + self.sliding_window // 2)
+                window_mask[:, :, i, window_start:window_end] = False
+            attn_weights = attn_weights.masked_fill(window_mask, float('-inf'))
+
+        # Causal mask for autoregressive generation
+        if self.config.scoring_func == "softmax":
+            causal_mask = torch.triu(torch.ones((q_len, kv_seq_len), dtype=torch.bool), diagonal=1)
+            causal_mask = causal_mask.unsqueeze(0).unsqueeze(0)
+            attn_weights = attn_weights.masked_fill(causal_mask.to(attn_weights.device), float('-inf'))
+            attn_weights = F.softmax(attn_weights, dim=-1)
+        else:
+            # Alternative scoring functions (e.g., RoPE-only, cosine similarity)
+            attn_weights = F.relu(attn_weights)
+            attn_weights = attn_weights / (attn_weights.sum(dim=-1, keepdim=True) + 1e-6)
+
+        attn_weights = self.attention_dropout(attn_weights)
+        attn_output = torch.matmul(attn_weights, value_states)
+
+        attn_output = attn_output.transpose(1, 2).contiguous()
+        attn_output = attn_output.reshape(bsz, q_len, self.hidden_size)
+
+        attn_output = self.o_proj(attn_output)
+
+        if not output_attentions:
+            attn_weights = None
+
+        return attn_output, attn_weights, past_key_value
+
+class SapnousBlock(nn.Module):
+    """Transformer block with attention, layer norm, and feed-forward network."""
+    def __init__(self, config):
+        super().__init__()
+        self.hidden_size = config.hidden_size
+        self.self_attn = SapnousAttention(config)
+        self.input_layernorm = nn.LayerNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.post_attention_layernorm = nn.LayerNorm(config.hidden_size, eps=config.rms_norm_eps)
+        
+        self.mlp = nn.Sequential(
+            nn.Linear(config.hidden_size, config.intermediate_size, bias=False),
+            nn.SiLU(),
+            nn.Linear(config.intermediate_size, config.hidden_size, bias=False),
+        )
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        freqs_cis: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_value: Optional[Tuple[torch.Tensor]] = None,
+        output_attentions: bool = False,
+        use_cache: bool = False,
+    ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
+        # Self Attention
+        residual = hidden_states
+        hidden_states = self.input_layernorm(hidden_states)
+        
+        hidden_states, self_attn_weights, present_key_value = self.self_attn(
+            hidden_states=hidden_states,
+            freqs_cis=freqs_cis,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_value=past_key_value,
+            output_attentions=output_attentions,
+            use_cache=use_cache,
+        )
+        hidden_states = residual + hidden_states
+
+        # Fully Connected
+        residual = hidden_states
+        hidden_states = self.post_attention_layernorm(hidden_states)
+        hidden_states = self.mlp(hidden_states)
+        hidden_states = residual + hidden_states
+
+        outputs = (hidden_states,)
+
+        if output_attentions:
+            outputs += (self_attn_weights,)
+
+        if use_cache:
+            outputs += (present_key_value,)
+
+        return outputs
+
+class SapnousVisionEmbeddings(nn.Module):
+    """Vision embeddings for multimodal support."""
+    def __init__(self, config):
+        super().__init__()
+        self.config = config
+        self.hidden_size = config.hidden_size
+        
+        # Vision embedding layers
+        self.patch_embed = nn.Conv2d(3, self.hidden_size, kernel_size=16, stride=16)
+        self.cls_token = nn.Parameter(torch.zeros(1, 1, self.hidden_size))
+        self.pos_embed = nn.Parameter(torch.zeros(1, (224 // 16) ** 2 + 1, self.hidden_size))
+        
+        # Layer normalization and dropout
+        self.norm = nn.LayerNorm(self.hidden_size, eps=config.rms_norm_eps)
+        self.dropout = nn.Dropout(config.attention_dropout)
+
+    def forward(self, pixel_values: torch.Tensor) -> torch.Tensor:
+        B = pixel_values.shape[0]
+        
+        # Create patch embeddings
+        x = self.patch_embed(pixel_values)
+        x = x.flatten(2).transpose(1, 2)  # B, N, C
+        
+        # Add cls token and position embeddings
+        cls_tokens = self.cls_token.expand(B, -1, -1)
+        x = torch.cat((cls_tokens, x), dim=1)
+        x = x + self.pos_embed
+        
+        # Apply normalization and dropout
+        x = self.norm(x)
+        x = self.dropout(x)
+        
+        return x

chat_template.json

@@ -0,0 +1,3 @@
+{
+  "chat_template": "{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system\nYou are a Sapnous by AIRAS.<|im_end|>\n{% endif %}<|im_start|>{{ message['role'] }}\n{% if message['content'] is string %}{{ message['content'] }}<|im_end|>\n{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}"
+}

config.json

@@ -0,0 +1,66 @@
+{
+  "architectures": [
+    "SapnousT1ForCausalLM"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_sapnous.SapnousT1Config",
+    "AutoModel": "modeling_sapnous.SapnousT1Model",
+    "AutoModelForCausalLM": "modeling_sapnous.SapnousT1ForCausalLM"
+  },
+  "attention_dropout": 0.0,
+  "bos_token_id": 151643,
+  "eos_token_id": 151645,
+  "vision_start_token_id": 151652,
+  "vision_end_token_id": 151653,
+  "vision_token_id": 151654,
+  "image_token_id": 151655,
+  "video_token_id": 151656,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 20480,
+  "max_position_embeddings": 128000,
+  "max_window_layers": 70,
+  "model_type": "sapnous_t1",
+  "num_attention_heads": 40,
+  "num_hidden_layers": 36,
+  "num_key_value_heads": 8,
+  "rms_norm_eps": 1e-06,
+  "rope_theta": 1000000.0,
+  "sliding_window": 32768,
+  "tie_word_embeddings": true,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.41.2",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vision_config": {
+    "depth": 32,
+    "hidden_act": "silu",
+    "hidden_size": 1280,
+    "intermediate_size": 3420,
+    "num_heads": 16,
+    "in_chans": 3,
+    "out_hidden_size": 2048,
+    "patch_size": 14,
+    "spatial_merge_size": 2,
+    "spatial_patch_size": 14,
+    "window_size": 112,
+    "fullatt_block_indexes": [
+      7,
+      15,
+      23,
+      31
+    ],
+    "tokens_per_second": 2,
+    "temporal_patch_size": 2
+  },
+  "rope_scaling": {
+    "type": "mrope",
+    "mrope_section": [
+      16,
+      24,
+      24
+    ]
+  },
+  "vocab_size": 151936
+}

configuration_sapnous.py

@@ -0,0 +1,101 @@
+# coding=utf-8
+# Copyright 2025-present, the HuggingFace Inc. Team and AIRAS Inc. Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from transformers.configuration_utils import PretrainedConfig
+from transformers.utils import logging
+from transformers import AutoConfig  
+
+logger = logging.get_logger(__name__)
+
+SAPNOUS_PRETRAINED_CONFIG_ARCHIVE_MAP = {
+    "Sapnous-AI/Sapnous-VR-6B": "https://huggingface.co/Sapnous-AI/Sapnous-VR-6B/resolve/main/config.json",
+}
+
+class SapnousT1Config(PretrainedConfig):
+    model_type = "sapnous_t1"
+
+    def __init__(
+        self,
+        vocab_size=151936,
+        hidden_size=5120,
+        intermediate_size=20480,
+        num_hidden_layers=36,
+        num_attention_heads=40,
+        num_key_value_heads=8,
+        hidden_act="silu",
+        max_position_embeddings=128000,
+        initializer_range=0.02,
+        rms_norm_eps=1e-6,
+        use_cache=True,
+        pad_token_id=None,
+        bos_token_id=151643,
+        eos_token_id=151645,
+        tie_word_embeddings=True,
+        vision_start_token_id=151652,
+        vision_end_token_id=151653,
+        vision_token_id=151654,
+        image_token_id=151655,
+        video_token_id=151656,
+        vision_config=None,
+        rope_theta=1000000.0,
+        sliding_window=32768,
+        use_sliding_window=False,
+        max_window_layers=70,
+        attention_dropout=0.0,
+        rope_scaling=None,
+        scoring_func="softmax",
+        aux_loss_alpha=0.001,
+        seq_aux=True,
+        **kwargs
+    ):
+        super().__init__(
+            pad_token_id=pad_token_id,
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            tie_word_embeddings=tie_word_embeddings,
+            **kwargs,
+        )
+
+        self.vocab_size = vocab_size
+        self.max_position_embeddings = max_position_embeddings
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.num_key_value_heads = num_key_value_heads
+        self.hidden_act = hidden_act
+        self.initializer_range = initializer_range
+        self.rms_norm_eps = rms_norm_eps
+        self.use_cache = use_cache
+        self.vision_start_token_id = vision_start_token_id
+        self.vision_end_token_id = vision_end_token_id
+        self.vision_token_id = vision_token_id
+        self.image_token_id = image_token_id
+        self.video_token_id = video_token_id
+        self.vision_config = vision_config
+        self.rope_theta = rope_theta
+        self.sliding_window = sliding_window
+        self.use_sliding_window = use_sliding_window
+        self.max_window_layers = max_window_layers
+        self.attention_dropout = attention_dropout
+        self.rope_scaling = rope_scaling
+        self.scoring_func = scoring_func
+        self.aux_loss_alpha = aux_loss_alpha
+        self.seq_aux = seq_aux
+
+    model_type = "sapnous_t1"
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+# ✅ Register after defining the class
+AutoConfig.register("sapnous_t1", SapnousT1Config)

convert_to_gguf.py

@@ -0,0 +1,186 @@
+# coding=utf-8
+# Copyright 2025-present, the HuggingFace Inc. Team and AIRAS Inc. Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import os
+import torch
+import json
+from pathlib import Path
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from ctransformers import AutoModelForCausalLM as GGUFModel
+from models.sapnous import SapnousT1Config
+
+def load_safetensors_state_dict(model_path, weight_map):
+    """Load state dict from safetensors shards with custom metadata handling."""
+    import safetensors
+    from safetensors.torch import load_file
+
+    state_dict = {}
+    metadata = {}
+    
+    # Load and validate each shard
+    for param_name, shard_file in weight_map['weight_map'].items():
+        shard_path = os.path.join(model_path, shard_file)
+        if not os.path.exists(shard_path):
+            raise OSError(f"Missing weight shard: {shard_path}")
+            
+        try:
+            # Load shard with metadata
+            shard_dict = load_file(shard_path)
+            shard_metadata = safetensors.safe_open(shard_path, framework="pt").metadata()
+            
+            if shard_metadata:
+                metadata.update(shard_metadata)
+            
+            # Add tensors to state dict
+            for key, tensor in shard_dict.items():
+                if key in state_dict:
+                    raise ValueError(f"Duplicate parameter {key} found in multiple shards")
+                state_dict[key] = tensor
+                
+        except Exception as e:
+            raise OSError(f"Error loading shard {shard_file}: {str(e)}")
+    
+    # Add metadata to state dict
+    if metadata:
+        state_dict['_metadata'] = metadata
+    
+    return state_dict
+    return state_dict
+
+def convert_to_gguf(model_path, output_path):
+    # Load configuration and weight map
+    config_path = os.path.join(model_path, 'config.json')
+    weight_map_path = os.path.join(model_path, 'model.safetensors.index.json')
+    
+    if not os.path.exists(config_path):
+        raise OSError(f"Missing config file: {config_path}")
+    if not os.path.exists(weight_map_path):
+        raise OSError(f"Missing weight map file: {weight_map_path}")
+        
+    with open(config_path, 'r') as f:
+        config = json.load(f)
+    with open(weight_map_path, 'r') as f:
+        weight_map = json.load(f)
+        
+    # Validate weight map structure
+    if 'weight_map' not in weight_map:
+        raise ValueError("Invalid weight map format: missing 'weight_map' key")
+    if 'metadata' not in weight_map:
+        raise ValueError("Invalid weight map format: missing 'metadata' key")
+    
+    # Load the model and tokenizer with vision-language support
+    model = AutoModelForCausalLM.from_pretrained(
+        model_path,
+        trust_remote_code=True,
+        device_map=None,  # Disable device mapping for conversion
+        torch_dtype=torch.float16,  # Use FP16 for memory efficiency
+        low_cpu_mem_usage=True,  # Enable low CPU memory usage
+        local_files_only=True,  # Use local files only
+        ignore_mismatched_sizes=True,  # Bypass size validation
+        use_safetensors=True,  # Explicitly enable safetensors
+        use_auth_token=False  # Disable auth token
+    )
+    tokenizer = AutoTokenizer.from_pretrained(
+        model_path,
+        trust_remote_code=True
+    )
+    
+    # Get model configuration
+    config = model.config
+    if not isinstance(config, SapnousT1Config):
+        raise ValueError("Model must be a SapnousT1 model")
+    
+    # Save in intermediate format
+    model.save_pretrained(output_path, safe_serialization=True)
+    tokenizer.save_pretrained(output_path)
+    
+    # Convert to GGUF using custom SapnousT1 architecture settings
+    gguf_model = GGUFModel.from_pretrained(
+        output_path,
+        model_type='sapnous_t1',  # Custom architecture type
+        gpu_layers=0,  # CPU only for conversion
+        config={
+            'context_length': config.sliding_window,
+            'attention_type': 'multihead',  # Custom attention implementation
+            'num_attention_heads': config.num_attention_heads,
+            'num_key_value_heads': config.num_key_value_heads,
+            'hidden_size': config.hidden_size,
+            'intermediate_size': config.intermediate_size,
+            'max_position_embeddings': config.max_position_embeddings,
+            'vocab_size': config.vocab_size,
+            'num_hidden_layers': config.num_hidden_layers,
+            'rms_norm_eps': config.rms_norm_eps,
+            'rope_theta': config.rope_theta,
+            # Vision model parameters
+            'vision_config': {
+                'hidden_size': config.vision_hidden_size,
+                'num_hidden_layers': config.vision_layers,
+                'num_attention_heads': config.vision_heads,
+                'intermediate_size': config.vision_intermediate_size,
+                'patch_size': config.patch_size,
+                'image_size': config.image_size
+            }
+        }
+    )
+    
+    print(f"Model converted and saved to {output_path}")
+    return gguf_model
+
+def convert_to_hf(gguf_path, output_path):
+    """Convert GGUF model back to Hugging Face format"""
+    # Load GGUF model configuration
+    config_path = Path(gguf_path) / "config.json"
+    with open(config_path, 'r') as f:
+        gguf_config = json.load(f)
+    
+    # Create SapnousT1 configuration
+    config = SapnousT1Config(
+        vocab_size=gguf_config['vocab_size'],
+        hidden_size=gguf_config['hidden_size'],
+        num_hidden_layers=gguf_config['num_hidden_layers'],
+        num_attention_heads=gguf_config['num_attention_heads'],
+        num_key_value_heads=gguf_config['num_key_value_heads'],
+        intermediate_size=gguf_config['intermediate_size'],
+        max_position_embeddings=gguf_config['max_position_embeddings'],
+        rms_norm_eps=gguf_config['rms_norm_eps'],
+        rope_theta=gguf_config['rope_theta'],
+        # Vision configuration
+        vision_hidden_size=gguf_config['vision_config']['hidden_size'],
+        vision_layers=gguf_config['vision_config']['num_hidden_layers'],
+        vision_heads=gguf_config['vision_config']['num_attention_heads'],
+        vision_intermediate_size=gguf_config['vision_config']['intermediate_size'],
+        patch_size=gguf_config['vision_config']['patch_size'],
+        image_size=gguf_config['vision_config']['image_size']
+    )
+    
+    # Load GGUF model
+    gguf_model = GGUFModel.from_pretrained(gguf_path)
+    
+    # Convert weights to HF format
+    model = AutoModelForCausalLM.from_config(config)
+    model.load_state_dict(gguf_model.state_dict())
+    
+    # Save converted model
+    model.save_pretrained(output_path)
+    print(f"Model converted back to Hugging Face format at {output_path}")
+    return model
+
+if __name__ == '__main__':
+    model_path = os.path.dirname(os.path.abspath(__file__))
+    output_path = os.path.join(model_path, 'gguf_model')
+    
+    if not os.path.exists(output_path):
+        os.makedirs(output_path)
+        
+    convert_to_gguf(model_path, output_path)

generation_config.json

@@ -0,0 +1,12 @@
+{
+  "bos_token_id": 151643,
+  "pad_token_id": 151643,
+  "do_sample": true,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "repetition_penalty": 1.05,
+  "temperature": 0.00001,
+  "transformers_version": "4.49.0"
+}

model.py

@@ -0,0 +1,40 @@
+# coding=utf-8
+# Copyright 2025-present, the HuggingFace Inc. Team and AIRAS Inc. Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from transformers import PreTrainedModel, AutoConfig
+import torch
+import torch.nn as nn
+
+class SapnousT1ForCausalLM(PreTrainedModel):
+    config_class = AutoConfig
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.hidden_size = config.hidden_size
+        self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size)
+        self.layers = nn.ModuleList([
+            nn.Linear(config.hidden_size, config.hidden_size) for _ in range(config.num_hidden_layers)
+        ])
+        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+
+    def forward(self, input_ids):
+        hidden_states = self.embed_tokens(input_ids)
+        for layer in self.layers:
+            hidden_states = layer(hidden_states)
+        logits = self.lm_head(hidden_states)
+        return logits
+
+# Register model with transformers
+from transformers import AutoModelForCausalLM
+AutoModelForCausalLM.register(SapnousT1ForCausalLM, "sapnous_t1")

modeling_sapnous.py

@@ -0,0 +1,271 @@
+import math
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from typing import Optional, Tuple, List, Union
+from transformers import PreTrainedModel, AutoModelForCausalLM
+from transformers.modeling_outputs import CausalLMOutputWithPast, BaseModelOutputWithPast
+from .configuration_sapnous import SapnousT1Config
+from .attention_sapnous import SapnousAttention, SapnousBlock, SapnousVisionEmbeddings, precompute_freqs_cis
+
+class SapnousT1PreTrainedModel(PreTrainedModel):
+    """Base class for all Sapnous-T1 models."""
+    config_class = SapnousT1Config
+    base_model_prefix = "sapnous"
+
+    def __init__(self, config: SapnousT1Config):
+        super().__init__(config)
+        self.config = config
+
+    def _init_weights(self, module):
+        """Initialize weights using the model's initialization configuration."""
+        std = self.config.initializer_range
+        if isinstance(module, nn.Linear):
+            module.weight.data.normal_(mean=0.0, std=std)
+            if module.bias is not None:
+                module.bias.data.zero_()
+        elif isinstance(module, nn.Embedding):
+            module.weight.data.normal_(mean=0.0, std=std)
+        elif isinstance(module, nn.LayerNorm):
+            module.bias.data.zero_()
+            module.weight.data.fill_(1.0)
+        elif isinstance(module, SapnousAttention):
+            module.q_proj.weight.data.normal_(mean=0.0, std=std)
+            module.k_proj.weight.data.normal_(mean=0.0, std=std)
+            module.v_proj.weight.data.normal_(mean=0.0, std=std)
+            module.o_proj.weight.data.normal_(mean=0.0, std=std)
+
+class SapnousT1Model(SapnousT1PreTrainedModel):
+    """Base Transformer Model with advanced attention mechanisms and optional vision support."""
+    def __init__(self, config: SapnousT1Config):
+        super().__init__(config)
+        
+        self.embeddings = nn.Embedding(config.vocab_size, config.hidden_size)
+        self.layers = nn.ModuleList([SapnousBlock(config) for _ in range(config.num_hidden_layers)])
+        self.norm = nn.LayerNorm(config.hidden_size, eps=config.rms_norm_eps)
+        
+        # Vision support
+        self.vision_embed = SapnousVisionEmbeddings(config) if getattr(config, 'vision_config', None) else None
+        
+        # Initialize weights and apply final processing
+        self.post_init()
+        
+        # Compute and cache RoPE frequencies
+        self.freqs_cis = precompute_freqs_cis(
+            self.config.hidden_size // self.config.num_attention_heads,
+            self.config.max_position_embeddings,
+            self.config.rope_theta,
+        )
+
+    def get_input_embeddings(self) -> nn.Module:
+        return self.embeddings
+
+    def set_input_embeddings(self, value: nn.Module):
+        self.embeddings = value
+
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[List[Tuple[torch.FloatTensor]]] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        pixel_values: Optional[torch.FloatTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+    ) -> Union[Tuple, BaseModelOutputWithPast]:
+        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        use_cache = use_cache if use_cache is not None else self.config.use_cache
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        if input_ids is not None and inputs_embeds is not None:
+            raise ValueError("You cannot specify both input_ids and inputs_embeds")
+
+        # Process text input
+        if input_ids is not None:
+            inputs_embeds = self.embeddings(input_ids)
+            batch_size, seq_length = input_ids.shape[:2]
+        else:
+            batch_size, seq_length = inputs_embeds.shape[:2]
+
+        # Process vision input if available
+        if pixel_values is not None and self.vision_embed is not None:
+            vision_embeds = self.vision_embed(pixel_values)
+            inputs_embeds = torch.cat([vision_embeds, inputs_embeds], dim=1)
+            seq_length = inputs_embeds.shape[1]
+
+        if position_ids is None:
+            device = input_ids.device if input_ids is not None else inputs_embeds.device
+            position_ids = torch.arange(seq_length, dtype=torch.long, device=device)
+            position_ids = position_ids.unsqueeze(0)
+
+        # Prepare attention mask
+        if attention_mask is not None:
+            attention_mask = attention_mask.view(batch_size, -1)
+            attention_mask = attention_mask[:, None, None, :]
+            attention_mask = attention_mask.to(dtype=inputs_embeds.dtype)
+            attention_mask = (1.0 - attention_mask) * torch.finfo(inputs_embeds.dtype).min
+
+        freqs_cis = self.freqs_cis.to(inputs_embeds.device)
+        
+        hidden_states = inputs_embeds
+        all_hidden_states = () if output_hidden_states else None
+        all_self_attns = () if output_attentions else None
+        next_decoder_cache = () if use_cache else None
+
+        for idx, decoder_layer in enumerate(self.layers):
+            if output_hidden_states:
+                all_hidden_states += (hidden_states,)
+
+            past_key_value = past_key_values[idx] if past_key_values is not None else None
+
+            layer_outputs = decoder_layer(
+                hidden_states,
+                freqs_cis=freqs_cis,
+                attention_mask=attention_mask,
+                position_ids=position_ids,
+                past_key_value=past_key_value,
+                output_attentions=output_attentions,
+                use_cache=use_cache,
+            )
+
+            hidden_states = layer_outputs[0]
+
+            if use_cache:
+                next_decoder_cache += (layer_outputs[2 if output_attentions else 1],)
+
+            if output_attentions:
+                all_self_attns += (layer_outputs[1],)
+
+        hidden_states = self.norm(hidden_states)
+
+        if output_hidden_states:
+            all_hidden_states += (hidden_states,)
+
+        if not return_dict:
+            return tuple(v for v in [
+                hidden_states,
+                next_decoder_cache,
+                all_hidden_states,
+                all_self_attns,
+            ] if v is not None)
+
+        return BaseModelOutputWithPast(
+            last_hidden_state=hidden_states,
+            past_key_values=next_decoder_cache,
+            hidden_states=all_hidden_states,
+            attentions=all_self_attns,
+        )
+
+class SapnousT1ForCausalLM(SapnousT1PreTrainedModel):
+    """Sapnous-T1 Model for Causal Language Modeling with vision support."""
+    _keys_to_ignore_on_load_missing = [r"lm_head.weight"]
+
+    def __init__(self, config: SapnousT1Config):
+        super().__init__(config)
+        self.model = SapnousT1Model(config)
+        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_input_embeddings(self) -> nn.Module:
+        return self.model.embeddings
+
+    def set_input_embeddings(self, value: nn.Module):
+        self.model.embeddings = value
+
+    def get_output_embeddings(self) -> nn.Module:
+        return self.lm_head
+
+    def set_output_embeddings(self, new_embeddings: nn.Module):
+        self.lm_head = new_embeddings
+
+    def prepare_inputs_for_generation(
+        self,
+        input_ids: torch.LongTensor,
+        past_key_values: Optional[List[Tuple[torch.Tensor]]] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        **kwargs,
+    ) -> dict:
+        if past_key_values:
+            input_ids = input_ids[:, -1:]
+
+        position_ids = kwargs.get("position_ids", None)
+        if position_ids is None:
+            position_ids = (attention_mask.long().cumsum(-1) - 1) if attention_mask is not None else None
+            if past_key_values:
+                position_ids = position_ids[:, -1].unsqueeze(-1)
+
+        return {
+            "input_ids": input_ids,
+            "attention_mask": attention_mask,
+            "position_ids": position_ids,
+            "past_key_values": past_key_values,
+            "use_cache": kwargs.get("use_cache"),
+            "pixel_values": kwargs.get("pixel_values", None),
+        }
+
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[List[Tuple[torch.FloatTensor]]] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        pixel_values: Optional[torch.FloatTensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+    ) -> Union[Tuple, CausalLMOutputWithPast]:
+        r"""Labels for computing the masked language modeling loss."""
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        outputs = self.model(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            pixel_values=pixel_values,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+
+        hidden_states = outputs[0]
+        logits = self.lm_head(hidden_states)
+
+        loss = None
+        if labels is not None:
+            shift_logits = logits[..., :-1, :].contiguous()
+            shift_labels = labels[..., 1:].contiguous()
+            loss_fct = nn.CrossEntropyLoss()
+            loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
+
+        if not return_dict:
+            output = (logits,) + outputs[1:]
+            return ((loss,) + output) if loss is not None else output
+
+        return CausalLMOutputWithPast(
+            loss=loss,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+    def tie_weights(self):
+        """Tie the weights between the input embeddings and the output embeddings."""
+        self.lm_head.weight = self.model.embeddings.weight
+
+# Register the model
+AutoModelForCausalLM.register(SapnousT1Config, SapnousT1ForCausalLM)

preprocessor_config.json

@@ -0,0 +1,19 @@
+{
+  "min_pixels": 3136,
+  "max_pixels": 12845056,
+  "patch_size": 14,
+  "temporal_patch_size": 2,
+  "merge_size": 2,
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "image_processor_type": "Sapnous12BImageProcessor",
+  "processor_class": "Sapnous12BProcessor"
+}

setup.py

@@ -0,0 +1,15 @@
+import sys
+import os
+
+# Add the current directory to sys.path so Python can find `configuration_sapnous.py`
+sys.path.append(os.path.dirname(os.path.abspath(__file__)))
+
+from transformers import AutoModel, AutoConfig
+from configuration_sapnous import SapnousT1Config  # Now it should work
+
+model_path = r"E:\git\Sapnous-47B\Sapnous-6B"
+
+config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
+model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True)
+
+print("Model loaded successfully!")

test_modeling_sapnous.py

@@ -0,0 +1,92 @@
+# coding=utf-8
+# Copyright 2025-present, the HuggingFace Inc. Team and AIRAS Inc. Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import unittest
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from .modeling_sapnous import SapnousT1ForCausalLM
+from .configuration_sapnous import SapnousT1Config
+
+class TestSapnousModel(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        cls.config = SapnousT1Config(
+            vocab_size=32000,
+            hidden_size=768,
+            num_hidden_layers=12,
+            num_attention_heads=12,
+            intermediate_size=3072
+        )
+        cls.model = SapnousT1ForCausalLM(cls.config)
+
+    def test_model_forward(self):
+        input_ids = torch.randint(0, self.config.vocab_size, (1, 10))
+        outputs = self.model(input_ids)
+        
+        self.assertIsNotNone(outputs)
+        self.assertTrue(hasattr(outputs, 'logits'))
+        self.assertEqual(outputs.logits.shape, (1, 10, self.config.vocab_size))
+
+    def test_weight_tying(self):
+        self.model.tie_weights()
+        self.assertTrue(torch.equal(self.model.lm_head.weight, self.model.model.embeddings.weight))
+
+    def test_auto_model_registration(self):
+        model = AutoModelForCausalLM.from_config(self.config)
+        self.assertIsInstance(model, SapnousT1ForCausalLM)
+
+    def test_vision_embeddings(self):
+        # Test vision input processing
+        batch_size = 1
+        pixel_values = torch.randn(batch_size, 3, 224, 224)
+        input_ids = torch.randint(0, self.config.vocab_size, (batch_size, 10))
+        
+        outputs = self.model(input_ids=input_ids, pixel_values=pixel_values)
+        self.assertIsNotNone(outputs)
+        self.assertTrue(hasattr(outputs, 'logits'))
+        
+        # Vision input should increase sequence length
+        expected_seq_length = 10 + (224 // 16) ** 2 + 1  # text_len + num_patches + cls_token
+        self.assertEqual(outputs.logits.shape, (batch_size, expected_seq_length, self.config.vocab_size))
+    
+    def test_attention_mask(self):
+        # Test attention mask handling
+        batch_size = 2
+        seq_length = 15
+        input_ids = torch.randint(0, self.config.vocab_size, (batch_size, seq_length))
+        attention_mask = torch.ones(batch_size, seq_length)
+        attention_mask[:, -5:] = 0  # Mask out last 5 tokens
+        
+        outputs = self.model(input_ids=input_ids, attention_mask=attention_mask)
+        self.assertIsNotNone(outputs)
+        self.assertEqual(outputs.logits.shape, (batch_size, seq_length, self.config.vocab_size))
+    
+    def test_generation_with_vision(self):
+        # Test text generation with vision input
+        pixel_values = torch.randn(1, 3, 224, 224)
+        input_ids = torch.randint(0, self.config.vocab_size, (1, 5))
+        
+        outputs = self.model.generate(
+            input_ids=input_ids,
+            pixel_values=pixel_values,
+            max_length=20,
+            num_beams=1
+        )
+        
+        self.assertIsInstance(outputs, torch.Tensor)
+        self.assertEqual(outputs.dim(), 2)
+        self.assertTrue(outputs.size(1) <= 20)
+
+if __name__ == '__main__':
+    unittest.main()

test_tokenization_sapnous.py

@@ -0,0 +1,157 @@
+# coding=utf-8
+# Copyright 2025-present, the HuggingFace Inc. Team and AIRAS Inc. Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import unittest
+import torch
+from pathlib import Path
+from transformers import AutoTokenizer
+from .tokenization_sapnous import SapnousTokenizer
+
+class TestSapnousTokenizer(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        # Create temporary vocab and merges files for testing
+        cls.temp_dir = Path('test_tokenizer_files')
+        cls.temp_dir.mkdir(exist_ok=True)
+        
+        # Create a simple test vocabulary
+        cls.vocab_file = cls.temp_dir / 'vocab.json'
+        cls.vocab = {
+            '<|endoftext|>': 0,
+            '<|startoftext|>': 1,
+            '<|pad|>': 2,
+            '<|vision_start|>': 3,
+            '<|vision_end|>': 4,
+            '<|image|>': 5,
+            '<|video|>': 6,
+            'hello': 7,
+            'world': 8,
+            'test': 9,
+        }
+        with cls.vocab_file.open('w', encoding='utf-8') as f:
+            import json
+            json.dump(cls.vocab, f)
+        
+        # Create test merges file
+        cls.merges_file = cls.temp_dir / 'merges.txt'
+        merges_content = "#version: 0.2\nh e\ne l\nl l\no w\nw o\no r\nr l\nl d"
+        cls.merges_file.write_text(merges_content)
+        
+        # Initialize tokenizer
+        cls.tokenizer = SapnousTokenizer(
+            str(cls.vocab_file),
+            str(cls.merges_file),
+        )
+    
+    @classmethod
+    def tearDownClass(cls):
+        # Clean up temporary files
+        import shutil
+        shutil.rmtree(cls.temp_dir)
+    
+    def test_tokenizer_initialization(self):
+        self.assertEqual(self.tokenizer.vocab_size, len(self.vocab))
+        self.assertEqual(self.tokenizer.get_vocab(), self.vocab)
+        
+        # Test special tokens
+        self.assertEqual(self.tokenizer.unk_token, '<|endoftext|>')
+        self.assertEqual(self.tokenizer.bos_token, '<|startoftext|>')
+        self.assertEqual(self.tokenizer.eos_token, '<|endoftext|>')
+        self.assertEqual(self.tokenizer.pad_token, '<|pad|>')
+    
+    def test_tokenization(self):
+        text = "hello world test"
+        tokens = self.tokenizer.tokenize(text)
+        self.assertIsInstance(tokens, list)
+        self.assertTrue(all(isinstance(token, str) for token in tokens))
+        
+        # Test encoding
+        input_ids = self.tokenizer.encode(text, add_special_tokens=False)
+        self.assertIsInstance(input_ids, list)
+        self.assertEqual(len(input_ids), 3)  # 'hello', 'world', 'test'
+        
+        # Test decoding
+        decoded_text = self.tokenizer.decode(input_ids)
+        self.assertEqual(decoded_text.strip(), text)
+    
+    def test_special_tokens_handling(self):
+        text = "hello world"
+        # Test with special tokens
+        tokens_with_special = self.tokenizer.encode(text, add_special_tokens=True)
+        self.assertTrue(tokens_with_special[0] == self.tokenizer.bos_token_id)
+        self.assertTrue(tokens_with_special[-1] == self.tokenizer.eos_token_id)
+        
+        # Test without special tokens
+        tokens_without_special = self.tokenizer.encode(text, add_special_tokens=False)
+        self.assertNotEqual(tokens_without_special[0], self.tokenizer.bos_token_id)
+        self.assertNotEqual(tokens_without_special[-1], self.tokenizer.eos_token_id)
+    
+    def test_vision_tokens(self):
+        # Test vision-specific token methods
+        text = "This is an image description"
+        vision_text = self.tokenizer.prepare_for_vision(text)
+        self.assertTrue(vision_text.startswith('<|vision_start|>'))
+        self.assertTrue(vision_text.endswith('<|vision_end|>'))
+        
+        image_text = self.tokenizer.prepare_for_image(text)
+        self.assertTrue(image_text.startswith('<|image|>'))
+        
+        video_text = self.tokenizer.prepare_for_video(text)
+        self.assertTrue(video_text.startswith('<|video|>'))
+    
+    def test_batch_encoding(self):
+        texts = ["hello world", "test hello"]
+        batch_encoding = self.tokenizer(texts, padding=True, truncation=True, return_tensors="pt")
+        
+        self.assertIsInstance(batch_encoding["input_ids"], torch.Tensor)
+        self.assertIsInstance(batch_encoding["attention_mask"], torch.Tensor)
+        self.assertEqual(batch_encoding["input_ids"].shape[0], len(texts))
+        self.assertEqual(batch_encoding["attention_mask"].shape[0], len(texts))
+    
+    def test_save_and_load(self):
+        # Test saving vocabulary
+        save_dir = Path('test_save_tokenizer')
+        save_dir.mkdir(exist_ok=True)
+        
+        try:
+            vocab_files = self.tokenizer.save_vocabulary(str(save_dir))
+            self.assertTrue(all(Path(f).exists() for f in vocab_files))
+            
+            # Test loading saved vocabulary
+            loaded_tokenizer = SapnousTokenizer(*vocab_files)
+            self.assertEqual(loaded_tokenizer.get_vocab(), self.tokenizer.get_vocab())
+            
+            # Test encoding/decoding with loaded tokenizer
+            text = "hello world test"
+            original_encoding = self.tokenizer.encode(text)
+            loaded_encoding = loaded_tokenizer.encode(text)
+            self.assertEqual(original_encoding, loaded_encoding)
+        finally:
+            # Clean up
+            import shutil
+            shutil.rmtree(save_dir)
+    
+    def test_auto_tokenizer_registration(self):
+        # Test if the tokenizer can be loaded using AutoTokenizer
+        config = {
+            "model_type": "sapnous",
+            "vocab_file": str(self.vocab_file),
+            "merges_file": str(self.merges_file)
+        }
+        
+        tokenizer = AutoTokenizer.from_pretrained(str(self.temp_dir), **config)
+        self.assertIsInstance(tokenizer, SapnousTokenizer)
+
+if __name__ == '__main__':
+    unittest.main()

tokenization_sapnous.py

@@ -0,0 +1,197 @@
+# coding=utf-8
+# Copyright 2025-present, the HuggingFace Inc. Team and AIRAS Inc. Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from typing import List, Optional, Tuple, Union
+from transformers.tokenization_utils import PreTrainedTokenizer
+from transformers import AutoTokenizer
+import json
+import regex as re
+from pathlib import Path
+from typing import Dict, List, Optional, Union
+
+BYTES_TO_UNICODE_REGEX = re.compile(r"'([^']+)':\s*([0-9]+)")
+
+def bytes_to_unicode():
+    bs = list(range(ord("!"), ord("~") + 1)) + list(range(ord("¡"), ord("¬") + 1)) + list(range(ord("®"), ord("ÿ") + 1))
+    cs = bs[:]
+    n = 0
+    for b in range(2**8):
+        if b not in bs:
+            bs.append(b)
+            cs.append(2**8 + n)
+            n += 1
+    cs = [chr(n) for n in cs]
+    return dict(zip(bs, cs))
+
+def get_pairs(word):
+    pairs = set()
+    prev_char = word[0]
+    for char in word[1:]:
+        pairs.add((prev_char, char))
+        prev_char = char
+    return pairs
+
+class SapnousTokenizer(PreTrainedTokenizer):
+    model_input_names = ["input_ids", "attention_mask"]
+
+    def __init__(
+        self,
+        vocab_file: str,
+        merges_file: Optional[str] = None,
+        unk_token: str = "<|endoftext|>",
+        bos_token: str = "<|startoftext|>",
+        eos_token: str = "<|endoftext|>",
+        pad_token: str = "<|pad|>",
+        vision_start_token: str = "<|vision_start|>",
+        vision_end_token: str = "<|vision_end|>",
+        image_token: str = "<|image|>",
+        video_token: str = "<|video|>",
+        add_prefix_space: bool = False,
+        **kwargs
+    ):
+        super().__init__(
+            unk_token=unk_token,
+            bos_token=bos_token,
+            eos_token=eos_token,
+            pad_token=pad_token,
+            **kwargs,
+        )
+
+        self.vocab_file = vocab_file
+        self.merges_file = merges_file
+        self.add_prefix_space = add_prefix_space
+
+        self.special_tokens = {
+            "unk_token": unk_token,
+            "bos_token": bos_token,
+            "eos_token": eos_token,
+            "pad_token": pad_token,
+            "vision_start_token": vision_start_token,
+            "vision_end_token": vision_end_token,
+            "image_token": image_token,
+            "video_token": video_token,
+        }
+
+        with Path(vocab_file).open(encoding="utf-8") as f:
+            self.encoder = json.load(f)
+        self.decoder = {v: k for k, v in self.encoder.items()}
+
+        if merges_file:
+            with Path(merges_file).open(encoding="utf-8") as f:
+                bpe_merges = f.read().strip().split('\n')[1:]
+            bpe_merges = [tuple(merge.split()) for merge in bpe_merges]
+            self.bpe_ranks = dict(zip(bpe_merges, range(len(bpe_merges))))
+        else:
+            self.bpe_ranks = {}
+
+        self.byte_encoder = bytes_to_unicode()
+        self.byte_decoder = {v: k for k, v in self.byte_encoder.items()}
+        self.pat = re.compile(r"""'s|'t|'re|'ve|'m|'ll|'d| ?\w+| ?\d+| ?[^\s\w\d]+|\s+(?!\S)|\s+""")
+
+    def bpe(self, token: str) -> str:
+        if token in self.special_tokens.values():
+            return token
+
+        word = tuple(token)
+        pairs = get_pairs(word)
+
+        if not pairs:
+            return token
+
+        while True:
+            bigram = min(pairs, key=lambda pair: self.bpe_ranks.get(pair, float('inf')))
+            if bigram not in self.bpe_ranks:
+                break
+            first, second = bigram
+            new_word = []
+            i = 0
+            while i < len(word):
+                try:
+                    j = word.index(first, i)
+                    new_word.extend(word[i:j])
+                    if word[j + 1] == second:
+                        new_word.append(first + second)
+                        i = j + 2
+                    else:
+                        new_word.append(word[j])
+                        i = j + 1
+                except ValueError:
+                    new_word.extend(word[i:])
+                    break
+            word = tuple(new_word)
+            if len(word) == 1:
+                break
+            pairs = get_pairs(word)
+        return ' '.join(word)
+
+    def _tokenize(self, text: str) -> List[str]:
+        if self.add_prefix_space:
+            text = ' ' + text
+
+        bpe_tokens = []
+        for token in re.findall(self.pat, text):
+            token = ''.join(self.byte_encoder[ord(b)] for b in token)
+            bpe_tokens.extend(bpe_token for bpe_token in self.bpe(token).split(' '))
+        return bpe_tokens
+
+    def _convert_token_to_id(self, token: str) -> int:
+        return self.encoder.get(token, self.encoder.get(self.unk_token))
+
+    def _convert_id_to_token(self, index: int) -> str:
+        return self.decoder.get(index, self.unk_token)
+
+    def convert_tokens_to_string(self, tokens: List[str]) -> str:
+        text = ''.join(tokens)
+        text = bytearray([self.byte_decoder[c] for c in text]).decode('utf-8', errors='replace')
+        return text
+
+    def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str, str]:
+        if not filename_prefix:
+            filename_prefix = ""
+
+        vocab_file = Path(save_directory) / f"{filename_prefix}vocab.json"
+        merge_file = Path(save_directory) / f"{filename_prefix}merges.txt"
+
+        with vocab_file.open('w', encoding='utf-8') as f:
+            json.dump(self.encoder, f, ensure_ascii=False)
+
+        if self.merges_file:
+            with merge_file.open('w', encoding='utf-8') as f:
+                for merge in self.bpe_ranks:
+                    f.write(f"{merge[0]} {merge[1]}\n")
+            return str(vocab_file), str(merge_file)
+
+        return str(vocab_file)
+
+    def prepare_for_vision(self, text: str) -> str:
+        """Prepare text for vision tasks by adding special tokens."""
+        return f"{self.vision_start_token}{text}{self.vision_end_token}"
+
+    def prepare_for_image(self, text: str) -> str:
+        """Prepare text for image tasks."""
+        return f"{self.image_token}{text}"
+
+    def prepare_for_video(self, text: str) -> str:
+        """Prepare text for video tasks."""
+        return f"{self.video_token}{text}"
+
+    @property
+    def vocab_size(self) -> int:
+        return len(self.encoder)
+
+    def get_vocab(self) -> Dict[str, int]:
+        return self.encoder.copy()
+
+# Register the tokenizer
+AutoTokenizer.register(SapnousTokenizer, "sapnous")

tokenizer_config.json

@@ -0,0 +1,207 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- messages[0]['content'] }}\n    {%- else %}\n        {{- 'You are a helpful assistant.' }}\n    {%- endif %}\n    {{- \"\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- '<|im_start|>system\\n' + messages[0]['content'] + '<|im_end|>\\n' }}\n    {%- else %}\n        {{- '<|im_start|>system\\nYou are a helpful assistant.<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- for message in messages %}\n    {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) or (message.role == \"assistant\" and not message.tool_calls) %}\n        {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {{- '<|im_start|>' + message.role }}\n        {%- if message.content %}\n            {{- '\\n' + message.content }}\n        {%- endif %}\n        {%- for tool_call in message.tool_calls %}\n            {%- if tool_call.function is defined %}\n                {%- set tool_call = tool_call.function %}\n            {%- endif %}\n            {{- '\\n<tool_call>\\n{\"name\": \"' }}\n            {{- tool_call.name }}\n            {{- '\", \"arguments\": ' }}\n            {{- tool_call.arguments | tojson }}\n            {{- '}\\n</tool_call>' }}\n        {%- endfor %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- message.content }}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "SapnousT1Tokenizer",
+  "unk_token": null,
+  "add_bos_token": false
+}