Initial commit

.gitattributes

@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
+*.json filter=lfs diff=lfs merge=lfs -text
+images/table.png filter=lfs diff=lfs merge=lfs -text
+images/tech.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,6 +1,171 @@
----
-license: other
-license_name: nvidia-open-model-license
-license_link: >-
-  https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/
----
+---
+license: other
+license_name: nvidia-open-model-license
+license_link: >-
+  https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/
+---
+
+# Llama-Nemotron-Nano-VL-8B-V1
+
+## Model Overview
+
+### Description
+
+ Llama-Nemotron-Nano-VL-8B-V1 is a leading document intelligence vision language model (VLMs) that enables the ability to query and summarize images and video from the physical or virtual world. Llama-Nemotron-Nano-VL-8B-V1 is deployable in the data center, cloud and at the edge, including Jetson Orin and laptop by AWQ 4bit quantization through TinyChat framework. We find: (1) image-text pairs are not enough, interleaved image-text is essential; (2) unfreezing LLM during interleaved image-text pre-training enables in-context learning; (3)re-blending text-only instruction data is crucial to boost both VLM and text-only performance.  
+
+This model was trained on commercial images and videos for all three stages of training and supports single image and video inference.
+
+### License/Terms of Use
+**Governing Terms:**  
+
+Your use of the model is governed by the [NVIDIA Open License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). Additional Information: Llama 3.1 Community Model License; Built with Llama.
+
+**Additional Information:**
+
+[Llama 3.1 Community Model License](https://www.llama.com/llama3_1/license/); Built with Llama.
+
+
+### Deployment Geography:
+
+Global
+
+### Use Case:
+
+Customers: AI foundry enterprise customers
+
+Use Cases: Image summarization. Text-image analysis, Optical Character Recognition, Interactive Q&A on images, Comparison and contrast of multiple images, Text Chain-of-Thought reasoning.
+  
+
+## Release Date:  
+
+- Build.Nvidia.com [June 3rd, 2025] via [nvidia/llama-3_1-nemotron-nano-vl-8b-v1](https://build.nvidia.com/nvidia/llama-3_1-nemotron-nano-vl-8b-v1)
+- Hugging Face [June 3rd, 2025]
+
+## Model Architecture:
+
+**Network Type:** Transformer 
+
+**Network Architecture:** 
+
+Vision Encoder: CRadioV2-H
+
+Language Encoder: Llama-3.1-8B-Instruct
+
+### Input
+
+Input Type(s): Image, Video, Text
+- Input Images Supported: Multiple images within 16K input + output tokens
+- Language Supported: English only
+
+Input Format(s): Image (Red, Green, Blue (RGB)), Video (.mp4), and Text (String)
+
+Input Parameters: Image (2D), Video (3D), Text (1D)
+
+Other Properties Related to Input:
+
+- Input + Output Token: 16K
+- Maximum Resolution: Determined by a 12-tile layout constraint, with each tile being 512 × 512 pixels. This supports aspect ratios such as:
+  - 4 × 3 layout: up to 2048 × 1536 pixels
+  - 3 × 4 layout: up to 1536 × 2048 pixels
+  - 2 × 6 layout: up to 1024 × 3072 pixels
+  - 6 × 2 layout: up to 3072 × 1024 pixels
+  - Other configurations allowed, provided total tiles ≤ 12
+- Channel Count: 3 channels (RGB)
+- Alpha Channel: Not supported (no transparency)
+
+### Output
+Output Type(s): Text
+
+Output Formats: String
+
+Output Parameters: 1D
+
+Other Properties Related to Output: Input + Output Token: 16K
+
+
+
+Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
+
+### Software Integration
+Runtime Engine(s): TensorRT-LLM<br>
+Supported Hardware Microarchitecture Compatibility: H100 SXM 80GB<br>
+Supported Operating System(s): Linux<br>
+
+### Model Versions:
+Llama-3.1-Nemotron-Nano-VL-8B-V1 
+
+## Usage
+
+```python
+from PIL import Image
+from transformers import AutoImageProcessor, AutoModel, AutoTokenizer
+
+path = "."
+model = AutoModel.from_pretrained(path, trust_remote_code=True, device_map="cuda").eval()
+tokenizer = AutoTokenizer.from_pretrained(path)
+image_processor = AutoImageProcessor.from_pretrained(path, trust_remote_code=True, device="cuda")
+
+image1 = Image.open("images/example1a.jpeg")
+image2 = Image.open("images/example1b.jpeg")
+image_features = image_processor([image1, image2])
+
+generation_config = dict(max_new_tokens=1024, do_sample=False, eos_token_id=tokenizer.eos_token_id)
+
+question = 'Describe the two images.'
+response = model.chat(
+    tokenizer=tokenizer, question=question, generation_config=generation_config,
+    **image_features)
+
+print(f'User: {question}\nAssistant: {response}')
+```
+
+
+## Training/Evaluation Dataset:
+NV-Pretraining and NV-CosmosNemotron-SFT were used for training and evaluation
+
+Data Collection Method by dataset (Training and Evaluation):  <br>
+* Hybrid: Human, Synthetic <br>
+
+Labeling Method by dataset (Training and Evaluation):  <br>
+* Hybrid: Human, Synthetic <br>
+
+
+Additionally, the dataset collection (for training and evaluation) consists of a mix of internal and public datasets designed for training and evaluation across various tasks. It includes: <br>
+• Internal datasets built with public commercial images and internal labels, supporting tasks like conversation modeling and document analysis.<br>
+• Public datasets sourced from publicly available images and annotations, adapted for tasks such as image captioning and visual question answering.<br>
+• Synthetic datasets generated programmatically for specific tasks like tabular data understanding.<br>
+• Specialized datasets for safety alignment, function calling, and domain-specific tasks (e.g., science diagrams, financial question answering).<br>
+
+
+
+## Evaluation Benchmarks:
+
+| Benchmark | Score |
+| --- | --- |
+| MMMU Val with chatGPT as a judge | 48.2% |
+| AI2D | 85.0% |
+| ChartQA  | 86.3% |
+| InfoVQA Val | 77.4% |
+| OCRBench | 839 |
+| OCRBenchV2 English | 60.1% |
+| OCRBenchV2 Chinese | 37.9% |
+| DocVQA val | 91.2% |
+| VideoMME  | 54.7% |
+
+
+
+# Inference:
+**Engine:** TTensorRT-LLM <br>
+**Test Hardware:** <br>
+* 1x NVIDIA H100 SXM 80GB
+
+
+## Ethical Considerations:
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications.  When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.  For more detailed information on ethical considerations for this model, please see the Model Card++ [Explainability](explainability.md), [Bias](bias.md), [Safety & Security](safety.md), and [Privacy](privacy.md) Subcards.  Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
+
+Users are responsible for model inputs and outputs. Users are responsible for ensuring safe integration of this model, including implementing guardrails as well as other safety mechanisms, prior to deployment. 
+
+Outputs generated by these models may contain political content or other potentially misleading information, issues with content security and safety, or unwanted bias that is independent of our oversight.
+
+
+ 

bias.md

@@ -0,0 +1,4 @@
+Field                                                                                               | Response
+:---------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+Participation considerations from adversely impacted groups [protected classes](https://www.senate.ca.gov/content/protected-classes) in model design and testing:  | We actively considered participation from adversely impacted groups and protected classes during model design and testing by engaging diverse stakeholders, reviewing data for representation, and evaluating outputs for bias. Feedback channels were provided throughout development.
+Measures taken to mitigate against unwanted bias:                                                   | We took several steps to reduce unwanted bias, including:<br>- **Evaluating** the model’s answers with regard to fairness for different groups<br>- Using tools to **identify** and measure unfairness. 

config.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d28e10441f08143e1bad3121dbdb702df97d5ca448758d1982920e05dba0c62d
+size 7662

configuration.py

@@ -0,0 +1,56 @@
+# --------------------------------------------------------
+# Adapted from https://huggingface.co/OpenGVLab/InternVL2-Llama3-76B under MIT License
+#     LICENSE is in incl_licenses directory.
+# --------------------------------------------------------
+
+from transformers import AutoConfig, LlamaConfig
+from transformers.configuration_utils import PretrainedConfig
+from transformers.utils import logging
+from transformers.dynamic_module_utils import get_class_from_dynamic_module
+
+logger = logging.get_logger(__name__)
+
+class Llama_Nemotron_Nano_VL_Config(PretrainedConfig):
+    model_type = 'Llama_Nemotron_Nano_VL'
+    is_composition = True
+
+    def __init__(
+        self,
+        vision_config=None,
+        llm_config=None,
+        force_image_size=None,
+        downsample_ratio=0.5,
+        template=None,
+        ps_version='v1',
+        image_tag_type="internvl",
+        projector_hidden_size=4096,
+        vit_hidden_size=1280,
+        attn_implementation="flash_attention_2",
+        **kwargs
+    ):
+        super().__init__(**kwargs)
+        
+        if vision_config is not None:
+            assert "auto_map" in vision_config and "AutoConfig" in vision_config["auto_map"]
+            vision_auto_config = get_class_from_dynamic_module(*vision_config["auto_map"]["AutoConfig"].split("--")[::-1])
+            self.vision_config = vision_auto_config(**vision_config)
+        else:
+            self.vision_config = PretrainedConfig()
+
+        if llm_config is None:
+            self.llm_config = LlamaConfig()
+        else:
+            self.llm_config = LlamaConfig(**llm_config)
+
+        # Assign configuration values
+        self.force_image_size = force_image_size
+        self.downsample_ratio = downsample_ratio
+        self.template = template  # TODO move out of here and into the tokenizer
+        self.ps_version = ps_version  # Pixel shuffle version
+        self.image_tag_type = image_tag_type # TODO: into the tokenizer too?
+        self.projector_hidden_size = projector_hidden_size
+        self.vit_hidden_size = vit_hidden_size
+
+        self._attn_implementation = attn_implementation
+        self.vision_config.use_flash_attn = "flash_attention" in self._attn_implementation
+        self.llm_config._attn_implementation = self._attn_implementation

examples.py

@@ -0,0 +1,56 @@
+import torch
+from transformers import AutoTokenizer, AutoModel, AutoImageProcessor
+from PIL import Image
+
+path = "."
+model = AutoModel.from_pretrained(
+    path,
+    torch_dtype=torch.bfloat16,
+    low_cpu_mem_usage=True,
+    trust_remote_code=True,
+    device_map="cuda",).eval()
+
+tokenizer = AutoTokenizer.from_pretrained(path)
+image_processor = AutoImageProcessor.from_pretrained(path, device="cuda", trust_remote_code=True)
+
+generation_config = dict(max_new_tokens=1024, do_sample=False, eos_token_id=tokenizer.eos_token_id)
+
+# pure-text conversation
+question = 'What happened in 1986?'
+response, history = model.chat(
+    tokenizer, None, question, generation_config, history=None, return_history=True
+)
+print(f'User: {question}\nAssistant: {response}')
+
+# single-image single-round conversation
+image_path = 'images/table.png'
+image_features = image_processor(Image.open(image_path))
+question = '<image>\nExtract the table in this image as HTML.'
+response = model.chat(
+    tokenizer=tokenizer, question=question, generation_config=generation_config,
+    **image_features
+)
+print(f'User: {question}\nAssistant: {response}')
+
+# single-image single-round conversation
+image_path = 'images/tech.png'
+image_features = image_processor(Image.open(image_path))
+question = '<image>\nList in bullet point the most important Technological breakthrough of Nvidia Hopper.'
+response = model.chat(
+    tokenizer=tokenizer, question=question, generation_config=generation_config,
+    **image_features
+)
+print(f'User: {question}\nAssistant: {response}')
+
+# two image single-round conversation
+image_features = image_processor([
+    Image.open('images/example1a.jpeg'),
+    Image.open('images/example1b.jpeg')
+])
+
+question = '<image-1>: <image>\n<image-2>: <image>\nBriefly describe the two images.'
+response = model.chat(
+    tokenizer=tokenizer, question=question, generation_config=generation_config,
+    **image_features
+)
+print(f'User: {question}\nAssistant: {response}')

explainability.md

@@ -0,0 +1,13 @@
+Field                                                                                                  | Response
+:------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------
+Intended Application & Domain:                                                                         | Visual Question Answering
+Model Type:                                                                                            | Transformer
+Intended Users:                                                                                        | Generative AI creators working with conversational AI models and image content.
+Output:                                                                                                | Text (Responds to posed question, stateful - remembers previous answers)
+Describe how the model works:                                                                          | Chat based on image/video content
+Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of:   | Not Applicable
+Technical Limitations:                                                                                 | Max Number of images supported: 4.<br><br>**Context Length:** Supports up to 16,000 tokens total (input + output). If exceeded, input is truncated from the start, and generation ends with an EOS token. Longer prompts may risk performance loss.<br><br>If the model fails (e.g., generates incorrect responses, repeats, or gives poor responses), issues are diagnosed via benchmarks, human review, and internal debugging tools. Only use NVIDIA provided models that use safetensors format. <br><br>Do not expose the vLLM host to a network where any untrusted connections may reach the host. Only use NVIDIA provided models that use safetensors format.
+Verified to have met prescribed NVIDIA quality standards:                                              | Yes
+Performance Metrics:                                                                                   | MMMU Val with chatGPT as a judge, AI2D, ChartQA Test, InfoVQA Val, OCRBench, OCRBenchV2 English, OCRBenchV2 Chinese, DocVQA val, VideoMME (16 frames), SlideQA (F1)
+Potential Known Risks:                                                                                 | The Model may produce output that is biased, toxic, or incorrect responses. Therefore, the model may amplify those biases and return toxic responses especially when prompted with toxic prompts. The Model may also generate answers that may be inaccurate, omit key information, or include irrelevant or redundant text, producing socially unacceptable or undesirable text, even if the prompt itself does not include anything explicitly offensive.<br>While we have taken safety and security into account and are continuously improving, outputs may still contain political content, misleading information, or unwanted bias beyond our control.
+Licensing:                                                                                             | **Governing Terms:**<br>Your use of the software container and model is governed by the [NVIDIA Software and Model Evaluation License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-and-model-evaluation-license/).<br><br>**Additional Information:**<br>[Llama 3.1 Community Model License](https://www.llama.com/llama3_1/license/); Built with Llama.    

image_processing.py

@@ -0,0 +1,112 @@
+from typing import List, Optional, Union
+
+from PIL import Image
+import torch
+from transformers.image_processing_base import BatchFeature
+from transformers.image_processing_utils_fast import (BaseImageProcessorFast,
+                                                      divide_to_patches)
+from transformers.image_utils import (ChannelDimension, SizeDict,
+                                      get_image_size, make_list_of_images,
+                                      get_image_type, ImageInput, ImageType)
+from transformers.utils import TensorType
+
+
+def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):
+    best_factor = float('-inf')
+    best_ratio = (1, 1)
+    area = width * height
+    for ratio in target_ratios:
+        target_aspect_ratio = ratio[0] / ratio[1]
+        factor_based_on_area_n_ratio = min(
+            (ratio[0]*ratio[1]*image_size*image_size)/ area, 0.6
+            )* min(
+                target_aspect_ratio/aspect_ratio, aspect_ratio/target_aspect_ratio)
+        if factor_based_on_area_n_ratio > best_factor:
+            best_factor = factor_based_on_area_n_ratio
+            best_ratio = ratio
+    return best_ratio
+
+
+class LlamaNemotronNanoVLImageProcessor(BaseImageProcessorFast):
+    model_input_names = ["pixel_values"]
+
+    def __init__(self, image_size=512, max_num_tiles=12, use_thumbnail=True, **kwargs):
+        super().__init__(**kwargs)
+        self.image_size = image_size
+        self.max_num_tiles = max_num_tiles
+        self.use_thumbnail = use_thumbnail
+
+    # Based on https://github.com/OpenGVLab/InternVL/blob/c62fa4f7c850165d7386bdc48ac6bc5a6fab0864/internvl_chat/internvl/train/dataset.py#L702
+    def dynamic_preprocess(self, image, image_size=448, max_num_tiles=12, use_thumbnail=False):
+        orig_height, orig_width = get_image_size(image, channel_dim=ChannelDimension.FIRST)
+        aspect_ratio = orig_width / orig_height
+
+        # calculate the existing image aspect ratio
+        target_ratios = set(
+            (i, j) for n in range(1, max_num_tiles + 1) for i in range(1, n + 1) for j in range(1, n + 1) if
+            i * j <= max_num_tiles and i * j >= 1)
+        target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])
+
+        # find the closest aspect ratio to the target
+        target_aspect_ratio = find_closest_aspect_ratio(
+            aspect_ratio, target_ratios, orig_width, orig_height, image_size)
+
+        # calculate the target width and height
+        target_width = image_size * target_aspect_ratio[0]
+        target_height = image_size * target_aspect_ratio[1]
+
+        resized_img = self.resize(image, SizeDict(height=target_height, width=target_width))
+        patches = divide_to_patches(resized_img, image_size)
+        if use_thumbnail and len(patches) != 1:
+            patches.append(self.resize(image, SizeDict(height=image_size, width=image_size)))
+
+        return patches
+
+    def _process_image(
+        self,
+        image: ImageInput,
+        **kwargs,
+    ) -> torch.Tensor:
+        image_type = get_image_type(image)
+        if image_type not in [ImageType.PIL]:
+            raise ValueError(f"Unsupported input image type {image_type}. Only PIL images supported")
+        image = image.resize((image.width * 2, image.height * 2), Image.BILINEAR)
+        return super()._process_image(image, **kwargs)
+
+    def _preprocess(
+        self,
+        images: List[torch.Tensor],
+        image_size: int = None,
+        max_num_tiles: int = None,
+        use_thumbnail: bool = None,
+        do_rescale: bool = None,
+        return_tensors: Optional[Union[str, TensorType]] = None,
+        **kwargs,
+    ) -> List[torch.Tensor]:
+        image_size = image_size if image_size is not None else self.image_size
+        max_num_tiles = max_num_tiles if max_num_tiles is not None else self.max_num_tiles
+        use_thumbnail = use_thumbnail if use_thumbnail is not None else self.use_thumbnail
+        do_rescale = do_rescale if do_rescale is not None else self.do_rescale
+
+        images = make_list_of_images(images)
+
+        all_patches = []
+        num_patches = []
+        for image in images:
+            patches = self.dynamic_preprocess(
+                image, image_size, max_num_tiles, use_thumbnail
+            )
+            all_patches.extend(patches)
+            num_patches.append(len(patches))
+
+        pixel_values = torch.stack(all_patches, dim=0)
+        pixel_values = self.rescale_and_normalize(
+            pixel_values,
+            do_rescale,
+            self.rescale_factor,
+            do_normalize=self.do_normalize,
+            image_mean=self.image_mean,
+            image_std=self.image_std
+        )
+
+        return BatchFeature(data={"pixel_values": pixel_values, "num_patches": num_patches}, tensor_type=return_tensors)

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:415e7a900327cc5a057b5e57641ba689a834e5103adc2eece0512f8793415f34
+size 17443626956

modeling.py

@@ -0,0 +1,323 @@
+# --------------------------------------------------------
+# Adapted from https://huggingface.co/OpenGVLab/InternVL2-Llama3-76B under MIT License
+#     LICENSE is in incl_licenses directory.
+# --------------------------------------------------------
+
+
+import warnings
+from typing import List, Optional, Tuple, Union
+
+import torch.utils.checkpoint
+import transformers
+from torch import nn
+from torch.nn import CrossEntropyLoss
+from transformers import AutoModel, AutoModelForCausalLM, GenerationConfig
+from transformers.modeling_outputs import CausalLMOutputWithPast
+from transformers.modeling_utils import PreTrainedModel
+from transformers.utils import logging
+
+from .configuration import Llama_Nemotron_Nano_VL_Config
+
+logger = logging.get_logger(__name__)
+
+
+"""
+The following code is adapted from the
+https://huggingface.co/OpenGVLab/InternVL2-Llama3-76B/blob/main/modeling_internvl_chat.py repository
+
+The chat function is adapted to handle NVLM 1-D tile-tagging design for dynamic high-resolution images.
+"""
+def version_cmp(v1, v2, op='eq'):
+    import operator
+
+    from packaging import version
+    op_func = getattr(operator, op)
+    return op_func(version.parse(v1), version.parse(v2))
+
+
+class Llama_Nemotron_Nano_VL_Model(PreTrainedModel):
+    config_class = Llama_Nemotron_Nano_VL_Config
+    main_input_name = 'pixel_values'
+    _supports_flash_attn_2 = True
+    _no_split_modules = ['InternVisionModel', 'SiglipVisionModel', 'Qwen2DecoderLayer']
+
+    def __init__(self, config: Llama_Nemotron_Nano_VL_Config):
+        super().__init__(config)
+
+        assert version_cmp(transformers.__version__, '4.36.2', 'ge')
+        image_size = config.force_image_size
+        patch_size = config.patch_size
+        self.patch_size = patch_size
+        self.template = config.template
+        self.num_image_token = int((image_size // patch_size) ** 2 * (config.downsample_ratio ** 2))
+        self.downsample_ratio = config.downsample_ratio
+        self.ps_version = config.ps_version
+        self.image_tag_type = config.image_tag_type
+
+        logger.info(f'num_image_token: {self.num_image_token}')
+        logger.info(f'ps_version: {self.ps_version}')
+
+        self.language_model = AutoModelForCausalLM.from_config(config.llm_config, torch_dtype=torch.bfloat16)
+        self.vision_model = AutoModel.from_config(config.vision_config, trust_remote_code=True)
+        self.vision_model.model._initialize_weights = self.vision_model.model._init_weights  # WAR for transformers issue 38358 
+
+        self.drop_vision_class_token = True
+
+        # Construct the vision projection.
+        # Default
+        vit_hidden_size = config.vit_hidden_size
+        vision_projection_hidden_size = config.projector_hidden_size
+        llm_hidden_size = config.llm_config.hidden_size
+
+        self.mlp1 = nn.Sequential(
+            nn.LayerNorm(vit_hidden_size * int(1 / self.downsample_ratio) ** 2, bias=True),
+            nn.Linear(vit_hidden_size * int(1 / self.downsample_ratio) ** 2, vision_projection_hidden_size, bias=True),
+            nn.GELU(),
+            nn.Linear(vision_projection_hidden_size, llm_hidden_size, bias=True)
+        )
+        self.mlp1 = self.mlp1.to(self.language_model.config.torch_dtype)
+
+        self.img_context_token_id = None
+
+    def forward(
+            self,
+            pixel_values: torch.FloatTensor,
+            input_ids: torch.LongTensor = None,
+            attention_mask: Optional[torch.Tensor] = None,
+            position_ids: Optional[torch.LongTensor] = None,
+            image_flags: Optional[torch.LongTensor] = None,
+            past_key_values: Optional[List[torch.FloatTensor]] = None,
+            labels: Optional[torch.LongTensor] = None,
+            use_cache: Optional[bool] = None,
+            output_attentions: Optional[bool] = None,
+            output_hidden_states: Optional[bool] = None,
+            return_dict: Optional[bool] = None,
+    ) -> Union[Tuple, CausalLMOutputWithPast]:
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        image_flags = image_flags.squeeze(-1)
+        input_embeds = self.language_model.get_input_embeddings()(input_ids)
+
+        vit_embeds = self.extract_feature(pixel_values)
+        vit_embeds = vit_embeds[image_flags == 1]
+        vit_batch_size = pixel_values.shape[0]
+
+        B, N, C = input_embeds.shape
+        input_embeds = input_embeds.reshape(B * N, C)
+
+        if torch.distributed.get_rank() == 0:
+            print(f'dynamic ViT batch size: {vit_batch_size}, images per sample: {vit_batch_size / B}, dynamic token length: {N}')
+
+        input_ids = input_ids.reshape(B * N)
+        selected = (input_ids == self.img_context_token_id)
+        try:
+            input_embeds[selected] = input_embeds[selected] * 0.0 + vit_embeds.reshape(-1, C)
+        except Exception as e:
+            vit_embeds = vit_embeds.reshape(-1, C)
+            print(f'warning: {e}, input_embeds[selected].shape={input_embeds[selected].shape}, '
+                  f'vit_embeds.shape={vit_embeds.shape}')
+            n_token = selected.sum()
+            input_embeds[selected] = input_embeds[selected] * 0.0 + vit_embeds[:n_token]
+
+        input_embeds = input_embeds.reshape(B, N, C)
+
+        outputs = self.language_model(
+            inputs_embeds=input_embeds,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+        logits = outputs.logits
+
+        loss = None
+        if labels is not None:
+            # Shift so that tokens < n predict n
+            shift_logits = logits[..., :-1, :].contiguous()
+            shift_labels = labels[..., 1:].contiguous()
+            # Flatten the tokens
+            loss_fct = CrossEntropyLoss()
+            shift_logits = shift_logits.view(-1, self.language_model.config.vocab_size)
+            shift_labels = shift_labels.view(-1)
+            # Enable model parallelism
+            shift_labels = shift_labels.to(shift_logits.device)
+            loss = loss_fct(shift_logits, shift_labels)
+
+        if not return_dict:
+            output = (logits,) + outputs[1:]
+            return (loss,) + output if loss is not None else output
+
+        return CausalLMOutputWithPast(
+            loss=loss,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+    def pixel_shuffle(self, x, scale_factor=0.5):
+        n, w, h, c = x.size()
+        # N, W, H, C --> N, W, H * scale, C // scale
+        x = x.view(n, w, int(h * scale_factor), int(c / scale_factor))
+        # N, W, H * scale, C // scale --> N, H * scale, W, C // scale
+        x = x.permute(0, 2, 1, 3).contiguous()
+        # N, H * scale, W, C // scale --> N, H * scale, W * scale, C // (scale ** 2)
+        x = x.view(n, int(h * scale_factor), int(w * scale_factor),
+                   int(c / (scale_factor * scale_factor)))
+        if self.ps_version == 'v1':
+            warnings.warn("In ps_version 'v1', the height and width have not been swapped back, "
+                          'which results in a transposed image.')
+        else:
+            x = x.permute(0, 2, 1, 3).contiguous()
+        return x
+
+    def extract_feature(self, pixel_values):
+        vit_embeds = self.vision_model(pixel_values).features
+        vit_embeds = vit_embeds.to(dtype=torch.bfloat16)
+        h = w = int(vit_embeds.shape[1] ** 0.5)
+        vit_embeds = vit_embeds.reshape(vit_embeds.shape[0], h, w, -1)
+        vit_embeds = self.pixel_shuffle(vit_embeds, scale_factor=self.downsample_ratio)
+        vit_embeds = vit_embeds.reshape(vit_embeds.shape[0], -1, vit_embeds.shape[-1])
+        vit_embeds = self.mlp1(vit_embeds)
+        return vit_embeds
+
+    def _format_image_token(self, query, num_patches_list, IMG_CONTEXT_TOKEN):
+        # Split by '<image>' and rejoin with appropriate tokens
+        parts = query.split('<image>')
+        if len(parts) - 1 != len(num_patches_list):
+            raise ValueError(f"Number of <image> tokens ({len(parts) - 1}) doesn't match num_patches_list length ({len(num_patches_list)})")
+        
+        result = parts[0]
+        for num_patches, part in zip(num_patches_list, parts[1:]):
+            if self.image_tag_type == "nvlm":
+                tile_pos_identifiers = [f"<tile_{j}>" for j in range(1, num_patches)] + ["<tile_global_thumbnail>"]
+                image_tokens = ''
+                for tile_pos_identifier in tile_pos_identifiers:
+                    image_tokens += tile_pos_identifier + IMG_CONTEXT_TOKEN * self.num_image_token
+                image_tokens = '<Image>' + image_tokens + '</Image>'
+            elif self.image_tag_type == "internvl":
+                image_tokens = IMG_CONTEXT_TOKEN * self.num_image_token * num_patches
+                image_tokens = '<img>' + image_tokens + '</img>'
+            else:
+                raise ValueError(f"Unknown image tag type {self.image_tag_type}")
+            
+            result += image_tokens + part
+        
+        return result
+
+    """
+    Adapts the chat function to handle NVLM 1-D tile-tagging design for dynamic high-resolution images.
+    Additionally, it supports the following:
+        - Chat without a system prompt.
+        - Chat without an image prompt.
+    """
+    def chat(self, tokenizer, pixel_values, question, generation_config, history=None, return_history=False,
+             num_patches=None, IMG_START_TOKEN='<img>', IMG_END_TOKEN='</img>',
+             IMG_CONTEXT_TOKEN='<image>', verbose=False, visual_features=None, system_prompt=None):
+
+        if num_patches is None:
+            num_patches_list = [pixel_values.shape[0]] if pixel_values is not None else []
+        elif isinstance(num_patches, torch.Tensor):
+            num_patches_list = num_patches.tolist()
+        else:
+            num_patches_list = num_patches
+
+        if history is None and pixel_values is not None and '<image>' not in question:
+            question = '<image>\n' * len(num_patches_list) + question
+
+        assert pixel_values is None or len(pixel_values) == sum(num_patches_list)
+
+        img_context_token_id = tokenizer.convert_tokens_to_ids(IMG_CONTEXT_TOKEN)
+        self.img_context_token_id = img_context_token_id
+
+        eos_token_id = tokenizer.eos_token_id
+
+        messages = []
+        if system_prompt is not None:
+            messages.append({"role": "system", "content": system_prompt})
+
+        history = [] if history is None else history
+        for (old_question, old_answer) in history:
+            messages.append({"role": "user", "content": old_question})
+            messages.append({"role": "assistant", "content": old_answer})
+        
+        messages.append({"role": "user", "content": question})
+        query = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+
+        if verbose and pixel_values is not None:
+            image_bs = pixel_values.shape[0]
+            print(f'dynamic ViT batch size: {image_bs}')
+
+        query = self._format_image_token(query, num_patches_list, IMG_CONTEXT_TOKEN)
+
+        model_inputs = tokenizer(query, return_tensors='pt', add_special_tokens=False)
+        input_ids = model_inputs['input_ids'].cuda()
+        attention_mask = model_inputs['attention_mask'].cuda()
+        generation_config['eos_token_id'] = eos_token_id
+        generation_output = self.generate(
+            pixel_values=pixel_values,
+            visual_features=visual_features,
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            **generation_config
+        )
+
+        response = tokenizer.batch_decode(generation_output)[0]
+        response = response.split(tokenizer.eos_token)[0].strip()
+        history.append((question, response))
+        if return_history:
+            return response, history
+        else:
+            query_to_print = query.replace(IMG_CONTEXT_TOKEN, '')
+            query_to_print = query_to_print.replace(f'{IMG_START_TOKEN}{IMG_END_TOKEN}', '<image>')
+            if verbose:
+                print(query_to_print, response)
+            return response
+
+    @torch.no_grad()
+    def generate(
+            self,
+            pixel_values: Optional[torch.FloatTensor] = None,
+            input_ids: Optional[torch.FloatTensor] = None,
+            attention_mask: Optional[torch.LongTensor] = None,
+            visual_features: Optional[torch.FloatTensor] = None,
+            generation_config: Optional[GenerationConfig] = None,
+            output_hidden_states: Optional[bool] = None,
+            return_dict: Optional[bool] = None,
+            **generate_kwargs,
+    ) -> torch.LongTensor:
+
+        assert self.img_context_token_id is not None
+        if pixel_values is not None:
+            if visual_features is not None:
+                vit_embeds = visual_features.cuda()
+                vit_embeds = self.mlp1(vit_embeds)
+            else:
+                vit_embeds = self.extract_feature(pixel_values)
+
+            input_embeds = self.language_model.get_input_embeddings()(input_ids)
+            B, N, C = input_embeds.shape
+            input_embeds = input_embeds.reshape(B * N, C)
+
+            input_ids = input_ids.reshape(B * N)
+            selected = (input_ids == self.img_context_token_id)
+            assert selected.sum() != 0
+            input_embeds[selected] = vit_embeds.reshape(-1, C).to(input_embeds.device)
+
+            input_embeds = input_embeds.reshape(B, N, C)
+        else:
+            input_embeds = self.language_model.get_input_embeddings()(input_ids)
+
+        outputs = self.language_model.generate(
+            inputs_embeds=input_embeds,
+            attention_mask=attention_mask,
+            generation_config=generation_config,
+            output_hidden_states=output_hidden_states,
+            use_cache=True,
+            **generate_kwargs,
+        )
+
+        return outputs

preprocessor_config.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:63ed8d1a7d866d2322cac5c73d8c3fa033a79ada289df02530ca0d16688fde8f
+size 283

privacy.md

@@ -0,0 +1,8 @@
+Field                                                                                                                              |  Response
+:----------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------
+Generatable or reverse engineerable personal data?                                                     |  None
+Personal data used to create this model?                                                                                       |  None
+How often is dataset reviewed?                                                                                                     |  Before Every Release                                                  |  Yes
+Does data labeling (annotation, metadata) comply with privacy laws?                                                                |  Yes
+Is data compliant with data subject requests for data correction or removal, if such a request was made?                           |  No, not possible with externally-sourced data. Applicable Privacy Policy: https://www.nvidia.com/en-us/about-nvidia/privacy-policy/      
+ 

safety.md

@@ -0,0 +1,7 @@
+Field                                               |  Response
+:---------------------------------------------------|:----------------------------------
+Model Application(s):                               | - Extracting and understanding information from text and images in documents (OCR, tables, charts, diagrams, math expressions)<br>- Recognizing objects, attributes, and semantic relationships in images<br>- Interactive Q&A based on images and text<br>- Analyzing and summarizing similarities and differences between images
+Describe the life critical impact (if present).     |  Not Applicable
+Use Case Restrictions:                              |  Governing Terms:Your use of the model is governed by the [NVIDIA Open License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). Additional Information: Llama 3.1 Community Model License; Built with Llama.. Additional Information: [Llama 3.1 Community Model License](https://www.llama.com/llama3_1/license/); Built with Llama.
+Model and dataset restrictions:                     |  The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development. Restrictions enforce dataset access during training, and dataset license constraints adhered to.  
+

special_tokens_map.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6f38c73729248f6c127296386e3cdde96e254636cc58b4169d3fd32328d9a8ec
+size 296

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f5725802f93aea2c6126c605128904b2feaad45cdc0b16240fb153481a229948
+size 17211566

tokenizer_config.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fce27bafb0f4fd5fcbf3fc9a7152fb659fad40f7371142d2cbdf20a3ad8dae59
+size 52949