README.md

---
license: mit
library_name: transformers
tags:
- dots_ocr
- image-to-text
- ocr
- document-parse
- layout
- table
- formula
- quantized
- 4-bit
base_model: rednote-hilab/dots.ocr
---

# dots.ocr-4bit: A 4-bit Quantized Version

This repository contains a 4-bit quantized version of the powerful `dots.ocr` model by the **Rednote HiLab**. The quantization was performed using `bitsandbytes` (NF4 precision), providing significant memory and speed improvements with minimal performance loss, making this state-of-the-art model accessible on consumer-grade GPUs.

This work is entirely a derivative of the original model. All credit for the model architecture, training, and groundbreaking research goes to the original authors. A huge thank you to them for open-sourcing their work.

*   **Original Model:** [rednote-hilab/dots.ocr](https://huggingface.co/rednote-hilab/dots.ocr)
*   **Original GitHub:** [https://github.com/rednote-hilab/dots.ocr](https://github.com/rednote-hilab/dots.ocr)
*   **Live Demo (Original):** [https://dotsocr.xiaohongshu.com](https://dotsocr.xiaohongshu.com)

## Model Description (from original authors)
> **dots.ocr** is a powerful, multilingual document parser that unifies layout detection and content recognition within a single vision-language model while maintaining good reading order. Despite its compact 1.7B-parameter LLM foundation, it achieves state-of-the-art(SOTA) performance.

## How to Use This 4-bit Version

First, ensure you have the necessary dependencies installed. Because this model uses custom code, you **must** clone the original repository and install it.

```bash
# It's recommended to clone the original repo to get all utility scripts
git clone https://github.com/rednote-hilab/dots.ocr.git
cd dots.ocr

# Install the custom code and dependencies
pip install -e .
pip install torch transformers accelerate bitsandbytes peft sentencepiece
```

You can then use the 4-bit model with the following Python script. Note the inclusion of generation parameters (repetition_penalty, do_sample, etc.), which are recommended to prevent potential looping with the quantized model.

```python
import torch
from transformers import AutoModelForCausalLM, AutoProcessor
from PIL import Image
from huggingface_hub import snapshot_download

from qwen_vl_utils import process_vision_info

MODEL_ID = "helizac/dots.ocr-4bit"

local_model_path = snapshot_download(repo_id=MODEL_ID)

model = AutoModelForCausalLM.from_pretrained(local_model_path, device_map="auto", trust_remote_code=True, torch_dtype=torch.bfloat16, attn_implementation="flash_attention_2")
processor = AutoProcessor.from_pretrained(local_model_path, trust_remote_code=True, use_fast=True)

image_path = "test.jpg"
image = Image.open(image_path)

prompt_text = """\
Please output the layout information from the image, including each layout element's bbox, its category, and the corresponding text content within the bbox.
1. Bbox format: [x1, y1, x2, y2]
2. Layout Categories: The possible categories are ['Caption', 'Footnote', 'Formula', 'List-item', 'Page-footer', 'Page-header', 'Picture', 'Section-header', 'Table', 'Text', 'Title'].
3. Text Extraction & Formatting Rules:
- Picture: For the 'Picture' category, the text field should be omitted.
- Formula: Format its text as LaTeX.
- Table: Format its text as HTML.
- All Others (Text, Title, etc.): Format their text as Markdown.
4. Constraints:
- The output text must be the original text from the image, with no translation.
- All layout elements must be sorted according to human reading order.
5. Final Output: The entire output must be a single JSON object.\
"""

messages = [{"role": "user", "content": [{"type": "image", "image": image_path}, {"type": "text", "text": prompt_text}]}]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, _ = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs, padding=True, return_tensors="pt").to(model.device)

generated_ids = model.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.6, top_p=0.9, repetition_penalty=1.15)

generated_ids_trimmed = [out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)]
output_text = processor.batch_decode(generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]

print(output_text)
```

## License

This model is released under the MIT License, same as the original model.