LightOnOCR-2-1B-htr-polish-1

This repository contains a PEFT/LoRA adapter for lightonai/LightOnOCR-2-1B-base. It was fine-tuned for handwritten text recognition (HTR) on Polish manuscript scans (manuscripts from the 20th century, particularly from the first half of the century).

This is not a standalone full model. Load it together with the base model:

• Base model: lightonai/LightOnOCR-2-1B-base
• Adapter: IHPAN/LightOnOCR-2-1B-htr-polish-1

Intended Use

The base model is a small model specialising primarily in OCR tasks. Its fine-tuning is an experiment to explore what can be achieved with it in the field of handwriting recognition.

The adapter is intended for OCR/HTR transcription of Polish handwritten source material, especially historical manuscript scans similar to the fine-tuning data. It should preserve Polish diacritics, punctuation, and line breaks when possible.

It is not intended for high-stakes fully automatic transcription without human review. Handwriting recognition errors should be expected, especially for degraded images, unusual hands, marginalia, crossed-out text, tables, stamps, or mixed printed/handwritten layouts.

Training Data

The adapter was fine-tuned on a local collection of paired image/transcription files:

• images: scanned manuscript pages or page fragments
• references: UTF-8 .txt transcriptions
• language: Polish
• training split used by the local script: 765 samples
• validation split used by the local script: 85 samples
• test split: 10 samples

The training data is not included in this model repository.

Training Procedure

Fine-tuning used QLoRA/LoRA with the LightOnOCR-2 base checkpoint.

Key settings:

• PEFT method: LoRA
• LoRA rank: 8
• LoRA alpha: 16
• LoRA dropout: 0.05
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Precision: bfloat16
• Quantization during training: 4-bit NF4
• Image preprocessing during training: longest edge 700 px
• Max sequence length during training: 2048
• Epochs: 5
• Effective batch size: 8

Prompt used during training:

Przepisz dokładnie tekst z obrazu. Zachowaj polskie znaki, interpunkcję i podział na wiersze.

Loading

import torch
from peft import PeftModel
from transformers import LightOnOcrForConditionalGeneration, LightOnOcrProcessor

base_model_id = "lightonai/LightOnOCR-2-1B-base"
adapter_id = "IHPAN/LightOnOCR-2-1B-htr-polish-1"

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

processor = LightOnOcrProcessor.from_pretrained(adapter_id)
base_model = LightOnOcrForConditionalGeneration.from_pretrained(
    base_model_id,
    torch_dtype=dtype,
).to(device)
model = PeftModel.from_pretrained(base_model, adapter_id).to(device)
model.eval()

Inference Example

from PIL import Image
import torch

image = Image.open("sample.jpg").convert("RGB")
prompt = (
    "Przepisz dokładnie tekst z obrazu. Zachowaj polskie znaki, "
    "interpunkcję i podział na wiersze."
)

messages = [
    {
        "role": "user",
        "content": [{"type": "image"}, {"type": "text", "text": prompt}],
    }
]

text = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=False,
)

inputs = processor(
    text=[text],
    images=[[image]],
    return_tensors="pt",
    padding=True,
    truncation=True,
    max_length=4096,
    size={"longest_edge": 1540},
)

if "pixel_values" in inputs:
    inputs["pixel_values"] = inputs["pixel_values"].to(dtype)
inputs = {key: value.to(device) for key, value in inputs.items()}

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=2048,
        do_sample=False,
    )

generated_ids = output_ids[0, inputs["input_ids"].shape[1]:]
transcription = processor.decode(generated_ids, skip_special_tokens=True)
print(transcription.strip())

Evaluation

Metric	Base Model	Fine-tuned Model
Character Error Rate (CER)	0.2591 (25.9%)	0.1067 (10.67%)
Word Error Rate (WER)	0.7654 (76.54%)	0.3614 (36.14%)

Output

Examples of manuscript transcription (excerpt from the death certificate of a victim of the German air raid on the town of Koło on 2 September 1939):

Source of the scan: Szukaj w archiwach

Base model output:

Anna olwudziestego piztego wrzeznia tyrige driejgéset brydries- 
tego driejgésego roku shawilisig, w haucelarii parapii Ym. Kreyia w Hole, 
petnoletui siadkowie: Jozef Rasaiejcyk, b. ekspedient pocetony ze dem 
i Stanislaw Gorrelaiejcyk, Pistonosz z Baskowa i oswiadcryli, ze 
Anna drugiego wrzeznia roku Gierjceego o godnim pystuastej zwort 
w Hole Marian-Jozef Matecki, ucei shisorshi, w wicen lat 
pistuascie, wierigcy driejgés, rauwierkary w Edunach, syn Sta- 
nistowa i Marianny z Szymanowskich wadzoukow Mateckich. 
Aka ten zostar stawajscym preceptany, przyjety i podpisany. 
Urednict Stann Cywilnego:

Fine-tuned model output:

Dnia dwudziestego piątego września dziewięć trzydzies-
tego dziewiątego roku stawili się, w kancelarii parafii Św. Krzyża w Koło,
pełnoletni świadkowie: Józef Ratajczyk, b. ekspedient pocztowy ze Zdun
i Stanisław Gorzelawiezyk, listonosz z Barzkowa i oświadczyli, że
dnia drugiego września roku bieżącego o godzinie piętnastej zmarł
w Koło Marian-Fózek Matecki, uczeń śląsarski, w wieku lat
piętnaście, miesięcy dziewięć, zamieszkały w Zdunach, syn Sta-
nisława i Marianny z Szymanowskich małżonków Mateckich. -
Akt ten został stawającym przeczytany, przyjęty i podpisany. -
Urzędnik Stanu Cywilnego:

The average quality of the text recognised was around 10% CER, but the model performed differently with different types of handwriting. The most typical and legible examples (such as the excerpt from the 1939 document shown above) achieved a CER of 4–5%, but other handwriting samples, such as those shown below, achieved a CER of only 13–19%.

Sources of the scans: Polona

Limitations

• The adapter is specialized for Polish handwritten material similar to the local fine-tuning set.
• It may underperform on printed documents, non-Polish texts, forms, tables, multi-column layouts, heavily degraded scans, or handwriting styles absent from the training data.
• Generated text can contain OCR hallucinations. Human review is recommended.
• This adapter depends on the base model license and usage terms.
• Manuscripts containing crossings-out often cause the model difficulty in deciphering them

Framework Versions

• PEFT 0.20.0
• Transformers 5.0.0
• PyTorch 2.13.0