xhresko/nomic-embed-vision-v1.5-patched

nomic-embed-vision-v1.5: Expanding the Latent Space

This is a locally patched copy of nomic-ai/nomic-embed-vision-v1.5. See Local Modifications below.

Local Modifications

The upstream model was published for transformers 4.x and fails to load under transformers 5.x (tested with 5.13.1). The following changes were made on 2026-07-14 to make it work without downgrading/pinning transformers. We deliberately patch the model in-repo rather than pin dependencies, since no further upstream updates to the remote code are expected.

1. Vendored remote code

Upstream, auto_map in config.json points to code hosted on the nomic-ai/nomic-bert-2048 Hub repo, which transformers downloads at load time. That code is incompatible with transformers 5.x, so configuration_hf_nomic_bert.py and modeling_hf_nomic_bert.py were copied into this directory and auto_map now references the local files:

"auto_map": {
  "AutoConfig": "configuration_hf_nomic_bert.NomicBertConfig",
  "AutoModel": "modeling_hf_nomic_bert.NomicVisionModel"
}

Consequence: updates to the upstream nomic-bert-2048 code are no longer picked up automatically.

2. post_init() call in NomicVisionModel (transformers 5.x compatibility)

transformers 5.x requires PreTrainedModel.post_init() to run during model construction — among other things it sets all_tied_weights_keys, which the weight-loading machinery accesses. The upstream NomicVisionModel.__init__ never calls it (the text-model classes in the same file do), causing:

AttributeError: 'NomicVisionModel' object has no attribute 'all_tied_weights_keys'

Fix: self.post_init() added at the end of NomicVisionModel.__init__ in modeling_hf_nomic_bert.py.

3. Recompute non-persistent buffers after loading (transformers 5.x compatibility)

transformers 5.x constructs models on the meta device, so any buffer values computed in a module's __init__ are discarded. The nomic code computes several non-persistent buffers there (norm_factor in the attention modules, the rotary pos_embed/bands in NomicVisionRotaryEmbeddingCat, inv_freq/scale in NomicBertRotaryEmbedding). These are not stored in the checkpoint, so after loading they contained uninitialized garbage — norm_factor was 0.0, making attention divide by zero and producing all-NaN embeddings (the model loaded without any error).

Fix in modeling_hf_nomic_bert.py:

• Added a module-level helper _recompute_nonpersistent_buffers(module) that recomputes these buffers with the same formulas the __init__ methods use.
• Wired it into _init_weights on both NomicBertPreTrainedModel and NomicVisionPreTrainedModel — the transformers 5 loading machinery calls _init_weights for modules whose tensors were not found in the checkpoint. Both classes need the override because NomicBertBlock subclasses NomicBertPreTrainedModel, and transformers dispatches initialization inside each PreTrainedModel subtree to that class's own _init_weights (an override only on the vision top-level class never reaches the attention layers inside the blocks).
• NomicVisionRotaryEmbeddingCat.__init__ now stores self.linear_bands, needed for the recomputation.

Verified: embeddings produced by the patched PyTorch model match the upstream ONNX export (onnx/model.onnx) with cosine similarity 1.0 (max element diff ~2e-7).

4. n_inner type fix in config.json

Upstream ships "n_inner": 2048.0 (a float). Newer transformers/huggingface_hub strictly validate config field types and require int or None, raising:

StrictDataclassFieldValidationError: Validation error for field 'n_inner'

Fix: changed to "n_inner": 2048.

Notes

• model.safetensors is stored via git-lfs; after cloning, run git lfs pull in this directory to fetch the actual weights (a 134-byte pointer file will otherwise cause SafetensorError: header too large).
• The modeling code requires the einops package at runtime.

---

Quick Start

Blog | Technical Report | AWS SageMaker | Atlas Embedding and Unstructured Data Analytics Platform

nomic-embed-vision-v1.5 is a high performing vision embedding model that shares the same embedding space as nomic-embed-text-v1.5.

All Nomic Embed Text models are now multimodal!

Name	Imagenet 0-shot	Datacomp (Avg. 38)	MTEB
nomic-embed-vision-v1.5	71.0	56.8	62.28
nomic-embed-vision-v1	70.7	56.7	62.39
OpenAI CLIP ViT B/16	68.3	56.3	43.82
Jina CLIP v1	59.1	52.2	60.1

Hosted Inference API

The easiest way to get started with Nomic Embed is through the Nomic Embedding API.

Generating embeddings with the nomic Python client is as easy as

from nomic import embed
import numpy as np

output = embed.image(
    images=[
        "image_path_1.jpeg",
        "image_path_2.png",
    ],
    model='nomic-embed-vision-v1.5',
)

print(output['usage'])
embeddings = np.array(output['embeddings'])
print(embeddings.shape)

For more information, see the API reference

Data Visualization

Click the Nomic Atlas map below to visualize a 100,000 sample CC3M comparing the Vision and Text Embedding Space!

Training Details

We align our vision embedder to the text embedding by employing a technique similar to LiT but instead lock the text embedder!

For more details, see the Nomic Embed Vision Technical Report (soon to be released!) and corresponding blog post

Training code is released in the contrastors repository

Usage

Remember nomic-embed-text requires prefixes and so, when using Nomic Embed in multimodal RAG scenarios (e.g. text to image retrieval), you should use the search_query: prefix.

Transformers

import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel, AutoImageProcessor
from PIL import Image
import requests

processor = AutoImageProcessor.from_pretrained("xhresko/nomic-embed-vision-v1.5-patched")
vision_model = AutoModel.from_pretrained("xhresko/nomic-embed-vision-v1.5-patched", trust_remote_code=True)

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(image, return_tensors="pt")

img_emb = vision_model(**inputs).last_hidden_state
img_embeddings = F.normalize(img_emb[:, 0], p=2, dim=1)

Additionally, you can perform multimodal retrieval!

def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0]
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

sentences = ['search_query: What are cute animals to cuddle with?', 'search_query: What do cats look like?']

tokenizer = AutoTokenizer.from_pretrained('nomic-ai/nomic-embed-text-v1.5')
text_model = AutoModel.from_pretrained('nomic-ai/nomic-embed-text-v1.5', trust_remote_code=True)
text_model.eval()

encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

with torch.no_grad():
    model_output = text_model(**encoded_input)

text_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
text_embeddings = F.layer_norm(text_embeddings, normalized_shape=(text_embeddings.shape[1],))
text_embeddings = F.normalize(text_embeddings, p=2, dim=1)

print(torch.matmul(img_embeddings, text_embeddings.T))