Mazyod/jina-embeddings-v4-vllm-mv

Jina Embeddings v4 — multi-vector, multimodal, drop-in for vLLM

A self-contained, drop-in checkpoint that serves Jina Embeddings v4 multi-vector (128-dim/token, ColBERT-style late interaction) multimodal (text + image) embeddings from a stock vLLM OpenAI server. Send text or an image, get back per-token multivectors on /pooling.

This is jinaai/jina-embeddings-v4-vllm-retrieval with three things baked in, so serving needs no extra flags or side files:

1. the multi_vector_projector (128×2048 + bias), with the retrieval LoRA merged in — this head is not present in the upstream vLLM checkpoint;
2. architectures: ["JinaV4MultiVector"] in config.json;
3. the Jina image chat template (chat_template.json) and a max_pixels ceiling in preprocessor_config.json.

Requires the jina-v4-vllm-plugin in your vLLM image — it registers the JinaV4MultiVector model that applies the projection in-engine. Without the plugin this checkpoint will not load (the architecture is unknown to stock vLLM). Plugin + full docs: https://github.com/Mazyod/jina-embeddings-v4-vllm-plugin

Serve (validated against vLLM 0.22.0)

# 1) install the plugin into your vLLM image/env (pin a tag/commit for production)
pip install --no-deps \
  "git+https://github.com/Mazyod/jina-embeddings-v4-vllm-plugin@main#subdirectory=src/jinav4_vllm/vllm_plugin"

# 2) serve — no --hf-overrides, no --chat-template, no projector env var (all baked in)
vllm serve Mazyod/jina-embeddings-v4-vllm-mv \
  --runner pooling --pooler-config.task token_embed \
  --served-model-name jina-v4

Query — per-token multivectors on /pooling

/pooling returns the [n, 128] matrix (L2-normalized). /v1/embeddings only returns a single pooled vector, so use /pooling for multi-vector / late interaction.

import base64, requests, numpy as np
B = "http://localhost:8000"

# text (prefix queries "Query: ", passages "Passage: ")
t = requests.post(f"{B}/pooling", json={"model": "jina-v4", "input": ["Query: hello world"]}).json()
text_mv = np.array(t["data"][0]["data"])          # [n, 128]

# image (chat-style messages)
b64 = base64.b64encode(open("page.png", "rb").read()).decode()
msg = [{"role": "user", "content": [
    {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
    {"type": "text", "text": "Describe the image."}]}]
i = requests.post(f"{B}/pooling", json={"model": "jina-v4", "messages": msg}).json()
image_mv = np.array(i["data"][0]["data"])         # [m, 128]

def maxsim(q, d):                                  # ColBERT late interaction (vectors are L2-normalized)
    return float((q @ d.T).max(axis=1).sum())

Image fidelity

max_pixels is baked to 3,211,264 (≈ up to ~4096 image tokens). Raise/lower it per request with --mm-processor-kwargs '{"max_pixels": N}', or re-bake. Keep image-token counts within your --max-model-len.

Parity / validation

Matches canonical Jina v4 to bf16 precision — per-token cosine ≈ 0.999 (text), ≈ 0.992–0.997 (image) at the checkpoint's default resolution. Per-token agreement with HF-transformers drifts at high resolution due to image-preprocessing differences between the two stacks; this is irrelevant for a vLLM-only pipeline. Embed queries and documents through the same vLLM pipeline for consistent MaxSim. Details: docs/VALIDATION.md in the plugin repo.

License & attribution

This checkpoint is derived from jinaai/jina-embeddings-v4-vllm-retrieval (Jina Embeddings v4), which is itself derived from Qwen/Qwen2.5-VL-3B. It is governed by the Qwen Research License and redistributed here under those terms (see license_link). The multi_vector_projector was extracted from jinaai/jina-embeddings-v4 with the retrieval adapter merged. All credit for the model to Jina AI and the Qwen team; this repo only repackages their public weights for drop-in vLLM serving.