2013khansohail/cartographer-ecommerce-reranker-MiniLM-L6-v2

Cartographer : A Lightweight E-Commerce Reranker

Cartographer is a small, fast cross-encoder that scores how well a shopping query matches a product. It takes a (query, product) pair and returns a single relevance score for reranking a candidate list. It is a drop-in upgrade for the general-purpose reranker it is built on: same size, same speed, measurably better on product search.

It was fine-tuned from cross-encoder/ms-marco-MiniLM-L-6-v2 on Amazon's public Shopping Queries Dataset (ESCI). The base model was trained on web-search passages; product listings are not web text, so a domain-tuned model of the same 22.7M-parameter footprint does meaningfully better.

Results

Evaluated on the held-out ESCI test split (US / English): 8,956 queries, 181,701 (query, product) pairs. Both models scored at max_length=256, identical settings : the comparison is apples-to-apples. The headline metric is nDCG (graded relevance), which is what the E/S/C/I labels are for.

Metric	Baseline (ms-marco-MiniLM-L6)	Cartographer	Δ abs	Δ rel
nDCG@5	0.6748	0.7272	+0.0524	+7.76%
nDCG@10	0.7193	0.7642	+0.0449	+6.24%
nDCG@20	0.8064	0.8354	+0.0290	+3.60%

The gain is largest at the top of the ranking (nDCG@5), which is where it matters most for product search : users see the first few results. There were no regressions: every reported cutoff improved.

Why it works: it learned the relevance gradation

ESCI labels relevance on four graded levels : Exact, Substitute, Complement, Irrelevant : and the model was trained to regress onto graded relevance, not a binary relevant/not flag. The effect is visible in how cleanly the model separates the four classes. Mean model score per true label on the test set:

True label	Baseline mean (± std)	Cartographer mean (± std)
E (Exact)	0.024 (± 6.14)	0.283 (± 1.27)
S (Substitute)	−2.228 (± 5.81)	−0.648 (± 1.02)
C (Complement)	−0.787 (± 6.09)	−0.781 (± 1.04)
I (Irrelevant)	−5.558 (± 5.93)	−1.140 (± 1.11)

The baseline's scores are noise-dominated : its per-class standard deviation (~6) is larger than the gaps between class means, so it barely separates the labels relative to its own spread. Cartographer collapses that noise to std ≈ 1 while producing a clean, monotonic E > S > C > I ordering. That tightening is the model learning to rank product relevance.

Intended use

A second-stage reranker: retrieve a candidate set of products with a fast first-stage system (BM25 or a bi-encoder), then rerank the top-k with Cartographer for sharper ordering. It is designed to be cheap to serve at query time, which is exactly why lightweight cross-encoders get deployed in production.

Usage

from sentence_transformers import CrossEncoder

model = CrossEncoder("2013khansohail/cartographer-ecommerce-reranker-MiniLM-L6-v2", max_length=256)

query = "wireless noise cancelling headphones"
products = [
    "Sony WH-1000XM5 Wireless Noise Cancelling Headphones",
    "Wired Earbuds with Microphone, 3.5mm",
    "Headphone Stand Aluminum Desk Holder",
    "Bluetooth Speaker Waterproof Portable",
]

scores = model.predict([(query, p) for p in products])
ranked = sorted(zip(products, scores), key=lambda x: x[1], reverse=True)
for product, score in ranked:
    print(f"{score:.3f}  {product}")

Training

• Base model: cross-encoder/ms-marco-MiniLM-L-6-v2 (22.7M params). Fine-tuning from a reranker checkpoint rather than a raw backbone converges faster and higher on a fixed dataset.
• Data: ESCI Shopping Queries, US/English, small_version split. Product text is the product title (title-only recipe for v1).
• Objective: pointwise regression (MSE with a sigmoid activation) onto graded relevance gains mapped as E=1.0, S=0.1, C=0.01, I=0.0 : Amazon's ranking-task convention. The gains and the model's sigmoid-bounded output share the same [0, 1] scale.
• Recipe: 2 epochs with early stopping on validation nDCG@10; best checkpoint loaded at the end. fp16, effective batch size 64 (16 × 4 gradient accumulation), lr 2e-5, on a single consumer GPU (NVIDIA RTX 3050, 4GB).

On training length (an honest finding): one epoch reaches ~99% of the final quality (val nDCG@10 ≈ 0.759). A second epoch with early stopping adds ~0.4 nDCG@10, the gain transfers cleanly to the sealed test set (not val overfitting), and it specifically resolves the Substitute-vs-Complement ordering that one epoch leaves tangled. So 2 epochs is the shipped recipe : but one epoch is a defensible, cheaper alternative.

Limitations

• Amazon distribution. ESCI is Amazon data, so the model inherits Amazon's product distribution and category mix. It may need further tuning to transfer to other catalogs.
• Substitute vs. Complement is the soft boundary. The model orders S above C directionally, but S and C remain the closest pair and their score distributions overlap (std ≈ 1, means ~0.13 apart). This is expected: the gain mapping places both near zero, so MSE has little incentive to push them apart. If distinguishing substitutes from complements is critical for your use case, a different gain mapping or a ranking loss is the place to look.
• Lightweight ceiling. This is the best model in its weight class, not the best model available. Some hard queries will need a heavier reranker. The pitch is "drop-in upgrade at the same cost," not "state of the art."
• English-only, title-only product text in this version.

Citation

Built on the Amazon Shopping Queries Dataset (ESCI):

Reddy et al., Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search (2022). https://github.com/amazon-science/esci-data

Base model: cross-encoder/ms-marco-MiniLM-L-6-v2.

---

Training code, data prep, and full evaluation (reproducible end to end): github.com/sohail000/cartographer-reranker