metadata

language:
  - en
license: apache-2.0
tags:
  - biencoder
  - sentence-transformers
  - text-classification
  - sentence-pair-classification
  - semantic-similarity
  - semantic-search
  - retrieval
  - reranking
  - generated_from_trainer
  - dataset_size:1451941
  - loss:MultipleNegativesRankingLoss
base_model: Alibaba-NLP/gte-modernbert-base
datasets:
  - redis/langcache-sentencepairs-v1
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
  - cosine_accuracy@1
  - cosine_precision@1
  - cosine_recall@1
  - cosine_ndcg@10
  - cosine_mrr@1
  - cosine_map@100
model-index:
  - name: Redis fine-tuned BiEncoder model for semantic caching on LangCache
    results:
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: train
          type: train
        metrics:
          - type: cosine_accuracy@1
            value: 0.5579129681749296
            name: Cosine Accuracy@1
          - type: cosine_precision@1
            value: 0.5579129681749296
            name: Cosine Precision@1
          - type: cosine_recall@1
            value: 0.5359784831006956
            name: Cosine Recall@1
          - type: cosine_ndcg@10
            value: 0.7522148521266401
            name: Cosine Ndcg@10
          - type: cosine_mrr@1
            value: 0.5579129681749296
            name: Cosine Mrr@1
          - type: cosine_map@100
            value: 0.6974638651409195
            name: Cosine Map@100

Redis fine-tuned BiEncoder model for semantic caching on LangCache

This is a sentence-transformers model finetuned from Alibaba-NLP/gte-modernbert-base on the LangCache Sentence Pairs (all) dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for sentence pair similarity.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: Alibaba-NLP/gte-modernbert-base
Maximum Sequence Length: 100 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity
Training Dataset:
- LangCache Sentence Pairs (all)
Language: en
License: apache-2.0

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 100, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("redis/langcache-embed-v3")
# Run inference
sentences = [
    'The weather is lovely today.',
    "It's so sunny outside!",
    'He drove to the stadium.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[0.9922, 0.7891, 0.4629],
#         [0.7891, 1.0000, 0.5117],
#         [0.4629, 0.5117, 1.0000]], dtype=torch.bfloat16)

Evaluation

Metrics

Information Retrieval

Dataset: train
Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.5579
cosine_precision@1	0.5579
cosine_recall@1	0.536
cosine_ndcg@10	0.7522
cosine_mrr@1	0.5579
cosine_map@100	0.6975

Training Details

Training Dataset

LangCache Sentence Pairs (all)

Dataset: LangCache Sentence Pairs (all)
Size: 109,885 training samples
Columns: texts
Approximate statistics based on the first 1000 samples:
texts
type list
details
min: 3 elements
mean: 3.50 elements
max: 4 elements

	texts
type	list
details	min: 3 elements mean: 3.50 elements max: 4 elements

Samples:

texts
`['The newer Punts are still very much in existence today and race in the same fleets as the older boats .', 'The newer punts are still very much in existence today and run in the same fleets as the older boats .', 'how can I get financial freedom as soon as possible?']`
`['The newer punts are still very much in existence today and run in the same fleets as the older boats .', 'The newer Punts are still very much in existence today and race in the same fleets as the older boats .', 'The older Punts are still very much in existence today and race in the same fleets as the newer boats .']`
`['Turner Valley , was at the Turner Valley Bar N Ranch Airport , southwest of the Turner Valley Bar N Ranch , Alberta , Canada .', 'Turner Valley , , was located at Turner Valley Bar N Ranch Airport , southwest of Turner Valley Bar N Ranch , Alberta , Canada .', 'Turner Valley Bar N Ranch Airport , , was located at Turner Valley Bar N Ranch , southwest of Turner Valley , Alberta , Canada .']`

Loss: MultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "gather_across_devices": false
}

Evaluation Dataset

LangCache Sentence Pairs (all)

Dataset: LangCache Sentence Pairs (all)
Size: 109,885 evaluation samples
Columns: texts
Approximate statistics based on the first 1000 samples:
texts
type list
details
min: 3 elements
mean: 3.50 elements
max: 4 elements

	texts
type	list
details	min: 3 elements mean: 3.50 elements max: 4 elements

Samples:

texts
`['The newer Punts are still very much in existence today and race in the same fleets as the older boats .', 'The newer punts are still very much in existence today and run in the same fleets as the older boats .', 'how can I get financial freedom as soon as possible?']`
`['The newer punts are still very much in existence today and run in the same fleets as the older boats .', 'The newer Punts are still very much in existence today and race in the same fleets as the older boats .', 'The older Punts are still very much in existence today and race in the same fleets as the newer boats .']`
`['Turner Valley , was at the Turner Valley Bar N Ranch Airport , southwest of the Turner Valley Bar N Ranch , Alberta , Canada .', 'Turner Valley , , was located at Turner Valley Bar N Ranch Airport , southwest of Turner Valley Bar N Ranch , Alberta , Canada .', 'Turner Valley Bar N Ranch Airport , , was located at Turner Valley Bar N Ranch , southwest of Turner Valley , Alberta , Canada .']`

Loss: MultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "gather_across_devices": false
}

Training Logs

Epoch	Step	train_cosine_ndcg@10
-1	-1	0.7522

Framework Versions

Python: 3.12.3
Sentence Transformers: 5.1.0
Transformers: 4.56.0
PyTorch: 2.8.0+cu128
Accelerate: 1.10.1
Datasets: 4.0.0
Tokenizers: 0.22.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}