Sentence Similarity
sentence-transformers
Safetensors
English
modernbert
biencoder
text-classification
sentence-pair-classification
semantic-similarity
semantic-search
retrieval
reranking
Generated from Trainer
dataset_size:1451941
loss:MultipleNegativesRankingLoss
Eval Results
text-embeddings-inference
| language: | |
| - en | |
| license: apache-2.0 | |
| tags: | |
| - biencoder | |
| - sentence-transformers | |
| - text-classification | |
| - sentence-pair-classification | |
| - semantic-similarity | |
| - semantic-search | |
| - retrieval | |
| - reranking | |
| - generated_from_trainer | |
| - dataset_size:1047690 | |
| - loss:CoSENTLoss | |
| base_model: Alibaba-NLP/gte-modernbert-base | |
| widget: | |
| - source_sentence: In 2015 Adolf Hitler appeared in the kickstarter short movie `` | |
| Kung Fury `` as Taccone ( A.K.A . | |
| sentences: | |
| - In 2015 , Adolf Hitler appeared in the Kickstarter - short film `` Kung Fury `` | |
| as Taccone ( A.K.A . | |
| - In 1795 , the only white residents were Dr. John Laidley and two brothers with | |
| the surname Ainslie . | |
| - The 125th University Match was played in March 2014 at the Rye Golf Club , Oxford | |
| , East Sussex won the game 8.5 - 6.5 . | |
| - source_sentence: From 1973 to 1974 , Aubrey toured with the Cambridge Theatre Company | |
| as Diggory in `` She Stoops to Conquer `` and again as Aguecheek . | |
| sentences: | |
| - Oxide can be reduced to metallic samarium at higher temperatures by heating with | |
| a reducing agent such as hydrogen or carbon monoxide . | |
| - From 1973 to 1974 Aguecheek toured with the Cambridge Theatre Company as Diggory | |
| in `` You Stoops to Conquer `` and again as Aubrey . | |
| - The medals were presented by Barry Maister , IOC member , New Zealand and Sarah | |
| Webb Gosling , Vice President of World Sailing . | |
| - source_sentence: There is no official wall on the border , although there are sections | |
| of fence near populated areas and continuous border crossings . | |
| sentences: | |
| - The 2014 -- 15 Boston Bruins season was the 91st season for the National Hockey | |
| League franchise that was established on November 1 , 1924 . | |
| - He was trained by the Inghams and owned by John Hawkes . | |
| - There is no continuous wall on the border , although there are fence sections | |
| near populated areas and official border crossings . | |
| - source_sentence: Capital . `` The French established similar hill stations in Indochina | |
| , such as Dalat built in 1921 . | |
| sentences: | |
| - Lubuk China is a small town in Alor Gajah District , Melaka , Malaysia . It is | |
| situated near the border with Negeri Sembilan . | |
| - The French established similar hill stations in Indochina , such as Dalat , built | |
| in 1921 . | |
| - John Potts ( or Pott ) was a doctor and colonial governor of Virginia in the Jamestown | |
| settlement at Virginia Colony in the early 17th century . | |
| - source_sentence: The band pursued `` signals `` in January 2012 in three weeks , | |
| and drums were recorded in a day and a half . | |
| sentences: | |
| - It was repaired at the beginning of the 20th century and is listed as closed in | |
| our records . | |
| - The band tracked `` Signals `` in three weeks in January 2012 . Drums were recorded | |
| in a day and a half . | |
| - Contributors include actor Anton LaVey , Satanist Christopher Lee , serial killer | |
| expert Clive Barker , author Karen Greenlee , and necrophile Robert Ressler . | |
| datasets: | |
| - redis/langcache-sentencepairs-v1 | |
| pipeline_tag: sentence-similarity | |
| library_name: sentence-transformers | |
| metrics: | |
| - cosine_accuracy | |
| - cosine_accuracy_threshold | |
| - cosine_f1 | |
| - cosine_f1_threshold | |
| - cosine_precision | |
| - cosine_recall | |
| - cosine_ap | |
| - cosine_mcc | |
| model-index: | |
| - name: Redis fine-tuned BiEncoder model for semantic caching on LangCache | |
| results: | |
| - task: | |
| type: binary-classification | |
| name: Binary Classification | |
| dataset: | |
| name: val | |
| type: val | |
| metrics: | |
| - type: cosine_accuracy | |
| value: 0.762879238548483 | |
| name: Cosine Accuracy | |
| - type: cosine_accuracy_threshold | |
| value: 0.8641344308853149 | |
| name: Cosine Accuracy Threshold | |
| - type: cosine_f1 | |
| value: 0.6906413705224409 | |
| name: Cosine F1 | |
| - type: cosine_f1_threshold | |
| value: 0.826151430606842 | |
| name: Cosine F1 Threshold | |
| - type: cosine_precision | |
| value: 0.6289324394017535 | |
| name: Cosine Precision | |
| - type: cosine_recall | |
| value: 0.7657770800627943 | |
| name: Cosine Recall | |
| - type: cosine_ap | |
| value: 0.7350886848165957 | |
| name: Cosine Ap | |
| - type: cosine_mcc | |
| value: 0.47694835496637344 | |
| name: Cosine Mcc | |
| - task: | |
| type: binary-classification | |
| name: Binary Classification | |
| dataset: | |
| name: test | |
| type: test | |
| metrics: | |
| - type: cosine_accuracy | |
| value: 0.7035036519888425 | |
| name: Cosine Accuracy | |
| - type: cosine_accuracy_threshold | |
| value: 0.8520702719688416 | |
| name: Cosine Accuracy Threshold | |
| - type: cosine_f1 | |
| value: 0.7118695167174169 | |
| name: Cosine F1 | |
| - type: cosine_f1_threshold | |
| value: 0.8109757900238037 | |
| name: Cosine F1 Threshold | |
| - type: cosine_precision | |
| value: 0.597953808752026 | |
| name: Cosine Precision | |
| - type: cosine_recall | |
| value: 0.8794040968342645 | |
| name: Cosine Recall | |
| - type: cosine_ap | |
| value: 0.6473233550443912 | |
| name: Cosine Ap | |
| - type: cosine_mcc | |
| value: 0.4409362621742405 | |
| name: Cosine Mcc | |
| # Redis fine-tuned BiEncoder model for semantic caching on LangCache | |
| This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [Alibaba-NLP/gte-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-modernbert-base) on the [LangCache Sentence Pairs (all)](https://huggingface.co/datasets/redis/langcache-sentencepairs-v1) dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for sentence pair similarity. | |
| ## Model Details | |
| ### Model Description | |
| - **Model Type:** Sentence Transformer | |
| - **Base model:** [Alibaba-NLP/gte-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-modernbert-base) <!-- at revision e7f32e3c00f91d699e8c43b53106206bcc72bb22 --> | |
| - **Maximum Sequence Length:** 8192 tokens | |
| - **Output Dimensionality:** 768 dimensions | |
| - **Similarity Function:** Cosine Similarity | |
| - **Training Dataset:** | |
| - [LangCache Sentence Pairs (all)](https://huggingface.co/datasets/redis/langcache-sentencepairs-v1) | |
| - **Language:** en | |
| - **License:** apache-2.0 | |
| ### Model Sources | |
| - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) | |
| - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) | |
| - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) | |
| ### Full Model Architecture | |
| ``` | |
| SentenceTransformer( | |
| (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'ModernBertModel'}) | |
| (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) | |
| ) | |
| ``` | |
| ## Usage | |
| ### Direct Usage (Sentence Transformers) | |
| First install the Sentence Transformers library: | |
| ```bash | |
| pip install -U sentence-transformers | |
| ``` | |
| Then you can load this model and run inference. | |
| ```python | |
| from sentence_transformers import SentenceTransformer | |
| # Download from the 🤗 Hub | |
| model = SentenceTransformer("redis/langcache-embed-v3") | |
| # Run inference | |
| sentences = [ | |
| 'The band pursued `` signals `` in January 2012 in three weeks , and drums were recorded in a day and a half .', | |
| 'The band tracked `` Signals `` in three weeks in January 2012 . Drums were recorded in a day and a half .', | |
| 'Contributors include actor Anton LaVey , Satanist Christopher Lee , serial killer expert Clive Barker , author Karen Greenlee , and necrophile Robert Ressler .', | |
| ] | |
| embeddings = model.encode(sentences) | |
| print(embeddings.shape) | |
| # [3, 768] | |
| # Get the similarity scores for the embeddings | |
| similarities = model.similarity(embeddings, embeddings) | |
| print(similarities) | |
| # tensor([[1.0000, 0.9598, 0.4943], | |
| # [0.9598, 0.9998, 0.5096], | |
| # [0.4943, 0.5096, 1.0001]]) | |
| ``` | |
| <!-- | |
| ### Direct Usage (Transformers) | |
| <details><summary>Click to see the direct usage in Transformers</summary> | |
| </details> | |
| --> | |
| <!-- | |
| ### Downstream Usage (Sentence Transformers) | |
| You can finetune this model on your own dataset. | |
| <details><summary>Click to expand</summary> | |
| </details> | |
| --> | |
| <!-- | |
| ### Out-of-Scope Use | |
| *List how the model may foreseeably be misused and address what users ought not to do with the model.* | |
| --> | |
| ## Evaluation | |
| ### Metrics | |
| #### Binary Classification | |
| * Datasets: `val` and `test` | |
| * Evaluated with [<code>BinaryClassificationEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.BinaryClassificationEvaluator) | |
| | Metric | val | test | | |
| |:--------------------------|:-----------|:-----------| | |
| | cosine_accuracy | 0.7629 | 0.7035 | | |
| | cosine_accuracy_threshold | 0.8641 | 0.8521 | | |
| | cosine_f1 | 0.6906 | 0.7119 | | |
| | cosine_f1_threshold | 0.8262 | 0.811 | | |
| | cosine_precision | 0.6289 | 0.598 | | |
| | cosine_recall | 0.7658 | 0.8794 | | |
| | **cosine_ap** | **0.7351** | **0.6473** | | |
| | cosine_mcc | 0.4769 | 0.4409 | | |
| <!-- | |
| ## Bias, Risks and Limitations | |
| *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* | |
| --> | |
| <!-- | |
| ### Recommendations | |
| *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* | |
| --> | |
| ## Training Details | |
| ### Training Dataset | |
| #### LangCache Sentence Pairs (all) | |
| * Dataset: [LangCache Sentence Pairs (all)](https://huggingface.co/datasets/redis/langcache-sentencepairs-v1) | |
| * Size: 62,021 training samples | |
| * Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>label</code> | |
| * Approximate statistics based on the first 1000 samples: | |
| | | sentence1 | sentence2 | label | | |
| |:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:------------------------------------------------| | |
| | type | string | string | int | | |
| | details | <ul><li>min: 8 tokens</li><li>mean: 27.46 tokens</li><li>max: 53 tokens</li></ul> | <ul><li>min: 9 tokens</li><li>mean: 27.36 tokens</li><li>max: 52 tokens</li></ul> | <ul><li>0: ~50.30%</li><li>1: ~49.70%</li></ul> | | |
| * Samples: | |
| | sentence1 | sentence2 | label | | |
| |:--------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------|:---------------| | |
| | <code>The newer Punts are still very much in existence today and race in the same fleets as the older boats .</code> | <code>The newer punts are still very much in existence today and run in the same fleets as the older boats .</code> | <code>1</code> | | |
| | <code>Turner Valley , was at the Turner Valley Bar N Ranch Airport , southwest of the Turner Valley Bar N Ranch , Alberta , Canada .</code> | <code>Turner Valley Bar N Ranch Airport , , was located at Turner Valley Bar N Ranch , southwest of Turner Valley , Alberta , Canada .</code> | <code>0</code> | | |
| | <code>After losing his second election , he resigned as opposition leader and was replaced by Geoff Pearsall .</code> | <code>Max Bingham resigned as opposition leader after losing his second election , and was replaced by Geoff Pearsall .</code> | <code>1</code> | | |
| * Loss: [<code>CoSENTLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss) with these parameters: | |
| ```json | |
| { | |
| "scale": 20.0, | |
| "similarity_fct": "pairwise_cos_sim" | |
| } | |
| ``` | |
| ### Evaluation Dataset | |
| #### LangCache Sentence Pairs (all) | |
| * Dataset: [LangCache Sentence Pairs (all)](https://huggingface.co/datasets/redis/langcache-sentencepairs-v1) | |
| * Size: 62,021 evaluation samples | |
| * Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>label</code> | |
| * Approximate statistics based on the first 1000 samples: | |
| | | sentence1 | sentence2 | label | | |
| |:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:------------------------------------------------| | |
| | type | string | string | int | | |
| | details | <ul><li>min: 8 tokens</li><li>mean: 27.46 tokens</li><li>max: 53 tokens</li></ul> | <ul><li>min: 9 tokens</li><li>mean: 27.36 tokens</li><li>max: 52 tokens</li></ul> | <ul><li>0: ~50.30%</li><li>1: ~49.70%</li></ul> | | |
| * Samples: | |
| | sentence1 | sentence2 | label | | |
| |:--------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------|:---------------| | |
| | <code>The newer Punts are still very much in existence today and race in the same fleets as the older boats .</code> | <code>The newer punts are still very much in existence today and run in the same fleets as the older boats .</code> | <code>1</code> | | |
| | <code>Turner Valley , was at the Turner Valley Bar N Ranch Airport , southwest of the Turner Valley Bar N Ranch , Alberta , Canada .</code> | <code>Turner Valley Bar N Ranch Airport , , was located at Turner Valley Bar N Ranch , southwest of Turner Valley , Alberta , Canada .</code> | <code>0</code> | | |
| | <code>After losing his second election , he resigned as opposition leader and was replaced by Geoff Pearsall .</code> | <code>Max Bingham resigned as opposition leader after losing his second election , and was replaced by Geoff Pearsall .</code> | <code>1</code> | | |
| * Loss: [<code>CoSENTLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss) with these parameters: | |
| ```json | |
| { | |
| "scale": 20.0, | |
| "similarity_fct": "pairwise_cos_sim" | |
| } | |
| ``` | |
| ### Training Logs | |
| | Epoch | Step | val_cosine_ap | test_cosine_ap | | |
| |:-----:|:----:|:-------------:|:--------------:| | |
| | -1 | -1 | 0.7351 | 0.6473 | | |
| ### Framework Versions | |
| - Python: 3.12.3 | |
| - Sentence Transformers: 5.1.0 | |
| - Transformers: 4.56.0 | |
| - PyTorch: 2.8.0+cu128 | |
| - Accelerate: 1.10.1 | |
| - Datasets: 4.0.0 | |
| - Tokenizers: 0.22.0 | |
| ## Citation | |
| ### BibTeX | |
| #### Sentence Transformers | |
| ```bibtex | |
| @inproceedings{reimers-2019-sentence-bert, | |
| title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", | |
| author = "Reimers, Nils and Gurevych, Iryna", | |
| booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", | |
| month = "11", | |
| year = "2019", | |
| publisher = "Association for Computational Linguistics", | |
| url = "https://arxiv.org/abs/1908.10084", | |
| } | |
| ``` | |
| #### CoSENTLoss | |
| ```bibtex | |
| @online{kexuefm-8847, | |
| title={CoSENT: A more efficient sentence vector scheme than Sentence-BERT}, | |
| author={Su Jianlin}, | |
| year={2022}, | |
| month={Jan}, | |
| url={https://kexue.fm/archives/8847}, | |
| } | |
| ``` | |
| <!-- | |
| ## Glossary | |
| *Clearly define terms in order to be accessible across audiences.* | |
| --> | |
| <!-- | |
| ## Model Card Authors | |
| *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* | |
| --> | |
| <!-- | |
| ## Model Card Contact | |
| *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* | |
| --> |