Add new SentenceTransformer model

README.md

@@ -12,132 +12,96 @@ tags:
 - retrieval
 - reranking
 - generated_from_trainer
-- dataset_size:1056095
-- loss:CoSENTLoss
+- dataset_size:483820
+- loss:CachedMultipleNegativesSymmetricRankingLoss
 base_model: Alibaba-NLP/gte-modernbert-base
 widget:
-- source_sentence: In 2015 Adolf Hitler appeared in the kickstarter short movie ``
-    Kung Fury `` as Taccone ( A.K.A .
+- source_sentence: 'See Precambrian time scale # Proposed Geologic timeline for another
+    set of periods 4600 -- 541 MYA .'
   sentences:
-  - In 2015 , Adolf Hitler appeared in the Kickstarter - short film `` Kung Fury ``
-    as Taccone ( A.K.A .
-  - In 1795 , the only white residents were Dr. John Laidley and two brothers with
-    the surname Ainslie .
-  - The 125th University Match was played in March 2014 at the Rye Golf Club , Oxford
-    , East Sussex won the game 8.5 - 6.5 .
-- source_sentence: From 1973 to 1974 , Aubrey toured with the Cambridge Theatre Company
-    as Diggory in `` She Stoops to Conquer `` and again as Aguecheek .
+  - In 2014 election , Biju Janata Dal candidate Tathagat Satapathy Bharatiya Janata
+    party candidate Rudra Narayan Pany defeated with a margin of 1.37,340 votes .
+  - In Scotland , the Strathclyde Partnership for Transport , formerly known as Strathclyde
+    Passenger Transport Executive , comprises the former Strathclyde region , which
+    includes the urban area around Glasgow .
+  - 'See Precambrian Time Scale # Proposed Geological Timeline for another set of
+    periods of 4600 -- 541 MYA .'
+- source_sentence: It is also 5 kilometers northeast of Tamaqua , 27 miles south of
+    Allentown and 9 miles northwest of Hazleton .
   sentences:
-  - Oxide can be reduced to metallic samarium at higher temperatures by heating with
-    a reducing agent such as hydrogen or carbon monoxide .
-  - From 1973 to 1974 Aguecheek toured with the Cambridge Theatre Company as Diggory
-    in `` You Stoops to Conquer `` and again as Aubrey .
-  - The medals were presented by Barry Maister , IOC member , New Zealand and Sarah
-    Webb Gosling , Vice President of World Sailing .
-- source_sentence: There is no official wall on the border , although there are sections
-    of fence near populated areas and continuous border crossings .
+  - In 1948 he moved to Massachusetts , and eventually settled in Vermont .
+  - Suddenly I remembered that I was a New Zealander , I caught the first plane home
+    and came back .
+  - It is also 5 miles northeast of Tamaqua , 27 miles south of Allentown , and 9
+    miles northwest of Hazleton .
+- source_sentence: The party has a Member of Parliament , a member of the House of
+    Lords , three members of the London Assembly and two Members of the European Parliament
+    .
   sentences:
-  - The 2014 -- 15 Boston Bruins season was the 91st season for the National Hockey
-    League franchise that was established on November 1 , 1924 .
-  - He was trained by the Inghams and owned by John Hawkes .
-  - There is no continuous wall on the border , although there are fence sections
-    near populated areas and official border crossings .
-- source_sentence: Capital . `` The French established similar hill stations in Indochina
-    , such as Dalat built in 1921 .
+  - The party has one Member of Parliament , one member of the House of Lords , three
+    Members of the London Assembly and two Members of the European Parliament .
+  - Grapsid crabs dominate in Australia , Malaysia and Panama , while gastropods Cerithidea
+    scalariformis and Melampus coeffeus are important seed predators in Florida mangroves
+    .
+  - Music Story is a music service website and international music data provider that
+    curates , aggregates and analyses metadata for digital music services .
+- source_sentence: 'The play received two 1969 Tony Award nominations : Best Actress
+    in a Play ( Michael Annals ) and Best Costume Design ( Charlotte Rae ) .'
   sentences:
-  - Lubuk China is a small town in Alor Gajah District , Melaka , Malaysia . It is
-    situated near the border with Negeri Sembilan .
-  - The French established similar hill stations in Indochina , such as Dalat , built
-    in 1921 .
-  - John Potts ( or Pott ) was a doctor and colonial governor of Virginia in the Jamestown
-    settlement at Virginia Colony in the early 17th century .
-- source_sentence: The band pursued `` signals `` in January 2012 in three weeks ,
-    and drums were recorded in a day and a half .
+  - Ravishanker is a fellow of the International Statistical Institute and an elected
+    member of the American Statistical Association .
+  - 'In 1969 , the play received two Tony - Award nominations : Best Actress in a
+    Theatre Play ( Michael Annals ) and Best Costume Design ( Charlotte Rae ) .'
+  - AMD and Nvidia both have proprietary methods of scaling , CrossFireX for AMD ,
+    and SLI for Nvidia .
+- source_sentence: He was a close friend of Ángel Cabrera and is a cousin of golfer
+    Tony Croatto .
   sentences:
-  - It was repaired at the beginning of the 20th century and is listed as closed in
-    our records .
-  - The band tracked `` Signals `` in three weeks in January 2012 . Drums were recorded
-    in a day and a half .
-  - Contributors include actor Anton LaVey , Satanist Christopher Lee , serial killer
-    expert Clive Barker , author Karen Greenlee , and necrophile Robert Ressler .
+  - He was a close friend of Ángel Cabrera , and is a cousin of golfer Tony Croatto
+    .
+  - Eugenijus Bartulis ( born December 7 , 1949 in Kaunas ) is a Lithuanian Roman
+    Catholic priest , and Bishop of Šiauliai .
+  - UWIRE also distributes its members content to professional media outlets , including
+    Yahoo , CNN and CBS News .
 datasets:
 - redis/langcache-sentencepairs-v1
 pipeline_tag: sentence-similarity
 library_name: sentence-transformers
 metrics:
-- cosine_accuracy
-- cosine_accuracy_threshold
-- cosine_f1
-- cosine_f1_threshold
-- cosine_precision
-- cosine_recall
-- cosine_ap
-- cosine_mcc
+- cosine_accuracy@1
+- cosine_precision@1
+- cosine_recall@1
+- cosine_ndcg@10
+- cosine_mrr@1
+- cosine_map@100
 model-index:
 - name: Redis fine-tuned BiEncoder model for semantic caching on LangCache
   results:
   - task:
-      type: binary-classification
-      name: Binary Classification
+      type: information-retrieval
+      name: Information Retrieval
     dataset:
-      name: val
-      type: val
+      name: train
+      type: train
     metrics:
-    - type: cosine_accuracy
-      value: 0.7629982153480072
-      name: Cosine Accuracy
-    - type: cosine_accuracy_threshold
-      value: 0.8639795780181885
-      name: Cosine Accuracy Threshold
-    - type: cosine_f1
-      value: 0.6907391673746814
-      name: Cosine F1
-    - type: cosine_f1_threshold
-      value: 0.8261561989784241
-      name: Cosine F1 Threshold
-    - type: cosine_precision
-      value: 0.6290946608202218
-      name: Cosine Precision
-    - type: cosine_recall
-      value: 0.7657770800627943
-      name: Cosine Recall
-    - type: cosine_ap
-      value: 0.7350867007914639
-      name: Cosine Ap
-    - type: cosine_mcc
-      value: 0.47714361581572273
-      name: Cosine Mcc
-  - task:
-      type: binary-classification
-      name: Binary Classification
-    dataset:
-      name: test
-      type: test
-    metrics:
-    - type: cosine_accuracy
-      value: 0.7034875284177939
-      name: Cosine Accuracy
-    - type: cosine_accuracy_threshold
-      value: 0.8523406982421875
-      name: Cosine Accuracy Threshold
-    - type: cosine_f1
-      value: 0.7118695167174169
-      name: Cosine F1
-    - type: cosine_f1_threshold
-      value: 0.8109798431396484
-      name: Cosine F1 Threshold
-    - type: cosine_precision
-      value: 0.597953808752026
-      name: Cosine Precision
-    - type: cosine_recall
-      value: 0.8794040968342645
-      name: Cosine Recall
-    - type: cosine_ap
-      value: 0.6473629818920917
-      name: Cosine Ap
-    - type: cosine_mcc
-      value: 0.4409362621742405
-      name: Cosine Mcc
+    - type: cosine_accuracy@1
+      value: 0.5579129681749296
+      name: Cosine Accuracy@1
+    - type: cosine_precision@1
+      value: 0.5579129681749296
+      name: Cosine Precision@1
+    - type: cosine_recall@1
+      value: 0.5359784831006956
+      name: Cosine Recall@1
+    - type: cosine_ndcg@10
+      value: 0.7522148521266401
+      name: Cosine Ndcg@10
+    - type: cosine_mrr@1
+      value: 0.5579129681749296
+      name: Cosine Mrr@1
+    - type: cosine_map@100
+      value: 0.6974638651409195
+      name: Cosine Map@100
 ---
 
 # Redis fine-tuned BiEncoder model for semantic caching on LangCache
@@ -149,7 +113,7 @@ This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [A
 ### Model Description
 - **Model Type:** Sentence Transformer
 - **Base model:** [Alibaba-NLP/gte-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-modernbert-base) <!-- at revision e7f32e3c00f91d699e8c43b53106206bcc72bb22 -->
-- **Maximum Sequence Length:** 8192 tokens
+- **Maximum Sequence Length:** 100 tokens
 - **Output Dimensionality:** 768 dimensions
 - **Similarity Function:** Cosine Similarity
 - **Training Dataset:**
@@ -167,7 +131,7 @@ This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [A
 
 ```
 SentenceTransformer(
-  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
+  (0): Transformer({'max_seq_length': 100, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
   (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
 )
 ```
@@ -190,9 +154,9 @@ from sentence_transformers import SentenceTransformer
 model = SentenceTransformer("redis/langcache-embed-v3")
 # Run inference
 sentences = [
-    'The band pursued `` signals `` in January 2012 in three weeks , and drums were recorded in a day and a half .',
-    'The band tracked `` Signals `` in three weeks in January 2012 . Drums were recorded in a day and a half .',
-    'Contributors include actor Anton LaVey , Satanist Christopher Lee , serial killer expert Clive Barker , author Karen Greenlee , and necrophile Robert Ressler .',
+    'He was a close friend of Ángel Cabrera and is a cousin of golfer Tony Croatto .',
+    'He was a close friend of Ángel Cabrera , and is a cousin of golfer Tony Croatto .',
+    'UWIRE also distributes its members content to professional media outlets , including Yahoo , CNN and CBS News .',
 ]
 embeddings = model.encode(sentences)
 print(embeddings.shape)
@@ -201,9 +165,9 @@ print(embeddings.shape)
 # Get the similarity scores for the embeddings
 similarities = model.similarity(embeddings, embeddings)
 print(similarities)
-# tensor([[0.9999, 0.9598, 0.4944],
-#         [0.9598, 0.9999, 0.5096],
-#         [0.4944, 0.5096, 0.9999]])
+# tensor([[0.9922, 0.9922, 0.5352],
+#         [0.9922, 0.9961, 0.5391],
+#         [0.5352, 0.5391, 1.0000]], dtype=torch.bfloat16)
 ```
 
 <!--
@@ -234,21 +198,19 @@ You can finetune this model on your own dataset.
 
 ### Metrics
 
-#### Binary Classification
+#### Information Retrieval
 
-* Datasets: `val` and `test`
-* Evaluated with [<code>BinaryClassificationEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.BinaryClassificationEvaluator)
+* Dataset: `train`
+* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
 
-| Metric                    | val        | test       |
-|:--------------------------|:-----------|:-----------|
-| cosine_accuracy           | 0.763      | 0.7035     |
-| cosine_accuracy_threshold | 0.864      | 0.8523     |
-| cosine_f1                 | 0.6907     | 0.7119     |
-| cosine_f1_threshold       | 0.8262     | 0.811      |
-| cosine_precision          | 0.6291     | 0.598      |
-| cosine_recall             | 0.7658     | 0.8794     |
-| **cosine_ap**             | **0.7351** | **0.6474** |
-| cosine_mcc                | 0.4771     | 0.4409     |
+| Metric             | Value      |
+|:-------------------|:-----------|
+| cosine_accuracy@1  | 0.5579     |
+| cosine_precision@1 | 0.5579     |
+| cosine_recall@1    | 0.536      |
+| **cosine_ndcg@10** | **0.7522** |
+| cosine_mrr@1       | 0.5579     |
+| cosine_map@100     | 0.6975     |
 
 <!--
 ## Bias, Risks and Limitations
@@ -269,24 +231,26 @@ You can finetune this model on your own dataset.
 #### LangCache Sentence Pairs (all)
 
 * Dataset: [LangCache Sentence Pairs (all)](https://huggingface.co/datasets/redis/langcache-sentencepairs-v1)
-* Size: 62,021 training samples
+* Size: 26,850 training samples
 * Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>label</code>
 * Approximate statistics based on the first 1000 samples:
-  |         | sentence1                                                                         | sentence2                                                                         | label                                           |
-  |:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:------------------------------------------------|
-  | type    | string                                                                            | string                                                                            | int                                             |
-  | details | <ul><li>min: 8 tokens</li><li>mean: 27.46 tokens</li><li>max: 53 tokens</li></ul> | <ul><li>min: 9 tokens</li><li>mean: 27.36 tokens</li><li>max: 52 tokens</li></ul> | <ul><li>0: ~50.30%</li><li>1: ~49.70%</li></ul> |
+  |         | sentence1                                                                         | sentence2                                                                         | label                        |
+  |:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:-----------------------------|
+  | type    | string                                                                            | string                                                                            | int                          |
+  | details | <ul><li>min: 8 tokens</li><li>mean: 27.35 tokens</li><li>max: 53 tokens</li></ul> | <ul><li>min: 8 tokens</li><li>mean: 27.27 tokens</li><li>max: 52 tokens</li></ul> | <ul><li>1: 100.00%</li></ul> |
 * Samples:
-  | sentence1                                                                                                                                   | sentence2                                                                                                                                     | label          |
-  |:--------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------|:---------------|
-  | <code>The newer Punts are still very much in existence today and race in the same fleets as the older boats .</code>                        | <code>The newer punts are still very much in existence today and run in the same fleets as the older boats .</code>                           | <code>1</code> |
-  | <code>Turner Valley , was at the Turner Valley Bar N Ranch Airport , southwest of the Turner Valley Bar N Ranch , Alberta , Canada .</code> | <code>Turner Valley Bar N Ranch Airport , , was located at Turner Valley Bar N Ranch , southwest of Turner Valley , Alberta , Canada .</code> | <code>0</code> |
-  | <code>After losing his second election , he resigned as opposition leader and was replaced by Geoff Pearsall .</code>                       | <code>Max Bingham resigned as opposition leader after losing his second election , and was replaced by Geoff Pearsall .</code>                | <code>1</code> |
-* Loss: [<code>CoSENTLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss) with these parameters:
+  | sentence1                                                                                                             | sentence2                                                                                                                      | label          |
+  |:----------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------|:---------------|
+  | <code>The newer Punts are still very much in existence today and race in the same fleets as the older boats .</code>  | <code>The newer punts are still very much in existence today and run in the same fleets as the older boats .</code>            | <code>1</code> |
+  | <code>After losing his second election , he resigned as opposition leader and was replaced by Geoff Pearsall .</code> | <code>Max Bingham resigned as opposition leader after losing his second election , and was replaced by Geoff Pearsall .</code> | <code>1</code> |
+  | <code>The 12F was officially homologated on August 21 , 1929 and exhibited at the Paris Salon in 1930 .</code>        | <code>The 12F was officially homologated on 21 August 1929 and displayed at the 1930 Paris Salon .</code>                      | <code>1</code> |
+* Loss: [<code>CachedMultipleNegativesSymmetricRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedmultiplenegativessymmetricrankingloss) with these parameters:
   ```json
   {
       "scale": 20.0,
-      "similarity_fct": "pairwise_cos_sim"
+      "similarity_fct": "cos_sim",
+      "mini_batch_size": 128,
+      "gather_across_devices": false
   }
   ```
 
@@ -295,31 +259,33 @@ You can finetune this model on your own dataset.
 #### LangCache Sentence Pairs (all)
 
 * Dataset: [LangCache Sentence Pairs (all)](https://huggingface.co/datasets/redis/langcache-sentencepairs-v1)
-* Size: 62,021 evaluation samples
+* Size: 26,850 evaluation samples
 * Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>label</code>
 * Approximate statistics based on the first 1000 samples:
-  |         | sentence1                                                                         | sentence2                                                                         | label                                           |
-  |:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:------------------------------------------------|
-  | type    | string                                                                            | string                                                                            | int                                             |
-  | details | <ul><li>min: 8 tokens</li><li>mean: 27.46 tokens</li><li>max: 53 tokens</li></ul> | <ul><li>min: 9 tokens</li><li>mean: 27.36 tokens</li><li>max: 52 tokens</li></ul> | <ul><li>0: ~50.30%</li><li>1: ~49.70%</li></ul> |
+  |         | sentence1                                                                         | sentence2                                                                         | label                        |
+  |:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:-----------------------------|
+  | type    | string                                                                            | string                                                                            | int                          |
+  | details | <ul><li>min: 8 tokens</li><li>mean: 27.35 tokens</li><li>max: 53 tokens</li></ul> | <ul><li>min: 8 tokens</li><li>mean: 27.27 tokens</li><li>max: 52 tokens</li></ul> | <ul><li>1: 100.00%</li></ul> |
 * Samples:
-  | sentence1                                                                                                                                   | sentence2                                                                                                                                     | label          |
-  |:--------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------|:---------------|
-  | <code>The newer Punts are still very much in existence today and race in the same fleets as the older boats .</code>                        | <code>The newer punts are still very much in existence today and run in the same fleets as the older boats .</code>                           | <code>1</code> |
-  | <code>Turner Valley , was at the Turner Valley Bar N Ranch Airport , southwest of the Turner Valley Bar N Ranch , Alberta , Canada .</code> | <code>Turner Valley Bar N Ranch Airport , , was located at Turner Valley Bar N Ranch , southwest of Turner Valley , Alberta , Canada .</code> | <code>0</code> |
-  | <code>After losing his second election , he resigned as opposition leader and was replaced by Geoff Pearsall .</code>                       | <code>Max Bingham resigned as opposition leader after losing his second election , and was replaced by Geoff Pearsall .</code>                | <code>1</code> |
-* Loss: [<code>CoSENTLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss) with these parameters:
+  | sentence1                                                                                                             | sentence2                                                                                                                      | label          |
+  |:----------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------|:---------------|
+  | <code>The newer Punts are still very much in existence today and race in the same fleets as the older boats .</code>  | <code>The newer punts are still very much in existence today and run in the same fleets as the older boats .</code>            | <code>1</code> |
+  | <code>After losing his second election , he resigned as opposition leader and was replaced by Geoff Pearsall .</code> | <code>Max Bingham resigned as opposition leader after losing his second election , and was replaced by Geoff Pearsall .</code> | <code>1</code> |
+  | <code>The 12F was officially homologated on August 21 , 1929 and exhibited at the Paris Salon in 1930 .</code>        | <code>The 12F was officially homologated on 21 August 1929 and displayed at the 1930 Paris Salon .</code>                      | <code>1</code> |
+* Loss: [<code>CachedMultipleNegativesSymmetricRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedmultiplenegativessymmetricrankingloss) with these parameters:
   ```json
   {
       "scale": 20.0,
-      "similarity_fct": "pairwise_cos_sim"
+      "similarity_fct": "cos_sim",
+      "mini_batch_size": 128,
+      "gather_across_devices": false
   }
   ```
 
 ### Training Logs
-| Epoch | Step | val_cosine_ap | test_cosine_ap |
-|:-----:|:----:|:-------------:|:--------------:|
-| -1    | -1   | 0.7351        | 0.6474         |
+| Epoch | Step | train_cosine_ndcg@10 |
+|:-----:|:----:|:--------------------:|
+| -1    | -1   | 0.7522               |
 
 
 ### Framework Versions
@@ -348,17 +314,6 @@ You can finetune this model on your own dataset.
 }
 ```
 
-#### CoSENTLoss
-```bibtex
-@online{kexuefm-8847,
-    title={CoSENT: A more efficient sentence vector scheme than Sentence-BERT},
-    author={Su Jianlin},
-    year={2022},
-    month={Jan},
-    url={https://kexue.fm/archives/8847},
-}
-```
-
 <!--
 ## Glossary
 

config.json

@@ -12,7 +12,7 @@
   "cls_token_id": 50281,
   "decoder_bias": true,
   "deterministic_flash_attn": false,
-  "dtype": "float32",
+  "dtype": "bfloat16",
   "embedding_dropout": 0.0,
   "eos_token_id": 50282,
   "global_attn_every_n_layers": 3,

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0f9247027e7d57e8b36440b5b3d10a785ded92c7c9f4a313ff7f54a549967290
-size 596070136
+oid sha256:95d02211c4cca89113f9f3e93ed91f5176bf50170faa2cb835f7bfea15bb9dd2
+size 298041696

sentence_bert_config.json

@@ -1,4 +1,4 @@
 {
-    "max_seq_length": 8192,
+    "max_seq_length": 100,
     "do_lower_case": false
 }

tokenizer.json

@@ -2,7 +2,7 @@
   "version": "1.0",
   "truncation": {
     "direction": "Right",
-    "max_length": 8192,
+    "max_length": 100,
     "strategy": "LongestFirst",
     "stride": 0
   },

tokenizer_config.json

@@ -933,6 +933,7 @@
   "cls_token": "[CLS]",
   "extra_special_tokens": {},
   "mask_token": "[MASK]",
+  "max_length": 100,
   "model_input_names": [
     "input_ids",
     "attention_mask"