Add new SentenceTransformer model

README.md

@@ -12,96 +12,132 @@ tags:
 - retrieval
 - reranking
 - generated_from_trainer
-- dataset_size:483820
-- loss:CachedMultipleNegativesSymmetricRankingLoss
+- dataset_size:1056095
+- loss:CoSENTLoss
 base_model: Alibaba-NLP/gte-modernbert-base
 widget:
-- source_sentence: 'See Precambrian time scale # Proposed Geologic timeline for another
-    set of periods 4600 -- 541 MYA .'
+- source_sentence: In 2015 Adolf Hitler appeared in the kickstarter short movie ``
+    Kung Fury `` as Taccone ( A.K.A .
   sentences:
-  - In 2014 election , Biju Janata Dal candidate Tathagat Satapathy Bharatiya Janata
-    party candidate Rudra Narayan Pany defeated with a margin of 1.37,340 votes .
-  - In Scotland , the Strathclyde Partnership for Transport , formerly known as Strathclyde
-    Passenger Transport Executive , comprises the former Strathclyde region , which
-    includes the urban area around Glasgow .
-  - 'See Precambrian Time Scale # Proposed Geological Timeline for another set of
-    periods of 4600 -- 541 MYA .'
-- source_sentence: It is also 5 kilometers northeast of Tamaqua , 27 miles south of
-    Allentown and 9 miles northwest of Hazleton .
+  - In 2015 , Adolf Hitler appeared in the Kickstarter - short film `` Kung Fury ``
+    as Taccone ( A.K.A .
+  - In 1795 , the only white residents were Dr. John Laidley and two brothers with
+    the surname Ainslie .
+  - The 125th University Match was played in March 2014 at the Rye Golf Club , Oxford
+    , East Sussex won the game 8.5 - 6.5 .
+- source_sentence: From 1973 to 1974 , Aubrey toured with the Cambridge Theatre Company
+    as Diggory in `` She Stoops to Conquer `` and again as Aguecheek .
   sentences:
-  - In 1948 he moved to Massachusetts , and eventually settled in Vermont .
-  - Suddenly I remembered that I was a New Zealander , I caught the first plane home
-    and came back .
-  - It is also 5 miles northeast of Tamaqua , 27 miles south of Allentown , and 9
-    miles northwest of Hazleton .
-- source_sentence: The party has a Member of Parliament , a member of the House of
-    Lords , three members of the London Assembly and two Members of the European Parliament
-    .
+  - Oxide can be reduced to metallic samarium at higher temperatures by heating with
+    a reducing agent such as hydrogen or carbon monoxide .
+  - From 1973 to 1974 Aguecheek toured with the Cambridge Theatre Company as Diggory
+    in `` You Stoops to Conquer `` and again as Aubrey .
+  - The medals were presented by Barry Maister , IOC member , New Zealand and Sarah
+    Webb Gosling , Vice President of World Sailing .
+- source_sentence: There is no official wall on the border , although there are sections
+    of fence near populated areas and continuous border crossings .
   sentences:
-  - The party has one Member of Parliament , one member of the House of Lords , three
-    Members of the London Assembly and two Members of the European Parliament .
-  - Grapsid crabs dominate in Australia , Malaysia and Panama , while gastropods Cerithidea
-    scalariformis and Melampus coeffeus are important seed predators in Florida mangroves
-    .
-  - Music Story is a music service website and international music data provider that
-    curates , aggregates and analyses metadata for digital music services .
-- source_sentence: 'The play received two 1969 Tony Award nominations : Best Actress
-    in a Play ( Michael Annals ) and Best Costume Design ( Charlotte Rae ) .'
+  - The 2014 -- 15 Boston Bruins season was the 91st season for the National Hockey
+    League franchise that was established on November 1 , 1924 .
+  - He was trained by the Inghams and owned by John Hawkes .
+  - There is no continuous wall on the border , although there are fence sections
+    near populated areas and official border crossings .
+- source_sentence: Capital . `` The French established similar hill stations in Indochina
+    , such as Dalat built in 1921 .
   sentences:
-  - Ravishanker is a fellow of the International Statistical Institute and an elected
-    member of the American Statistical Association .
-  - 'In 1969 , the play received two Tony - Award nominations : Best Actress in a
-    Theatre Play ( Michael Annals ) and Best Costume Design ( Charlotte Rae ) .'
-  - AMD and Nvidia both have proprietary methods of scaling , CrossFireX for AMD ,
-    and SLI for Nvidia .
-- source_sentence: He was a close friend of Ángel Cabrera and is a cousin of golfer
-    Tony Croatto .
+  - Lubuk China is a small town in Alor Gajah District , Melaka , Malaysia . It is
+    situated near the border with Negeri Sembilan .
+  - The French established similar hill stations in Indochina , such as Dalat , built
+    in 1921 .
+  - John Potts ( or Pott ) was a doctor and colonial governor of Virginia in the Jamestown
+    settlement at Virginia Colony in the early 17th century .
+- source_sentence: The band pursued `` signals `` in January 2012 in three weeks ,
+    and drums were recorded in a day and a half .
   sentences:
-  - He was a close friend of Ángel Cabrera , and is a cousin of golfer Tony Croatto
-    .
-  - Eugenijus Bartulis ( born December 7 , 1949 in Kaunas ) is a Lithuanian Roman
-    Catholic priest , and Bishop of Šiauliai .
-  - UWIRE also distributes its members content to professional media outlets , including
-    Yahoo , CNN and CBS News .
+  - It was repaired at the beginning of the 20th century and is listed as closed in
+    our records .
+  - The band tracked `` Signals `` in three weeks in January 2012 . Drums were recorded
+    in a day and a half .
+  - Contributors include actor Anton LaVey , Satanist Christopher Lee , serial killer
+    expert Clive Barker , author Karen Greenlee , and necrophile Robert Ressler .
 datasets:
 - redis/langcache-sentencepairs-v1
 pipeline_tag: sentence-similarity
 library_name: sentence-transformers
 metrics:
-- cosine_accuracy@1
-- cosine_precision@1
-- cosine_recall@1
-- cosine_ndcg@10
-- cosine_mrr@1
-- cosine_map@100
+- cosine_accuracy
+- cosine_accuracy_threshold
+- cosine_f1
+- cosine_f1_threshold
+- cosine_precision
+- cosine_recall
+- cosine_ap
+- cosine_mcc
 model-index:
 - name: Redis fine-tuned BiEncoder model for semantic caching on LangCache
   results:
   - task:
-      type: information-retrieval
-      name: Information Retrieval
+      type: binary-classification
+      name: Binary Classification
     dataset:
-      name: train
-      type: train
+      name: val
+      type: val
     metrics:
-    - type: cosine_accuracy@1
-      value: 0.5576531716821823
-      name: Cosine Accuracy@1
-    - type: cosine_precision@1
-      value: 0.5576531716821823
-      name: Cosine Precision@1
-    - type: cosine_recall@1
-      value: 0.5355707469384673
-      name: Cosine Recall@1
-    - type: cosine_ndcg@10
-      value: 0.752122458721726
-      name: Cosine Ndcg@10
-    - type: cosine_mrr@1
-      value: 0.5576531716821823
-      name: Cosine Mrr@1
-    - type: cosine_map@100
-      value: 0.6973125692592467
-      name: Cosine Map@100
+    - type: cosine_accuracy
+      value: 0.7629982153480072
+      name: Cosine Accuracy
+    - type: cosine_accuracy_threshold
+      value: 0.8639795780181885
+      name: Cosine Accuracy Threshold
+    - type: cosine_f1
+      value: 0.6907391673746814
+      name: Cosine F1
+    - type: cosine_f1_threshold
+      value: 0.8261561989784241
+      name: Cosine F1 Threshold
+    - type: cosine_precision
+      value: 0.6290946608202218
+      name: Cosine Precision
+    - type: cosine_recall
+      value: 0.7657770800627943
+      name: Cosine Recall
+    - type: cosine_ap
+      value: 0.7350867007914639
+      name: Cosine Ap
+    - type: cosine_mcc
+      value: 0.47714361581572273
+      name: Cosine Mcc
+  - task:
+      type: binary-classification
+      name: Binary Classification
+    dataset:
+      name: test
+      type: test
+    metrics:
+    - type: cosine_accuracy
+      value: 0.7034875284177939
+      name: Cosine Accuracy
+    - type: cosine_accuracy_threshold
+      value: 0.8523406982421875
+      name: Cosine Accuracy Threshold
+    - type: cosine_f1
+      value: 0.7118695167174169
+      name: Cosine F1
+    - type: cosine_f1_threshold
+      value: 0.8109798431396484
+      name: Cosine F1 Threshold
+    - type: cosine_precision
+      value: 0.597953808752026
+      name: Cosine Precision
+    - type: cosine_recall
+      value: 0.8794040968342645
+      name: Cosine Recall
+    - type: cosine_ap
+      value: 0.6473629818920917
+      name: Cosine Ap
+    - type: cosine_mcc
+      value: 0.4409362621742405
+      name: Cosine Mcc
 ---
 
 # Redis fine-tuned BiEncoder model for semantic caching on LangCache
@@ -113,7 +149,7 @@ This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [A
 ### Model Description
 - **Model Type:** Sentence Transformer
 - **Base model:** [Alibaba-NLP/gte-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-modernbert-base) <!-- at revision e7f32e3c00f91d699e8c43b53106206bcc72bb22 -->
-- **Maximum Sequence Length:** 100 tokens
+- **Maximum Sequence Length:** 8192 tokens
 - **Output Dimensionality:** 768 dimensions
 - **Similarity Function:** Cosine Similarity
 - **Training Dataset:**
@@ -131,7 +167,7 @@ This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [A
 
 ```
 SentenceTransformer(
-  (0): Transformer({'max_seq_length': 100, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
+  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
   (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
 )
 ```
@@ -154,9 +190,9 @@ from sentence_transformers import SentenceTransformer
 model = SentenceTransformer("redis/langcache-embed-v3")
 # Run inference
 sentences = [
-    'He was a close friend of Ángel Cabrera and is a cousin of golfer Tony Croatto .',
-    'He was a close friend of Ángel Cabrera , and is a cousin of golfer Tony Croatto .',
-    'UWIRE also distributes its members content to professional media outlets , including Yahoo , CNN and CBS News .',
+    'The band pursued `` signals `` in January 2012 in three weeks , and drums were recorded in a day and a half .',
+    'The band tracked `` Signals `` in three weeks in January 2012 . Drums were recorded in a day and a half .',
+    'Contributors include actor Anton LaVey , Satanist Christopher Lee , serial killer expert Clive Barker , author Karen Greenlee , and necrophile Robert Ressler .',
 ]
 embeddings = model.encode(sentences)
 print(embeddings.shape)
@@ -165,9 +201,9 @@ print(embeddings.shape)
 # Get the similarity scores for the embeddings
 similarities = model.similarity(embeddings, embeddings)
 print(similarities)
-# tensor([[0.9922, 0.9922, 0.5352],
-#         [0.9922, 0.9961, 0.5391],
-#         [0.5352, 0.5391, 1.0000]], dtype=torch.bfloat16)
+# tensor([[0.9999, 0.9598, 0.4944],
+#         [0.9598, 0.9999, 0.5096],
+#         [0.4944, 0.5096, 0.9999]])
 ```
 
 <!--
@@ -198,19 +234,21 @@ You can finetune this model on your own dataset.
 
 ### Metrics
 
-#### Information Retrieval
+#### Binary Classification
 
-* Dataset: `train`
-* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
+* Datasets: `val` and `test`
+* Evaluated with [<code>BinaryClassificationEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.BinaryClassificationEvaluator)
 
-| Metric             | Value      |
-|:-------------------|:-----------|
-| cosine_accuracy@1  | 0.5577     |
-| cosine_precision@1 | 0.5577     |
-| cosine_recall@1    | 0.5356     |
-| **cosine_ndcg@10** | **0.7521** |
-| cosine_mrr@1       | 0.5577     |
-| cosine_map@100     | 0.6973     |
+| Metric                    | val        | test       |
+|:--------------------------|:-----------|:-----------|
+| cosine_accuracy           | 0.763      | 0.7035     |
+| cosine_accuracy_threshold | 0.864      | 0.8523     |
+| cosine_f1                 | 0.6907     | 0.7119     |
+| cosine_f1_threshold       | 0.8262     | 0.811      |
+| cosine_precision          | 0.6291     | 0.598      |
+| cosine_recall             | 0.7658     | 0.8794     |
+| **cosine_ap**             | **0.7351** | **0.6474** |
+| cosine_mcc                | 0.4771     | 0.4409     |
 
 <!--
 ## Bias, Risks and Limitations
@@ -231,26 +269,24 @@ You can finetune this model on your own dataset.
 #### LangCache Sentence Pairs (all)
 
 * Dataset: [LangCache Sentence Pairs (all)](https://huggingface.co/datasets/redis/langcache-sentencepairs-v1)
-* Size: 26,850 training samples
+* Size: 62,021 training samples
 * Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>label</code>
 * Approximate statistics based on the first 1000 samples:
-  |         | sentence1                                                                         | sentence2                                                                         | label                        |
-  |:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:-----------------------------|
-  | type    | string                                                                            | string                                                                            | int                          |
-  | details | <ul><li>min: 8 tokens</li><li>mean: 27.35 tokens</li><li>max: 53 tokens</li></ul> | <ul><li>min: 8 tokens</li><li>mean: 27.27 tokens</li><li>max: 52 tokens</li></ul> | <ul><li>1: 100.00%</li></ul> |
+  |         | sentence1                                                                         | sentence2                                                                         | label                                           |
+  |:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:------------------------------------------------|
+  | type    | string                                                                            | string                                                                            | int                                             |
+  | details | <ul><li>min: 8 tokens</li><li>mean: 27.46 tokens</li><li>max: 53 tokens</li></ul> | <ul><li>min: 9 tokens</li><li>mean: 27.36 tokens</li><li>max: 52 tokens</li></ul> | <ul><li>0: ~50.30%</li><li>1: ~49.70%</li></ul> |
 * Samples:
-  | sentence1                                                                                                             | sentence2                                                                                                                      | label          |
-  |:----------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------|:---------------|
-  | <code>The newer Punts are still very much in existence today and race in the same fleets as the older boats .</code>  | <code>The newer punts are still very much in existence today and run in the same fleets as the older boats .</code>            | <code>1</code> |
-  | <code>After losing his second election , he resigned as opposition leader and was replaced by Geoff Pearsall .</code> | <code>Max Bingham resigned as opposition leader after losing his second election , and was replaced by Geoff Pearsall .</code> | <code>1</code> |
-  | <code>The 12F was officially homologated on August 21 , 1929 and exhibited at the Paris Salon in 1930 .</code>        | <code>The 12F was officially homologated on 21 August 1929 and displayed at the 1930 Paris Salon .</code>                      | <code>1</code> |
-* Loss: [<code>CachedMultipleNegativesSymmetricRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedmultiplenegativessymmetricrankingloss) with these parameters:
+  | sentence1                                                                                                                                   | sentence2                                                                                                                                     | label          |
+  |:--------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------|:---------------|
+  | <code>The newer Punts are still very much in existence today and race in the same fleets as the older boats .</code>                        | <code>The newer punts are still very much in existence today and run in the same fleets as the older boats .</code>                           | <code>1</code> |
+  | <code>Turner Valley , was at the Turner Valley Bar N Ranch Airport , southwest of the Turner Valley Bar N Ranch , Alberta , Canada .</code> | <code>Turner Valley Bar N Ranch Airport , , was located at Turner Valley Bar N Ranch , southwest of Turner Valley , Alberta , Canada .</code> | <code>0</code> |
+  | <code>After losing his second election , he resigned as opposition leader and was replaced by Geoff Pearsall .</code>                       | <code>Max Bingham resigned as opposition leader after losing his second election , and was replaced by Geoff Pearsall .</code>                | <code>1</code> |
+* Loss: [<code>CoSENTLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss) with these parameters:
   ```json
   {
       "scale": 20.0,
-      "similarity_fct": "cos_sim",
-      "mini_batch_size": 96,
-      "gather_across_devices": false
+      "similarity_fct": "pairwise_cos_sim"
   }
   ```
 
@@ -259,33 +295,31 @@ You can finetune this model on your own dataset.
 #### LangCache Sentence Pairs (all)
 
 * Dataset: [LangCache Sentence Pairs (all)](https://huggingface.co/datasets/redis/langcache-sentencepairs-v1)
-* Size: 26,850 evaluation samples
+* Size: 62,021 evaluation samples
 * Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>label</code>
 * Approximate statistics based on the first 1000 samples:
-  |         | sentence1                                                                         | sentence2                                                                         | label                        |
-  |:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:-----------------------------|
-  | type    | string                                                                            | string                                                                            | int                          |
-  | details | <ul><li>min: 8 tokens</li><li>mean: 27.35 tokens</li><li>max: 53 tokens</li></ul> | <ul><li>min: 8 tokens</li><li>mean: 27.27 tokens</li><li>max: 52 tokens</li></ul> | <ul><li>1: 100.00%</li></ul> |
+  |         | sentence1                                                                         | sentence2                                                                         | label                                           |
+  |:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:------------------------------------------------|
+  | type    | string                                                                            | string                                                                            | int                                             |
+  | details | <ul><li>min: 8 tokens</li><li>mean: 27.46 tokens</li><li>max: 53 tokens</li></ul> | <ul><li>min: 9 tokens</li><li>mean: 27.36 tokens</li><li>max: 52 tokens</li></ul> | <ul><li>0: ~50.30%</li><li>1: ~49.70%</li></ul> |
 * Samples:
-  | sentence1                                                                                                             | sentence2                                                                                                                      | label          |
-  |:----------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------|:---------------|
-  | <code>The newer Punts are still very much in existence today and race in the same fleets as the older boats .</code>  | <code>The newer punts are still very much in existence today and run in the same fleets as the older boats .</code>            | <code>1</code> |
-  | <code>After losing his second election , he resigned as opposition leader and was replaced by Geoff Pearsall .</code> | <code>Max Bingham resigned as opposition leader after losing his second election , and was replaced by Geoff Pearsall .</code> | <code>1</code> |
-  | <code>The 12F was officially homologated on August 21 , 1929 and exhibited at the Paris Salon in 1930 .</code>        | <code>The 12F was officially homologated on 21 August 1929 and displayed at the 1930 Paris Salon .</code>                      | <code>1</code> |
-* Loss: [<code>CachedMultipleNegativesSymmetricRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedmultiplenegativessymmetricrankingloss) with these parameters:
+  | sentence1                                                                                                                                   | sentence2                                                                                                                                     | label          |
+  |:--------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------|:---------------|
+  | <code>The newer Punts are still very much in existence today and race in the same fleets as the older boats .</code>                        | <code>The newer punts are still very much in existence today and run in the same fleets as the older boats .</code>                           | <code>1</code> |
+  | <code>Turner Valley , was at the Turner Valley Bar N Ranch Airport , southwest of the Turner Valley Bar N Ranch , Alberta , Canada .</code> | <code>Turner Valley Bar N Ranch Airport , , was located at Turner Valley Bar N Ranch , southwest of Turner Valley , Alberta , Canada .</code> | <code>0</code> |
+  | <code>After losing his second election , he resigned as opposition leader and was replaced by Geoff Pearsall .</code>                       | <code>Max Bingham resigned as opposition leader after losing his second election , and was replaced by Geoff Pearsall .</code>                | <code>1</code> |
+* Loss: [<code>CoSENTLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss) with these parameters:
   ```json
   {
       "scale": 20.0,
-      "similarity_fct": "cos_sim",
-      "mini_batch_size": 96,
-      "gather_across_devices": false
+      "similarity_fct": "pairwise_cos_sim"
   }
   ```
 
 ### Training Logs
-| Epoch | Step | train_cosine_ndcg@10 |
-|:-----:|:----:|:--------------------:|
-| -1    | -1   | 0.7521               |
+| Epoch | Step | val_cosine_ap | test_cosine_ap |
+|:-----:|:----:|:-------------:|:--------------:|
+| -1    | -1   | 0.7351        | 0.6474         |
 
 
 ### Framework Versions
@@ -314,6 +348,17 @@ You can finetune this model on your own dataset.
 }
 ```
 
+#### CoSENTLoss
+```bibtex
+@online{kexuefm-8847,
+    title={CoSENT: A more efficient sentence vector scheme than Sentence-BERT},
+    author={Su Jianlin},
+    year={2022},
+    month={Jan},
+    url={https://kexue.fm/archives/8847},
+}
+```
+
 <!--
 ## Glossary
 

config.json

@@ -12,7 +12,7 @@
   "cls_token_id": 50281,
   "decoder_bias": true,
   "deterministic_flash_attn": false,
-  "dtype": "bfloat16",
+  "dtype": "float32",
   "embedding_dropout": 0.0,
   "eos_token_id": 50282,
   "global_attn_every_n_layers": 3,

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c00d45c397ee18dccac8a87b21c268ce14fed5bc561700ec08a2a0fc056f251b
-size 298041696
+oid sha256:0f9247027e7d57e8b36440b5b3d10a785ded92c7c9f4a313ff7f54a549967290
+size 596070136

sentence_bert_config.json

@@ -1,4 +1,4 @@
 {
-    "max_seq_length": 100,
+    "max_seq_length": 8192,
     "do_lower_case": false
 }

tokenizer.json

@@ -2,7 +2,7 @@
   "version": "1.0",
   "truncation": {
     "direction": "Right",
-    "max_length": 100,
+    "max_length": 8192,
     "strategy": "LongestFirst",
     "stride": 0
   },

tokenizer_config.json

@@ -933,7 +933,6 @@
   "cls_token": "[CLS]",
   "extra_special_tokens": {},
   "mask_token": "[MASK]",
-  "max_length": 100,
   "model_input_names": [
     "input_ids",
     "attention_mask"