|
|
---
|
|
|
tags:
|
|
|
- mteb
|
|
|
- Sentence Transformers
|
|
|
- sentence-similarity
|
|
|
- feature-extraction
|
|
|
- sentence-transformers
|
|
|
model-index:
|
|
|
- name: multilingual-e5-large
|
|
|
results:
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_counterfactual
|
|
|
name: MTEB AmazonCounterfactualClassification (en)
|
|
|
config: en
|
|
|
split: test
|
|
|
revision: e8379541af4e31359cca9fbcf4b00f2671dba205
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 79.05970149253731
|
|
|
- type: ap
|
|
|
value: 43.486574390835635
|
|
|
- type: f1
|
|
|
value: 73.32700092140148
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_counterfactual
|
|
|
name: MTEB AmazonCounterfactualClassification (de)
|
|
|
config: de
|
|
|
split: test
|
|
|
revision: e8379541af4e31359cca9fbcf4b00f2671dba205
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 71.22055674518201
|
|
|
- type: ap
|
|
|
value: 81.55756710830498
|
|
|
- type: f1
|
|
|
value: 69.28271787752661
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_counterfactual
|
|
|
name: MTEB AmazonCounterfactualClassification (en-ext)
|
|
|
config: en-ext
|
|
|
split: test
|
|
|
revision: e8379541af4e31359cca9fbcf4b00f2671dba205
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 80.41979010494754
|
|
|
- type: ap
|
|
|
value: 29.34879922376344
|
|
|
- type: f1
|
|
|
value: 67.62475449011278
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_counterfactual
|
|
|
name: MTEB AmazonCounterfactualClassification (ja)
|
|
|
config: ja
|
|
|
split: test
|
|
|
revision: e8379541af4e31359cca9fbcf4b00f2671dba205
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 77.8372591006424
|
|
|
- type: ap
|
|
|
value: 26.557560591210738
|
|
|
- type: f1
|
|
|
value: 64.96619417368707
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_polarity
|
|
|
name: MTEB AmazonPolarityClassification
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: e2d317d38cd51312af73b3d32a06d1a08b442046
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 93.489875
|
|
|
- type: ap
|
|
|
value: 90.98758636917603
|
|
|
- type: f1
|
|
|
value: 93.48554819717332
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_reviews_multi
|
|
|
name: MTEB AmazonReviewsClassification (en)
|
|
|
config: en
|
|
|
split: test
|
|
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 47.564
|
|
|
- type: f1
|
|
|
value: 46.75122173518047
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_reviews_multi
|
|
|
name: MTEB AmazonReviewsClassification (de)
|
|
|
config: de
|
|
|
split: test
|
|
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 45.400000000000006
|
|
|
- type: f1
|
|
|
value: 44.17195682400632
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_reviews_multi
|
|
|
name: MTEB AmazonReviewsClassification (es)
|
|
|
config: es
|
|
|
split: test
|
|
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 43.068
|
|
|
- type: f1
|
|
|
value: 42.38155696855596
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_reviews_multi
|
|
|
name: MTEB AmazonReviewsClassification (fr)
|
|
|
config: fr
|
|
|
split: test
|
|
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 41.89
|
|
|
- type: f1
|
|
|
value: 40.84407321682663
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_reviews_multi
|
|
|
name: MTEB AmazonReviewsClassification (ja)
|
|
|
config: ja
|
|
|
split: test
|
|
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 40.120000000000005
|
|
|
- type: f1
|
|
|
value: 39.522976223819114
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_reviews_multi
|
|
|
name: MTEB AmazonReviewsClassification (zh)
|
|
|
config: zh
|
|
|
split: test
|
|
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 38.832
|
|
|
- type: f1
|
|
|
value: 38.0392533394713
|
|
|
- task:
|
|
|
type: Retrieval
|
|
|
dataset:
|
|
|
type: arguana
|
|
|
name: MTEB ArguAna
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: None
|
|
|
metrics:
|
|
|
- type: map_at_1
|
|
|
value: 30.725
|
|
|
- type: map_at_10
|
|
|
value: 46.055
|
|
|
- type: map_at_100
|
|
|
value: 46.900999999999996
|
|
|
- type: map_at_1000
|
|
|
value: 46.911
|
|
|
- type: map_at_3
|
|
|
value: 41.548
|
|
|
- type: map_at_5
|
|
|
value: 44.297
|
|
|
- type: mrr_at_1
|
|
|
value: 31.152
|
|
|
- type: mrr_at_10
|
|
|
value: 46.231
|
|
|
- type: mrr_at_100
|
|
|
value: 47.07
|
|
|
- type: mrr_at_1000
|
|
|
value: 47.08
|
|
|
- type: mrr_at_3
|
|
|
value: 41.738
|
|
|
- type: mrr_at_5
|
|
|
value: 44.468999999999994
|
|
|
- type: ndcg_at_1
|
|
|
value: 30.725
|
|
|
- type: ndcg_at_10
|
|
|
value: 54.379999999999995
|
|
|
- type: ndcg_at_100
|
|
|
value: 58.138
|
|
|
- type: ndcg_at_1000
|
|
|
value: 58.389
|
|
|
- type: ndcg_at_3
|
|
|
value: 45.156
|
|
|
- type: ndcg_at_5
|
|
|
value: 50.123
|
|
|
- type: precision_at_1
|
|
|
value: 30.725
|
|
|
- type: precision_at_10
|
|
|
value: 8.087
|
|
|
- type: precision_at_100
|
|
|
value: 0.9769999999999999
|
|
|
- type: precision_at_1000
|
|
|
value: 0.1
|
|
|
- type: precision_at_3
|
|
|
value: 18.54
|
|
|
- type: precision_at_5
|
|
|
value: 13.542000000000002
|
|
|
- type: recall_at_1
|
|
|
value: 30.725
|
|
|
- type: recall_at_10
|
|
|
value: 80.868
|
|
|
- type: recall_at_100
|
|
|
value: 97.653
|
|
|
- type: recall_at_1000
|
|
|
value: 99.57300000000001
|
|
|
- type: recall_at_3
|
|
|
value: 55.619
|
|
|
- type: recall_at_5
|
|
|
value: 67.71000000000001
|
|
|
- task:
|
|
|
type: Clustering
|
|
|
dataset:
|
|
|
type: mteb/arxiv-clustering-p2p
|
|
|
name: MTEB ArxivClusteringP2P
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d
|
|
|
metrics:
|
|
|
- type: v_measure
|
|
|
value: 44.30960650674069
|
|
|
- task:
|
|
|
type: Clustering
|
|
|
dataset:
|
|
|
type: mteb/arxiv-clustering-s2s
|
|
|
name: MTEB ArxivClusteringS2S
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53
|
|
|
metrics:
|
|
|
- type: v_measure
|
|
|
value: 38.427074197498996
|
|
|
- task:
|
|
|
type: Reranking
|
|
|
dataset:
|
|
|
type: mteb/askubuntudupquestions-reranking
|
|
|
name: MTEB AskUbuntuDupQuestions
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: 2000358ca161889fa9c082cb41daa8dcfb161a54
|
|
|
metrics:
|
|
|
- type: map
|
|
|
value: 60.28270056031872
|
|
|
- type: mrr
|
|
|
value: 74.38332673789738
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/biosses-sts
|
|
|
name: MTEB BIOSSES
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: d3fb88f8f02e40887cd149695127462bbcf29b4a
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 84.05942144105269
|
|
|
- type: cos_sim_spearman
|
|
|
value: 82.51212105850809
|
|
|
- type: euclidean_pearson
|
|
|
value: 81.95639829909122
|
|
|
- type: euclidean_spearman
|
|
|
value: 82.3717564144213
|
|
|
- type: manhattan_pearson
|
|
|
value: 81.79273425468256
|
|
|
- type: manhattan_spearman
|
|
|
value: 82.20066817871039
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/bucc-bitext-mining
|
|
|
name: MTEB BUCC (de-en)
|
|
|
config: de-en
|
|
|
split: test
|
|
|
revision: d51519689f32196a32af33b075a01d0e7c51e252
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 99.46764091858039
|
|
|
- type: f1
|
|
|
value: 99.37717466945023
|
|
|
- type: precision
|
|
|
value: 99.33194154488518
|
|
|
- type: recall
|
|
|
value: 99.46764091858039
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/bucc-bitext-mining
|
|
|
name: MTEB BUCC (fr-en)
|
|
|
config: fr-en
|
|
|
split: test
|
|
|
revision: d51519689f32196a32af33b075a01d0e7c51e252
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 98.29407880255337
|
|
|
- type: f1
|
|
|
value: 98.11248073959938
|
|
|
- type: precision
|
|
|
value: 98.02443319392472
|
|
|
- type: recall
|
|
|
value: 98.29407880255337
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/bucc-bitext-mining
|
|
|
name: MTEB BUCC (ru-en)
|
|
|
config: ru-en
|
|
|
split: test
|
|
|
revision: d51519689f32196a32af33b075a01d0e7c51e252
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 97.79009352268791
|
|
|
- type: f1
|
|
|
value: 97.5176076665512
|
|
|
- type: precision
|
|
|
value: 97.38136473848286
|
|
|
- type: recall
|
|
|
value: 97.79009352268791
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/bucc-bitext-mining
|
|
|
name: MTEB BUCC (zh-en)
|
|
|
config: zh-en
|
|
|
split: test
|
|
|
revision: d51519689f32196a32af33b075a01d0e7c51e252
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 99.26276987888363
|
|
|
- type: f1
|
|
|
value: 99.20133403545726
|
|
|
- type: precision
|
|
|
value: 99.17500438827453
|
|
|
- type: recall
|
|
|
value: 99.26276987888363
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/banking77
|
|
|
name: MTEB Banking77Classification
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: 0fd18e25b25c072e09e0d92ab615fda904d66300
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 84.72727272727273
|
|
|
- type: f1
|
|
|
value: 84.67672206031433
|
|
|
- task:
|
|
|
type: Clustering
|
|
|
dataset:
|
|
|
type: mteb/biorxiv-clustering-p2p
|
|
|
name: MTEB BiorxivClusteringP2P
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40
|
|
|
metrics:
|
|
|
- type: v_measure
|
|
|
value: 35.34220182511161
|
|
|
- task:
|
|
|
type: Clustering
|
|
|
dataset:
|
|
|
type: mteb/biorxiv-clustering-s2s
|
|
|
name: MTEB BiorxivClusteringS2S
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908
|
|
|
metrics:
|
|
|
- type: v_measure
|
|
|
value: 33.4987096128766
|
|
|
- task:
|
|
|
type: Retrieval
|
|
|
dataset:
|
|
|
type: BeIR/cqadupstack
|
|
|
name: MTEB CQADupstackRetrieval
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: None
|
|
|
metrics:
|
|
|
- type: map_at_1
|
|
|
value: 25.558249999999997
|
|
|
- type: map_at_10
|
|
|
value: 34.44425000000001
|
|
|
- type: map_at_100
|
|
|
value: 35.59833333333333
|
|
|
- type: map_at_1000
|
|
|
value: 35.706916666666665
|
|
|
- type: map_at_3
|
|
|
value: 31.691749999999995
|
|
|
- type: map_at_5
|
|
|
value: 33.252916666666664
|
|
|
- type: mrr_at_1
|
|
|
value: 30.252666666666666
|
|
|
- type: mrr_at_10
|
|
|
value: 38.60675
|
|
|
- type: mrr_at_100
|
|
|
value: 39.42666666666666
|
|
|
- type: mrr_at_1000
|
|
|
value: 39.48408333333334
|
|
|
- type: mrr_at_3
|
|
|
value: 36.17441666666665
|
|
|
- type: mrr_at_5
|
|
|
value: 37.56275
|
|
|
- type: ndcg_at_1
|
|
|
value: 30.252666666666666
|
|
|
- type: ndcg_at_10
|
|
|
value: 39.683
|
|
|
- type: ndcg_at_100
|
|
|
value: 44.68541666666667
|
|
|
- type: ndcg_at_1000
|
|
|
value: 46.94316666666668
|
|
|
- type: ndcg_at_3
|
|
|
value: 34.961749999999995
|
|
|
- type: ndcg_at_5
|
|
|
value: 37.215666666666664
|
|
|
- type: precision_at_1
|
|
|
value: 30.252666666666666
|
|
|
- type: precision_at_10
|
|
|
value: 6.904166666666667
|
|
|
- type: precision_at_100
|
|
|
value: 1.0989999999999995
|
|
|
- type: precision_at_1000
|
|
|
value: 0.14733333333333334
|
|
|
- type: precision_at_3
|
|
|
value: 16.037666666666667
|
|
|
- type: precision_at_5
|
|
|
value: 11.413583333333333
|
|
|
- type: recall_at_1
|
|
|
value: 25.558249999999997
|
|
|
- type: recall_at_10
|
|
|
value: 51.13341666666666
|
|
|
- type: recall_at_100
|
|
|
value: 73.08366666666667
|
|
|
- type: recall_at_1000
|
|
|
value: 88.79483333333334
|
|
|
- type: recall_at_3
|
|
|
value: 37.989083333333326
|
|
|
- type: recall_at_5
|
|
|
value: 43.787833333333325
|
|
|
- task:
|
|
|
type: Retrieval
|
|
|
dataset:
|
|
|
type: climate-fever
|
|
|
name: MTEB ClimateFEVER
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: None
|
|
|
metrics:
|
|
|
- type: map_at_1
|
|
|
value: 10.338
|
|
|
- type: map_at_10
|
|
|
value: 18.360000000000003
|
|
|
- type: map_at_100
|
|
|
value: 19.942
|
|
|
- type: map_at_1000
|
|
|
value: 20.134
|
|
|
- type: map_at_3
|
|
|
value: 15.174000000000001
|
|
|
- type: map_at_5
|
|
|
value: 16.830000000000002
|
|
|
- type: mrr_at_1
|
|
|
value: 23.257
|
|
|
- type: mrr_at_10
|
|
|
value: 33.768
|
|
|
- type: mrr_at_100
|
|
|
value: 34.707
|
|
|
- type: mrr_at_1000
|
|
|
value: 34.766000000000005
|
|
|
- type: mrr_at_3
|
|
|
value: 30.977
|
|
|
- type: mrr_at_5
|
|
|
value: 32.528
|
|
|
- type: ndcg_at_1
|
|
|
value: 23.257
|
|
|
- type: ndcg_at_10
|
|
|
value: 25.733
|
|
|
- type: ndcg_at_100
|
|
|
value: 32.288
|
|
|
- type: ndcg_at_1000
|
|
|
value: 35.992000000000004
|
|
|
- type: ndcg_at_3
|
|
|
value: 20.866
|
|
|
- type: ndcg_at_5
|
|
|
value: 22.612
|
|
|
- type: precision_at_1
|
|
|
value: 23.257
|
|
|
- type: precision_at_10
|
|
|
value: 8.124
|
|
|
- type: precision_at_100
|
|
|
value: 1.518
|
|
|
- type: precision_at_1000
|
|
|
value: 0.219
|
|
|
- type: precision_at_3
|
|
|
value: 15.679000000000002
|
|
|
- type: precision_at_5
|
|
|
value: 12.117
|
|
|
- type: recall_at_1
|
|
|
value: 10.338
|
|
|
- type: recall_at_10
|
|
|
value: 31.154
|
|
|
- type: recall_at_100
|
|
|
value: 54.161
|
|
|
- type: recall_at_1000
|
|
|
value: 75.21900000000001
|
|
|
- type: recall_at_3
|
|
|
value: 19.427
|
|
|
- type: recall_at_5
|
|
|
value: 24.214
|
|
|
- task:
|
|
|
type: Retrieval
|
|
|
dataset:
|
|
|
type: dbpedia-entity
|
|
|
name: MTEB DBPedia
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: None
|
|
|
metrics:
|
|
|
- type: map_at_1
|
|
|
value: 8.498
|
|
|
- type: map_at_10
|
|
|
value: 19.103
|
|
|
- type: map_at_100
|
|
|
value: 27.375
|
|
|
- type: map_at_1000
|
|
|
value: 28.981
|
|
|
- type: map_at_3
|
|
|
value: 13.764999999999999
|
|
|
- type: map_at_5
|
|
|
value: 15.950000000000001
|
|
|
- type: mrr_at_1
|
|
|
value: 65.5
|
|
|
- type: mrr_at_10
|
|
|
value: 74.53800000000001
|
|
|
- type: mrr_at_100
|
|
|
value: 74.71799999999999
|
|
|
- type: mrr_at_1000
|
|
|
value: 74.725
|
|
|
- type: mrr_at_3
|
|
|
value: 72.792
|
|
|
- type: mrr_at_5
|
|
|
value: 73.554
|
|
|
- type: ndcg_at_1
|
|
|
value: 53.37499999999999
|
|
|
- type: ndcg_at_10
|
|
|
value: 41.286
|
|
|
- type: ndcg_at_100
|
|
|
value: 45.972
|
|
|
- type: ndcg_at_1000
|
|
|
value: 53.123
|
|
|
- type: ndcg_at_3
|
|
|
value: 46.172999999999995
|
|
|
- type: ndcg_at_5
|
|
|
value: 43.033
|
|
|
- type: precision_at_1
|
|
|
value: 65.5
|
|
|
- type: precision_at_10
|
|
|
value: 32.725
|
|
|
- type: precision_at_100
|
|
|
value: 10.683
|
|
|
- type: precision_at_1000
|
|
|
value: 1.978
|
|
|
- type: precision_at_3
|
|
|
value: 50
|
|
|
- type: precision_at_5
|
|
|
value: 41.349999999999994
|
|
|
- type: recall_at_1
|
|
|
value: 8.498
|
|
|
- type: recall_at_10
|
|
|
value: 25.070999999999998
|
|
|
- type: recall_at_100
|
|
|
value: 52.383
|
|
|
- type: recall_at_1000
|
|
|
value: 74.91499999999999
|
|
|
- type: recall_at_3
|
|
|
value: 15.207999999999998
|
|
|
- type: recall_at_5
|
|
|
value: 18.563
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/emotion
|
|
|
name: MTEB EmotionClassification
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 46.5
|
|
|
- type: f1
|
|
|
value: 41.93833713984145
|
|
|
- task:
|
|
|
type: Retrieval
|
|
|
dataset:
|
|
|
type: fever
|
|
|
name: MTEB FEVER
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: None
|
|
|
metrics:
|
|
|
- type: map_at_1
|
|
|
value: 67.914
|
|
|
- type: map_at_10
|
|
|
value: 78.10000000000001
|
|
|
- type: map_at_100
|
|
|
value: 78.333
|
|
|
- type: map_at_1000
|
|
|
value: 78.346
|
|
|
- type: map_at_3
|
|
|
value: 76.626
|
|
|
- type: map_at_5
|
|
|
value: 77.627
|
|
|
- type: mrr_at_1
|
|
|
value: 72.74199999999999
|
|
|
- type: mrr_at_10
|
|
|
value: 82.414
|
|
|
- type: mrr_at_100
|
|
|
value: 82.511
|
|
|
- type: mrr_at_1000
|
|
|
value: 82.513
|
|
|
- type: mrr_at_3
|
|
|
value: 81.231
|
|
|
- type: mrr_at_5
|
|
|
value: 82.065
|
|
|
- type: ndcg_at_1
|
|
|
value: 72.74199999999999
|
|
|
- type: ndcg_at_10
|
|
|
value: 82.806
|
|
|
- type: ndcg_at_100
|
|
|
value: 83.677
|
|
|
- type: ndcg_at_1000
|
|
|
value: 83.917
|
|
|
- type: ndcg_at_3
|
|
|
value: 80.305
|
|
|
- type: ndcg_at_5
|
|
|
value: 81.843
|
|
|
- type: precision_at_1
|
|
|
value: 72.74199999999999
|
|
|
- type: precision_at_10
|
|
|
value: 10.24
|
|
|
- type: precision_at_100
|
|
|
value: 1.089
|
|
|
- type: precision_at_1000
|
|
|
value: 0.11299999999999999
|
|
|
- type: precision_at_3
|
|
|
value: 31.268
|
|
|
- type: precision_at_5
|
|
|
value: 19.706000000000003
|
|
|
- type: recall_at_1
|
|
|
value: 67.914
|
|
|
- type: recall_at_10
|
|
|
value: 92.889
|
|
|
- type: recall_at_100
|
|
|
value: 96.42699999999999
|
|
|
- type: recall_at_1000
|
|
|
value: 97.92
|
|
|
- type: recall_at_3
|
|
|
value: 86.21
|
|
|
- type: recall_at_5
|
|
|
value: 90.036
|
|
|
- task:
|
|
|
type: Retrieval
|
|
|
dataset:
|
|
|
type: fiqa
|
|
|
name: MTEB FiQA2018
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: None
|
|
|
metrics:
|
|
|
- type: map_at_1
|
|
|
value: 22.166
|
|
|
- type: map_at_10
|
|
|
value: 35.57
|
|
|
- type: map_at_100
|
|
|
value: 37.405
|
|
|
- type: map_at_1000
|
|
|
value: 37.564
|
|
|
- type: map_at_3
|
|
|
value: 30.379
|
|
|
- type: map_at_5
|
|
|
value: 33.324
|
|
|
- type: mrr_at_1
|
|
|
value: 43.519000000000005
|
|
|
- type: mrr_at_10
|
|
|
value: 51.556000000000004
|
|
|
- type: mrr_at_100
|
|
|
value: 52.344
|
|
|
- type: mrr_at_1000
|
|
|
value: 52.373999999999995
|
|
|
- type: mrr_at_3
|
|
|
value: 48.868
|
|
|
- type: mrr_at_5
|
|
|
value: 50.319
|
|
|
- type: ndcg_at_1
|
|
|
value: 43.519000000000005
|
|
|
- type: ndcg_at_10
|
|
|
value: 43.803
|
|
|
- type: ndcg_at_100
|
|
|
value: 50.468999999999994
|
|
|
- type: ndcg_at_1000
|
|
|
value: 53.111
|
|
|
- type: ndcg_at_3
|
|
|
value: 38.893
|
|
|
- type: ndcg_at_5
|
|
|
value: 40.653
|
|
|
- type: precision_at_1
|
|
|
value: 43.519000000000005
|
|
|
- type: precision_at_10
|
|
|
value: 12.253
|
|
|
- type: precision_at_100
|
|
|
value: 1.931
|
|
|
- type: precision_at_1000
|
|
|
value: 0.242
|
|
|
- type: precision_at_3
|
|
|
value: 25.617
|
|
|
- type: precision_at_5
|
|
|
value: 19.383
|
|
|
- type: recall_at_1
|
|
|
value: 22.166
|
|
|
- type: recall_at_10
|
|
|
value: 51.6
|
|
|
- type: recall_at_100
|
|
|
value: 76.574
|
|
|
- type: recall_at_1000
|
|
|
value: 92.192
|
|
|
- type: recall_at_3
|
|
|
value: 34.477999999999994
|
|
|
- type: recall_at_5
|
|
|
value: 41.835
|
|
|
- task:
|
|
|
type: Retrieval
|
|
|
dataset:
|
|
|
type: hotpotqa
|
|
|
name: MTEB HotpotQA
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: None
|
|
|
metrics:
|
|
|
- type: map_at_1
|
|
|
value: 39.041
|
|
|
- type: map_at_10
|
|
|
value: 62.961999999999996
|
|
|
- type: map_at_100
|
|
|
value: 63.79899999999999
|
|
|
- type: map_at_1000
|
|
|
value: 63.854
|
|
|
- type: map_at_3
|
|
|
value: 59.399
|
|
|
- type: map_at_5
|
|
|
value: 61.669
|
|
|
- type: mrr_at_1
|
|
|
value: 78.082
|
|
|
- type: mrr_at_10
|
|
|
value: 84.321
|
|
|
- type: mrr_at_100
|
|
|
value: 84.49600000000001
|
|
|
- type: mrr_at_1000
|
|
|
value: 84.502
|
|
|
- type: mrr_at_3
|
|
|
value: 83.421
|
|
|
- type: mrr_at_5
|
|
|
value: 83.977
|
|
|
- type: ndcg_at_1
|
|
|
value: 78.082
|
|
|
- type: ndcg_at_10
|
|
|
value: 71.229
|
|
|
- type: ndcg_at_100
|
|
|
value: 74.10900000000001
|
|
|
- type: ndcg_at_1000
|
|
|
value: 75.169
|
|
|
- type: ndcg_at_3
|
|
|
value: 66.28699999999999
|
|
|
- type: ndcg_at_5
|
|
|
value: 69.084
|
|
|
- type: precision_at_1
|
|
|
value: 78.082
|
|
|
- type: precision_at_10
|
|
|
value: 14.993
|
|
|
- type: precision_at_100
|
|
|
value: 1.7239999999999998
|
|
|
- type: precision_at_1000
|
|
|
value: 0.186
|
|
|
- type: precision_at_3
|
|
|
value: 42.737
|
|
|
- type: precision_at_5
|
|
|
value: 27.843
|
|
|
- type: recall_at_1
|
|
|
value: 39.041
|
|
|
- type: recall_at_10
|
|
|
value: 74.96300000000001
|
|
|
- type: recall_at_100
|
|
|
value: 86.199
|
|
|
- type: recall_at_1000
|
|
|
value: 93.228
|
|
|
- type: recall_at_3
|
|
|
value: 64.105
|
|
|
- type: recall_at_5
|
|
|
value: 69.608
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/imdb
|
|
|
name: MTEB ImdbClassification
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 90.23160000000001
|
|
|
- type: ap
|
|
|
value: 85.5674856808308
|
|
|
- type: f1
|
|
|
value: 90.18033354786317
|
|
|
- task:
|
|
|
type: Retrieval
|
|
|
dataset:
|
|
|
type: msmarco
|
|
|
name: MTEB MSMARCO
|
|
|
config: default
|
|
|
split: dev
|
|
|
revision: None
|
|
|
metrics:
|
|
|
- type: map_at_1
|
|
|
value: 24.091
|
|
|
- type: map_at_10
|
|
|
value: 36.753
|
|
|
- type: map_at_100
|
|
|
value: 37.913000000000004
|
|
|
- type: map_at_1000
|
|
|
value: 37.958999999999996
|
|
|
- type: map_at_3
|
|
|
value: 32.818999999999996
|
|
|
- type: map_at_5
|
|
|
value: 35.171
|
|
|
- type: mrr_at_1
|
|
|
value: 24.742
|
|
|
- type: mrr_at_10
|
|
|
value: 37.285000000000004
|
|
|
- type: mrr_at_100
|
|
|
value: 38.391999999999996
|
|
|
- type: mrr_at_1000
|
|
|
value: 38.431
|
|
|
- type: mrr_at_3
|
|
|
value: 33.440999999999995
|
|
|
- type: mrr_at_5
|
|
|
value: 35.75
|
|
|
- type: ndcg_at_1
|
|
|
value: 24.742
|
|
|
- type: ndcg_at_10
|
|
|
value: 43.698
|
|
|
- type: ndcg_at_100
|
|
|
value: 49.145
|
|
|
- type: ndcg_at_1000
|
|
|
value: 50.23800000000001
|
|
|
- type: ndcg_at_3
|
|
|
value: 35.769
|
|
|
- type: ndcg_at_5
|
|
|
value: 39.961999999999996
|
|
|
- type: precision_at_1
|
|
|
value: 24.742
|
|
|
- type: precision_at_10
|
|
|
value: 6.7989999999999995
|
|
|
- type: precision_at_100
|
|
|
value: 0.95
|
|
|
- type: precision_at_1000
|
|
|
value: 0.104
|
|
|
- type: precision_at_3
|
|
|
value: 15.096000000000002
|
|
|
- type: precision_at_5
|
|
|
value: 11.183
|
|
|
- type: recall_at_1
|
|
|
value: 24.091
|
|
|
- type: recall_at_10
|
|
|
value: 65.068
|
|
|
- type: recall_at_100
|
|
|
value: 89.899
|
|
|
- type: recall_at_1000
|
|
|
value: 98.16
|
|
|
- type: recall_at_3
|
|
|
value: 43.68
|
|
|
- type: recall_at_5
|
|
|
value: 53.754999999999995
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/mtop_domain
|
|
|
name: MTEB MTOPDomainClassification (en)
|
|
|
config: en
|
|
|
split: test
|
|
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 93.66621067031465
|
|
|
- type: f1
|
|
|
value: 93.49622853272142
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/mtop_domain
|
|
|
name: MTEB MTOPDomainClassification (de)
|
|
|
config: de
|
|
|
split: test
|
|
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 91.94702733164272
|
|
|
- type: f1
|
|
|
value: 91.17043441745282
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/mtop_domain
|
|
|
name: MTEB MTOPDomainClassification (es)
|
|
|
config: es
|
|
|
split: test
|
|
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 92.20146764509674
|
|
|
- type: f1
|
|
|
value: 91.98359080555608
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/mtop_domain
|
|
|
name: MTEB MTOPDomainClassification (fr)
|
|
|
config: fr
|
|
|
split: test
|
|
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 88.99780770435328
|
|
|
- type: f1
|
|
|
value: 89.19746342724068
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/mtop_domain
|
|
|
name: MTEB MTOPDomainClassification (hi)
|
|
|
config: hi
|
|
|
split: test
|
|
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 89.78486912871998
|
|
|
- type: f1
|
|
|
value: 89.24578823628642
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/mtop_domain
|
|
|
name: MTEB MTOPDomainClassification (th)
|
|
|
config: th
|
|
|
split: test
|
|
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 88.74502712477394
|
|
|
- type: f1
|
|
|
value: 89.00297573881542
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/mtop_intent
|
|
|
name: MTEB MTOPIntentClassification (en)
|
|
|
config: en
|
|
|
split: test
|
|
|
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 77.9046967624259
|
|
|
- type: f1
|
|
|
value: 59.36787125785957
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/mtop_intent
|
|
|
name: MTEB MTOPIntentClassification (de)
|
|
|
config: de
|
|
|
split: test
|
|
|
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 74.5280360664976
|
|
|
- type: f1
|
|
|
value: 57.17723440888718
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/mtop_intent
|
|
|
name: MTEB MTOPIntentClassification (es)
|
|
|
config: es
|
|
|
split: test
|
|
|
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 75.44029352901934
|
|
|
- type: f1
|
|
|
value: 54.052855531072964
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/mtop_intent
|
|
|
name: MTEB MTOPIntentClassification (fr)
|
|
|
config: fr
|
|
|
split: test
|
|
|
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 70.5606013153774
|
|
|
- type: f1
|
|
|
value: 52.62215934386531
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/mtop_intent
|
|
|
name: MTEB MTOPIntentClassification (hi)
|
|
|
config: hi
|
|
|
split: test
|
|
|
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 73.11581211903908
|
|
|
- type: f1
|
|
|
value: 52.341291845645465
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/mtop_intent
|
|
|
name: MTEB MTOPIntentClassification (th)
|
|
|
config: th
|
|
|
split: test
|
|
|
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 74.28933092224233
|
|
|
- type: f1
|
|
|
value: 57.07918745504911
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (af)
|
|
|
config: af
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 62.38063214525892
|
|
|
- type: f1
|
|
|
value: 59.46463723443009
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (am)
|
|
|
config: am
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 56.06926698049766
|
|
|
- type: f1
|
|
|
value: 52.49084283283562
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (ar)
|
|
|
config: ar
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 60.74983187626093
|
|
|
- type: f1
|
|
|
value: 56.960640620165904
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (az)
|
|
|
config: az
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 64.86550100874243
|
|
|
- type: f1
|
|
|
value: 62.47370548140688
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (bn)
|
|
|
config: bn
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 63.971082716879636
|
|
|
- type: f1
|
|
|
value: 61.03812421957381
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (cy)
|
|
|
config: cy
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 54.98318762609282
|
|
|
- type: f1
|
|
|
value: 51.51207916008392
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (da)
|
|
|
config: da
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 69.45527908540686
|
|
|
- type: f1
|
|
|
value: 66.16631905400318
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (de)
|
|
|
config: de
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 69.32750504371216
|
|
|
- type: f1
|
|
|
value: 66.16755288646591
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (el)
|
|
|
config: el
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 69.09213180901143
|
|
|
- type: f1
|
|
|
value: 66.95654394661507
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (en)
|
|
|
config: en
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 73.75588433086752
|
|
|
- type: f1
|
|
|
value: 71.79973779656923
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (es)
|
|
|
config: es
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 70.49428379287154
|
|
|
- type: f1
|
|
|
value: 68.37494379215734
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (fa)
|
|
|
config: fa
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 69.90921318090115
|
|
|
- type: f1
|
|
|
value: 66.79517376481645
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (fi)
|
|
|
config: fi
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 70.12104909213181
|
|
|
- type: f1
|
|
|
value: 67.29448842879584
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (fr)
|
|
|
config: fr
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 69.34095494283793
|
|
|
- type: f1
|
|
|
value: 67.01134288992947
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (he)
|
|
|
config: he
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 67.61264290517822
|
|
|
- type: f1
|
|
|
value: 64.68730512660757
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (hi)
|
|
|
config: hi
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 67.79757901815738
|
|
|
- type: f1
|
|
|
value: 65.24938539425598
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (hu)
|
|
|
config: hu
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 69.68728984532616
|
|
|
- type: f1
|
|
|
value: 67.0487169762553
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (hy)
|
|
|
config: hy
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 62.07464694014795
|
|
|
- type: f1
|
|
|
value: 59.183532276789286
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (id)
|
|
|
config: id
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 70.04707464694015
|
|
|
- type: f1
|
|
|
value: 67.66829629003848
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (is)
|
|
|
config: is
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 62.42434431741762
|
|
|
- type: f1
|
|
|
value: 59.01617226544757
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (it)
|
|
|
config: it
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 70.53127101546738
|
|
|
- type: f1
|
|
|
value: 68.10033760906255
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (ja)
|
|
|
config: ja
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 72.50504371217215
|
|
|
- type: f1
|
|
|
value: 69.74931103158923
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (jv)
|
|
|
config: jv
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 57.91190316072628
|
|
|
- type: f1
|
|
|
value: 54.05551136648796
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (ka)
|
|
|
config: ka
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 51.78211163416275
|
|
|
- type: f1
|
|
|
value: 49.874888544058535
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (km)
|
|
|
config: km
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 47.017484868863484
|
|
|
- type: f1
|
|
|
value: 44.53364263352014
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (kn)
|
|
|
config: kn
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 62.16207128446537
|
|
|
- type: f1
|
|
|
value: 59.01185692320829
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (ko)
|
|
|
config: ko
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 69.42501681237391
|
|
|
- type: f1
|
|
|
value: 67.13169450166086
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (lv)
|
|
|
config: lv
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 67.0780094149294
|
|
|
- type: f1
|
|
|
value: 64.41720167850707
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (ml)
|
|
|
config: ml
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 65.57162071284466
|
|
|
- type: f1
|
|
|
value: 62.414138683804424
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (mn)
|
|
|
config: mn
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 61.71149966375252
|
|
|
- type: f1
|
|
|
value: 58.594805125087234
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (ms)
|
|
|
config: ms
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 66.03900470746471
|
|
|
- type: f1
|
|
|
value: 63.87937257883887
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (my)
|
|
|
config: my
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 60.8776059179556
|
|
|
- type: f1
|
|
|
value: 57.48587618059131
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (nb)
|
|
|
config: nb
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 69.87895090786819
|
|
|
- type: f1
|
|
|
value: 66.8141299430347
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (nl)
|
|
|
config: nl
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 70.45057162071285
|
|
|
- type: f1
|
|
|
value: 67.46444039673516
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (pl)
|
|
|
config: pl
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 71.546738399462
|
|
|
- type: f1
|
|
|
value: 68.63640876702655
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (pt)
|
|
|
config: pt
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 70.72965702757229
|
|
|
- type: f1
|
|
|
value: 68.54119560379115
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (ro)
|
|
|
config: ro
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 68.35574983187625
|
|
|
- type: f1
|
|
|
value: 65.88844917691927
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (ru)
|
|
|
config: ru
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 71.70477471418964
|
|
|
- type: f1
|
|
|
value: 69.19665697061978
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (sl)
|
|
|
config: sl
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 67.0880968392737
|
|
|
- type: f1
|
|
|
value: 64.76962317666086
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (sq)
|
|
|
config: sq
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 65.18493611297916
|
|
|
- type: f1
|
|
|
value: 62.49984559035371
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (sv)
|
|
|
config: sv
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 71.75857431069265
|
|
|
- type: f1
|
|
|
value: 69.20053687623418
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (sw)
|
|
|
config: sw
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 58.500336247478145
|
|
|
- type: f1
|
|
|
value: 55.2972398687929
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (ta)
|
|
|
config: ta
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 62.68997982515132
|
|
|
- type: f1
|
|
|
value: 59.36848202755348
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (te)
|
|
|
config: te
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 63.01950235373235
|
|
|
- type: f1
|
|
|
value: 60.09351954625423
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (th)
|
|
|
config: th
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 68.29186281102892
|
|
|
- type: f1
|
|
|
value: 67.57860496703447
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (tl)
|
|
|
config: tl
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 64.77471418964357
|
|
|
- type: f1
|
|
|
value: 61.913983147713836
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (tr)
|
|
|
config: tr
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 69.87222595830532
|
|
|
- type: f1
|
|
|
value: 66.03679033708141
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (ur)
|
|
|
config: ur
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 64.04505716207127
|
|
|
- type: f1
|
|
|
value: 61.28569169817908
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (vi)
|
|
|
config: vi
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 69.38466711499663
|
|
|
- type: f1
|
|
|
value: 67.20532357036844
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (zh-CN)
|
|
|
config: zh-CN
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 71.12306657700067
|
|
|
- type: f1
|
|
|
value: 68.91251226588182
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_intent
|
|
|
name: MTEB MassiveIntentClassification (zh-TW)
|
|
|
config: zh-TW
|
|
|
split: test
|
|
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 66.20040349697378
|
|
|
- type: f1
|
|
|
value: 66.02657347714175
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (af)
|
|
|
config: af
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 68.73907195696032
|
|
|
- type: f1
|
|
|
value: 66.98484521791418
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (am)
|
|
|
config: am
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 60.58843308675185
|
|
|
- type: f1
|
|
|
value: 58.95591723092005
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (ar)
|
|
|
config: ar
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 66.22730329522528
|
|
|
- type: f1
|
|
|
value: 66.0894499712115
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (az)
|
|
|
config: az
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 66.48285137861465
|
|
|
- type: f1
|
|
|
value: 65.21963176785157
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (bn)
|
|
|
config: bn
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 67.74714189643578
|
|
|
- type: f1
|
|
|
value: 66.8212192745412
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (cy)
|
|
|
config: cy
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 59.09213180901143
|
|
|
- type: f1
|
|
|
value: 56.70735546356339
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (da)
|
|
|
config: da
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 75.05716207128448
|
|
|
- type: f1
|
|
|
value: 74.8413712365364
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (de)
|
|
|
config: de
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 74.69737726967047
|
|
|
- type: f1
|
|
|
value: 74.7664341963
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (el)
|
|
|
config: el
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 73.90383322125084
|
|
|
- type: f1
|
|
|
value: 73.59201554448323
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (en)
|
|
|
config: en
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 77.51176866173503
|
|
|
- type: f1
|
|
|
value: 77.46104434577758
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (es)
|
|
|
config: es
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 74.31069266980496
|
|
|
- type: f1
|
|
|
value: 74.61048660675635
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (fa)
|
|
|
config: fa
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 72.95225285810356
|
|
|
- type: f1
|
|
|
value: 72.33160006574627
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (fi)
|
|
|
config: fi
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 73.12373907195696
|
|
|
- type: f1
|
|
|
value: 73.20921012557481
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (fr)
|
|
|
config: fr
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 73.86684599865501
|
|
|
- type: f1
|
|
|
value: 73.82348774610831
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (he)
|
|
|
config: he
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 71.40215198386012
|
|
|
- type: f1
|
|
|
value: 71.11945183971858
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (hi)
|
|
|
config: hi
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 72.12844653665098
|
|
|
- type: f1
|
|
|
value: 71.34450495911766
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (hu)
|
|
|
config: hu
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 74.52252858103566
|
|
|
- type: f1
|
|
|
value: 73.98878711342999
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (hy)
|
|
|
config: hy
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 64.93611297915265
|
|
|
- type: f1
|
|
|
value: 63.723200467653385
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (id)
|
|
|
config: id
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 74.11903160726295
|
|
|
- type: f1
|
|
|
value: 73.82138439467096
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (is)
|
|
|
config: is
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 67.15198386012105
|
|
|
- type: f1
|
|
|
value: 66.02172193802167
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (it)
|
|
|
config: it
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 74.32414256893072
|
|
|
- type: f1
|
|
|
value: 74.30943421170574
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (ja)
|
|
|
config: ja
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 77.46805648957633
|
|
|
- type: f1
|
|
|
value: 77.62808409298209
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (jv)
|
|
|
config: jv
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 63.318762609280434
|
|
|
- type: f1
|
|
|
value: 62.094284066075076
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (ka)
|
|
|
config: ka
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 58.34902488231338
|
|
|
- type: f1
|
|
|
value: 57.12893860987984
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (km)
|
|
|
config: km
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 50.88433086751849
|
|
|
- type: f1
|
|
|
value: 48.2272350802058
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (kn)
|
|
|
config: kn
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 66.4425016812374
|
|
|
- type: f1
|
|
|
value: 64.61463095996173
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (ko)
|
|
|
config: ko
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 75.04707464694015
|
|
|
- type: f1
|
|
|
value: 75.05099199098998
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (lv)
|
|
|
config: lv
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 70.50437121721586
|
|
|
- type: f1
|
|
|
value: 69.83397721096314
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (ml)
|
|
|
config: ml
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 69.94283792871553
|
|
|
- type: f1
|
|
|
value: 68.8704663703913
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (mn)
|
|
|
config: mn
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 64.79488903833222
|
|
|
- type: f1
|
|
|
value: 63.615424063345436
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (ms)
|
|
|
config: ms
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 69.88231338264963
|
|
|
- type: f1
|
|
|
value: 68.57892302593237
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (my)
|
|
|
config: my
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 63.248150638870214
|
|
|
- type: f1
|
|
|
value: 61.06680605338809
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (nb)
|
|
|
config: nb
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 74.84196368527236
|
|
|
- type: f1
|
|
|
value: 74.52566464968763
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (nl)
|
|
|
config: nl
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 74.8285137861466
|
|
|
- type: f1
|
|
|
value: 74.8853197608802
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (pl)
|
|
|
config: pl
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 74.13248150638869
|
|
|
- type: f1
|
|
|
value: 74.3982040999179
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (pt)
|
|
|
config: pt
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 73.49024882313383
|
|
|
- type: f1
|
|
|
value: 73.82153848368573
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (ro)
|
|
|
config: ro
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 71.72158708809684
|
|
|
- type: f1
|
|
|
value: 71.85049433180541
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (ru)
|
|
|
config: ru
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 75.137861466039
|
|
|
- type: f1
|
|
|
value: 75.37628348188467
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (sl)
|
|
|
config: sl
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 71.86953597848016
|
|
|
- type: f1
|
|
|
value: 71.87537624521661
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (sq)
|
|
|
config: sq
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 70.27572293207801
|
|
|
- type: f1
|
|
|
value: 68.80017302344231
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (sv)
|
|
|
config: sv
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 76.09952925353059
|
|
|
- type: f1
|
|
|
value: 76.07992707688408
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (sw)
|
|
|
config: sw
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 63.140551445864155
|
|
|
- type: f1
|
|
|
value: 61.73855010331415
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (ta)
|
|
|
config: ta
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 66.27774041694687
|
|
|
- type: f1
|
|
|
value: 64.83664868894539
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (te)
|
|
|
config: te
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 66.69468728984533
|
|
|
- type: f1
|
|
|
value: 64.76239666920868
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (th)
|
|
|
config: th
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 73.44653665097512
|
|
|
- type: f1
|
|
|
value: 73.14646052013873
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (tl)
|
|
|
config: tl
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 67.71351714862139
|
|
|
- type: f1
|
|
|
value: 66.67212180163382
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (tr)
|
|
|
config: tr
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 73.9946200403497
|
|
|
- type: f1
|
|
|
value: 73.87348793725525
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (ur)
|
|
|
config: ur
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 68.15400134498992
|
|
|
- type: f1
|
|
|
value: 67.09433241421094
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (vi)
|
|
|
config: vi
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 73.11365164761264
|
|
|
- type: f1
|
|
|
value: 73.59502539433753
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (zh-CN)
|
|
|
config: zh-CN
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 76.82582380632145
|
|
|
- type: f1
|
|
|
value: 76.89992945316313
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/amazon_massive_scenario
|
|
|
name: MTEB MassiveScenarioClassification (zh-TW)
|
|
|
config: zh-TW
|
|
|
split: test
|
|
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 71.81237390719569
|
|
|
- type: f1
|
|
|
value: 72.36499770986265
|
|
|
- task:
|
|
|
type: Clustering
|
|
|
dataset:
|
|
|
type: mteb/medrxiv-clustering-p2p
|
|
|
name: MTEB MedrxivClusteringP2P
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73
|
|
|
metrics:
|
|
|
- type: v_measure
|
|
|
value: 31.480506569594695
|
|
|
- task:
|
|
|
type: Clustering
|
|
|
dataset:
|
|
|
type: mteb/medrxiv-clustering-s2s
|
|
|
name: MTEB MedrxivClusteringS2S
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: 35191c8c0dca72d8ff3efcd72aa802307d469663
|
|
|
metrics:
|
|
|
- type: v_measure
|
|
|
value: 29.71252128004552
|
|
|
- task:
|
|
|
type: Reranking
|
|
|
dataset:
|
|
|
type: mteb/mind_small
|
|
|
name: MTEB MindSmallReranking
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69
|
|
|
metrics:
|
|
|
- type: map
|
|
|
value: 31.421396787056548
|
|
|
- type: mrr
|
|
|
value: 32.48155274872267
|
|
|
- task:
|
|
|
type: Retrieval
|
|
|
dataset:
|
|
|
type: nfcorpus
|
|
|
name: MTEB NFCorpus
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: None
|
|
|
metrics:
|
|
|
- type: map_at_1
|
|
|
value: 5.595
|
|
|
- type: map_at_10
|
|
|
value: 12.642000000000001
|
|
|
- type: map_at_100
|
|
|
value: 15.726
|
|
|
- type: map_at_1000
|
|
|
value: 17.061999999999998
|
|
|
- type: map_at_3
|
|
|
value: 9.125
|
|
|
- type: map_at_5
|
|
|
value: 10.866000000000001
|
|
|
- type: mrr_at_1
|
|
|
value: 43.344
|
|
|
- type: mrr_at_10
|
|
|
value: 52.227999999999994
|
|
|
- type: mrr_at_100
|
|
|
value: 52.898999999999994
|
|
|
- type: mrr_at_1000
|
|
|
value: 52.944
|
|
|
- type: mrr_at_3
|
|
|
value: 49.845
|
|
|
- type: mrr_at_5
|
|
|
value: 51.115
|
|
|
- type: ndcg_at_1
|
|
|
value: 41.949999999999996
|
|
|
- type: ndcg_at_10
|
|
|
value: 33.995
|
|
|
- type: ndcg_at_100
|
|
|
value: 30.869999999999997
|
|
|
- type: ndcg_at_1000
|
|
|
value: 39.487
|
|
|
- type: ndcg_at_3
|
|
|
value: 38.903999999999996
|
|
|
- type: ndcg_at_5
|
|
|
value: 37.236999999999995
|
|
|
- type: precision_at_1
|
|
|
value: 43.344
|
|
|
- type: precision_at_10
|
|
|
value: 25.480000000000004
|
|
|
- type: precision_at_100
|
|
|
value: 7.672
|
|
|
- type: precision_at_1000
|
|
|
value: 2.028
|
|
|
- type: precision_at_3
|
|
|
value: 36.636
|
|
|
- type: precision_at_5
|
|
|
value: 32.632
|
|
|
- type: recall_at_1
|
|
|
value: 5.595
|
|
|
- type: recall_at_10
|
|
|
value: 16.466
|
|
|
- type: recall_at_100
|
|
|
value: 31.226
|
|
|
- type: recall_at_1000
|
|
|
value: 62.778999999999996
|
|
|
- type: recall_at_3
|
|
|
value: 9.931
|
|
|
- type: recall_at_5
|
|
|
value: 12.884
|
|
|
- task:
|
|
|
type: Retrieval
|
|
|
dataset:
|
|
|
type: nq
|
|
|
name: MTEB NQ
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: None
|
|
|
metrics:
|
|
|
- type: map_at_1
|
|
|
value: 40.414
|
|
|
- type: map_at_10
|
|
|
value: 56.754000000000005
|
|
|
- type: map_at_100
|
|
|
value: 57.457
|
|
|
- type: map_at_1000
|
|
|
value: 57.477999999999994
|
|
|
- type: map_at_3
|
|
|
value: 52.873999999999995
|
|
|
- type: map_at_5
|
|
|
value: 55.175
|
|
|
- type: mrr_at_1
|
|
|
value: 45.278
|
|
|
- type: mrr_at_10
|
|
|
value: 59.192
|
|
|
- type: mrr_at_100
|
|
|
value: 59.650000000000006
|
|
|
- type: mrr_at_1000
|
|
|
value: 59.665
|
|
|
- type: mrr_at_3
|
|
|
value: 56.141
|
|
|
- type: mrr_at_5
|
|
|
value: 57.998000000000005
|
|
|
- type: ndcg_at_1
|
|
|
value: 45.278
|
|
|
- type: ndcg_at_10
|
|
|
value: 64.056
|
|
|
- type: ndcg_at_100
|
|
|
value: 66.89
|
|
|
- type: ndcg_at_1000
|
|
|
value: 67.364
|
|
|
- type: ndcg_at_3
|
|
|
value: 56.97
|
|
|
- type: ndcg_at_5
|
|
|
value: 60.719
|
|
|
- type: precision_at_1
|
|
|
value: 45.278
|
|
|
- type: precision_at_10
|
|
|
value: 9.994
|
|
|
- type: precision_at_100
|
|
|
value: 1.165
|
|
|
- type: precision_at_1000
|
|
|
value: 0.121
|
|
|
- type: precision_at_3
|
|
|
value: 25.512
|
|
|
- type: precision_at_5
|
|
|
value: 17.509
|
|
|
- type: recall_at_1
|
|
|
value: 40.414
|
|
|
- type: recall_at_10
|
|
|
value: 83.596
|
|
|
- type: recall_at_100
|
|
|
value: 95.72
|
|
|
- type: recall_at_1000
|
|
|
value: 99.24
|
|
|
- type: recall_at_3
|
|
|
value: 65.472
|
|
|
- type: recall_at_5
|
|
|
value: 74.039
|
|
|
- task:
|
|
|
type: Retrieval
|
|
|
dataset:
|
|
|
type: quora
|
|
|
name: MTEB QuoraRetrieval
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: None
|
|
|
metrics:
|
|
|
- type: map_at_1
|
|
|
value: 70.352
|
|
|
- type: map_at_10
|
|
|
value: 84.369
|
|
|
- type: map_at_100
|
|
|
value: 85.02499999999999
|
|
|
- type: map_at_1000
|
|
|
value: 85.04
|
|
|
- type: map_at_3
|
|
|
value: 81.42399999999999
|
|
|
- type: map_at_5
|
|
|
value: 83.279
|
|
|
- type: mrr_at_1
|
|
|
value: 81.05
|
|
|
- type: mrr_at_10
|
|
|
value: 87.401
|
|
|
- type: mrr_at_100
|
|
|
value: 87.504
|
|
|
- type: mrr_at_1000
|
|
|
value: 87.505
|
|
|
- type: mrr_at_3
|
|
|
value: 86.443
|
|
|
- type: mrr_at_5
|
|
|
value: 87.10799999999999
|
|
|
- type: ndcg_at_1
|
|
|
value: 81.04
|
|
|
- type: ndcg_at_10
|
|
|
value: 88.181
|
|
|
- type: ndcg_at_100
|
|
|
value: 89.411
|
|
|
- type: ndcg_at_1000
|
|
|
value: 89.507
|
|
|
- type: ndcg_at_3
|
|
|
value: 85.28099999999999
|
|
|
- type: ndcg_at_5
|
|
|
value: 86.888
|
|
|
- type: precision_at_1
|
|
|
value: 81.04
|
|
|
- type: precision_at_10
|
|
|
value: 13.406
|
|
|
- type: precision_at_100
|
|
|
value: 1.5350000000000001
|
|
|
- type: precision_at_1000
|
|
|
value: 0.157
|
|
|
- type: precision_at_3
|
|
|
value: 37.31
|
|
|
- type: precision_at_5
|
|
|
value: 24.54
|
|
|
- type: recall_at_1
|
|
|
value: 70.352
|
|
|
- type: recall_at_10
|
|
|
value: 95.358
|
|
|
- type: recall_at_100
|
|
|
value: 99.541
|
|
|
- type: recall_at_1000
|
|
|
value: 99.984
|
|
|
- type: recall_at_3
|
|
|
value: 87.111
|
|
|
- type: recall_at_5
|
|
|
value: 91.643
|
|
|
- task:
|
|
|
type: Clustering
|
|
|
dataset:
|
|
|
type: mteb/reddit-clustering
|
|
|
name: MTEB RedditClustering
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: 24640382cdbf8abc73003fb0fa6d111a705499eb
|
|
|
metrics:
|
|
|
- type: v_measure
|
|
|
value: 46.54068723291946
|
|
|
- task:
|
|
|
type: Clustering
|
|
|
dataset:
|
|
|
type: mteb/reddit-clustering-p2p
|
|
|
name: MTEB RedditClusteringP2P
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: 282350215ef01743dc01b456c7f5241fa8937f16
|
|
|
metrics:
|
|
|
- type: v_measure
|
|
|
value: 63.216287629895994
|
|
|
- task:
|
|
|
type: Retrieval
|
|
|
dataset:
|
|
|
type: scidocs
|
|
|
name: MTEB SCIDOCS
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: None
|
|
|
metrics:
|
|
|
- type: map_at_1
|
|
|
value: 4.023000000000001
|
|
|
- type: map_at_10
|
|
|
value: 10.071
|
|
|
- type: map_at_100
|
|
|
value: 11.892
|
|
|
- type: map_at_1000
|
|
|
value: 12.196
|
|
|
- type: map_at_3
|
|
|
value: 7.234
|
|
|
- type: map_at_5
|
|
|
value: 8.613999999999999
|
|
|
- type: mrr_at_1
|
|
|
value: 19.900000000000002
|
|
|
- type: mrr_at_10
|
|
|
value: 30.516
|
|
|
- type: mrr_at_100
|
|
|
value: 31.656000000000002
|
|
|
- type: mrr_at_1000
|
|
|
value: 31.723000000000003
|
|
|
- type: mrr_at_3
|
|
|
value: 27.400000000000002
|
|
|
- type: mrr_at_5
|
|
|
value: 29.270000000000003
|
|
|
- type: ndcg_at_1
|
|
|
value: 19.900000000000002
|
|
|
- type: ndcg_at_10
|
|
|
value: 17.474
|
|
|
- type: ndcg_at_100
|
|
|
value: 25.020999999999997
|
|
|
- type: ndcg_at_1000
|
|
|
value: 30.728
|
|
|
- type: ndcg_at_3
|
|
|
value: 16.588
|
|
|
- type: ndcg_at_5
|
|
|
value: 14.498
|
|
|
- type: precision_at_1
|
|
|
value: 19.900000000000002
|
|
|
- type: precision_at_10
|
|
|
value: 9.139999999999999
|
|
|
- type: precision_at_100
|
|
|
value: 2.011
|
|
|
- type: precision_at_1000
|
|
|
value: 0.33899999999999997
|
|
|
- type: precision_at_3
|
|
|
value: 15.667
|
|
|
- type: precision_at_5
|
|
|
value: 12.839999999999998
|
|
|
- type: recall_at_1
|
|
|
value: 4.023000000000001
|
|
|
- type: recall_at_10
|
|
|
value: 18.497
|
|
|
- type: recall_at_100
|
|
|
value: 40.8
|
|
|
- type: recall_at_1000
|
|
|
value: 68.812
|
|
|
- type: recall_at_3
|
|
|
value: 9.508
|
|
|
- type: recall_at_5
|
|
|
value: 12.983
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sickr-sts
|
|
|
name: MTEB SICK-R
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: a6ea5a8cab320b040a23452cc28066d9beae2cee
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 83.967008785134
|
|
|
- type: cos_sim_spearman
|
|
|
value: 80.23142141101837
|
|
|
- type: euclidean_pearson
|
|
|
value: 81.20166064704539
|
|
|
- type: euclidean_spearman
|
|
|
value: 80.18961335654585
|
|
|
- type: manhattan_pearson
|
|
|
value: 81.13925443187625
|
|
|
- type: manhattan_spearman
|
|
|
value: 80.07948723044424
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts12-sts
|
|
|
name: MTEB STS12
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: a0d554a64d88156834ff5ae9920b964011b16384
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 86.94262461316023
|
|
|
- type: cos_sim_spearman
|
|
|
value: 80.01596278563865
|
|
|
- type: euclidean_pearson
|
|
|
value: 83.80799622922581
|
|
|
- type: euclidean_spearman
|
|
|
value: 79.94984954947103
|
|
|
- type: manhattan_pearson
|
|
|
value: 83.68473841756281
|
|
|
- type: manhattan_spearman
|
|
|
value: 79.84990707951822
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts13-sts
|
|
|
name: MTEB STS13
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 80.57346443146068
|
|
|
- type: cos_sim_spearman
|
|
|
value: 81.54689837570866
|
|
|
- type: euclidean_pearson
|
|
|
value: 81.10909881516007
|
|
|
- type: euclidean_spearman
|
|
|
value: 81.56746243261762
|
|
|
- type: manhattan_pearson
|
|
|
value: 80.87076036186582
|
|
|
- type: manhattan_spearman
|
|
|
value: 81.33074987964402
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts14-sts
|
|
|
name: MTEB STS14
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 79.54733787179849
|
|
|
- type: cos_sim_spearman
|
|
|
value: 77.72202105610411
|
|
|
- type: euclidean_pearson
|
|
|
value: 78.9043595478849
|
|
|
- type: euclidean_spearman
|
|
|
value: 77.93422804309435
|
|
|
- type: manhattan_pearson
|
|
|
value: 78.58115121621368
|
|
|
- type: manhattan_spearman
|
|
|
value: 77.62508135122033
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts15-sts
|
|
|
name: MTEB STS15
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 88.59880017237558
|
|
|
- type: cos_sim_spearman
|
|
|
value: 89.31088630824758
|
|
|
- type: euclidean_pearson
|
|
|
value: 88.47069261564656
|
|
|
- type: euclidean_spearman
|
|
|
value: 89.33581971465233
|
|
|
- type: manhattan_pearson
|
|
|
value: 88.40774264100956
|
|
|
- type: manhattan_spearman
|
|
|
value: 89.28657485627835
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts16-sts
|
|
|
name: MTEB STS16
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: 4d8694f8f0e0100860b497b999b3dbed754a0513
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 84.08055117917084
|
|
|
- type: cos_sim_spearman
|
|
|
value: 85.78491813080304
|
|
|
- type: euclidean_pearson
|
|
|
value: 84.99329155500392
|
|
|
- type: euclidean_spearman
|
|
|
value: 85.76728064677287
|
|
|
- type: manhattan_pearson
|
|
|
value: 84.87947428989587
|
|
|
- type: manhattan_spearman
|
|
|
value: 85.62429454917464
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts17-crosslingual-sts
|
|
|
name: MTEB STS17 (ko-ko)
|
|
|
config: ko-ko
|
|
|
split: test
|
|
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 82.14190939287384
|
|
|
- type: cos_sim_spearman
|
|
|
value: 82.27331573306041
|
|
|
- type: euclidean_pearson
|
|
|
value: 81.891896953716
|
|
|
- type: euclidean_spearman
|
|
|
value: 82.37695542955998
|
|
|
- type: manhattan_pearson
|
|
|
value: 81.73123869460504
|
|
|
- type: manhattan_spearman
|
|
|
value: 82.19989168441421
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts17-crosslingual-sts
|
|
|
name: MTEB STS17 (ar-ar)
|
|
|
config: ar-ar
|
|
|
split: test
|
|
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 76.84695301843362
|
|
|
- type: cos_sim_spearman
|
|
|
value: 77.87790986014461
|
|
|
- type: euclidean_pearson
|
|
|
value: 76.91981583106315
|
|
|
- type: euclidean_spearman
|
|
|
value: 77.88154772749589
|
|
|
- type: manhattan_pearson
|
|
|
value: 76.94953277451093
|
|
|
- type: manhattan_spearman
|
|
|
value: 77.80499230728604
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts17-crosslingual-sts
|
|
|
name: MTEB STS17 (en-ar)
|
|
|
config: en-ar
|
|
|
split: test
|
|
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 75.44657840482016
|
|
|
- type: cos_sim_spearman
|
|
|
value: 75.05531095119674
|
|
|
- type: euclidean_pearson
|
|
|
value: 75.88161755829299
|
|
|
- type: euclidean_spearman
|
|
|
value: 74.73176238219332
|
|
|
- type: manhattan_pearson
|
|
|
value: 75.63984765635362
|
|
|
- type: manhattan_spearman
|
|
|
value: 74.86476440770737
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts17-crosslingual-sts
|
|
|
name: MTEB STS17 (en-de)
|
|
|
config: en-de
|
|
|
split: test
|
|
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 85.64700140524133
|
|
|
- type: cos_sim_spearman
|
|
|
value: 86.16014210425672
|
|
|
- type: euclidean_pearson
|
|
|
value: 86.49086860843221
|
|
|
- type: euclidean_spearman
|
|
|
value: 86.09729326815614
|
|
|
- type: manhattan_pearson
|
|
|
value: 86.43406265125513
|
|
|
- type: manhattan_spearman
|
|
|
value: 86.17740150939994
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts17-crosslingual-sts
|
|
|
name: MTEB STS17 (en-en)
|
|
|
config: en-en
|
|
|
split: test
|
|
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 87.91170098764921
|
|
|
- type: cos_sim_spearman
|
|
|
value: 88.12437004058931
|
|
|
- type: euclidean_pearson
|
|
|
value: 88.81828254494437
|
|
|
- type: euclidean_spearman
|
|
|
value: 88.14831794572122
|
|
|
- type: manhattan_pearson
|
|
|
value: 88.93442183448961
|
|
|
- type: manhattan_spearman
|
|
|
value: 88.15254630778304
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts17-crosslingual-sts
|
|
|
name: MTEB STS17 (en-tr)
|
|
|
config: en-tr
|
|
|
split: test
|
|
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 72.91390577997292
|
|
|
- type: cos_sim_spearman
|
|
|
value: 71.22979457536074
|
|
|
- type: euclidean_pearson
|
|
|
value: 74.40314008106749
|
|
|
- type: euclidean_spearman
|
|
|
value: 72.54972136083246
|
|
|
- type: manhattan_pearson
|
|
|
value: 73.85687539530218
|
|
|
- type: manhattan_spearman
|
|
|
value: 72.09500771742637
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts17-crosslingual-sts
|
|
|
name: MTEB STS17 (es-en)
|
|
|
config: es-en
|
|
|
split: test
|
|
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 80.9301067983089
|
|
|
- type: cos_sim_spearman
|
|
|
value: 80.74989828346473
|
|
|
- type: euclidean_pearson
|
|
|
value: 81.36781301814257
|
|
|
- type: euclidean_spearman
|
|
|
value: 80.9448819964426
|
|
|
- type: manhattan_pearson
|
|
|
value: 81.0351322685609
|
|
|
- type: manhattan_spearman
|
|
|
value: 80.70192121844177
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts17-crosslingual-sts
|
|
|
name: MTEB STS17 (es-es)
|
|
|
config: es-es
|
|
|
split: test
|
|
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 87.13820465980005
|
|
|
- type: cos_sim_spearman
|
|
|
value: 86.73532498758757
|
|
|
- type: euclidean_pearson
|
|
|
value: 87.21329451846637
|
|
|
- type: euclidean_spearman
|
|
|
value: 86.57863198601002
|
|
|
- type: manhattan_pearson
|
|
|
value: 87.06973713818554
|
|
|
- type: manhattan_spearman
|
|
|
value: 86.47534918791499
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts17-crosslingual-sts
|
|
|
name: MTEB STS17 (fr-en)
|
|
|
config: fr-en
|
|
|
split: test
|
|
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 85.48720108904415
|
|
|
- type: cos_sim_spearman
|
|
|
value: 85.62221757068387
|
|
|
- type: euclidean_pearson
|
|
|
value: 86.1010129512749
|
|
|
- type: euclidean_spearman
|
|
|
value: 85.86580966509942
|
|
|
- type: manhattan_pearson
|
|
|
value: 86.26800938808971
|
|
|
- type: manhattan_spearman
|
|
|
value: 85.88902721678429
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts17-crosslingual-sts
|
|
|
name: MTEB STS17 (it-en)
|
|
|
config: it-en
|
|
|
split: test
|
|
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 83.98021347333516
|
|
|
- type: cos_sim_spearman
|
|
|
value: 84.53806553803501
|
|
|
- type: euclidean_pearson
|
|
|
value: 84.61483347248364
|
|
|
- type: euclidean_spearman
|
|
|
value: 85.14191408011702
|
|
|
- type: manhattan_pearson
|
|
|
value: 84.75297588825967
|
|
|
- type: manhattan_spearman
|
|
|
value: 85.33176753669242
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts17-crosslingual-sts
|
|
|
name: MTEB STS17 (nl-en)
|
|
|
config: nl-en
|
|
|
split: test
|
|
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 84.51856644893233
|
|
|
- type: cos_sim_spearman
|
|
|
value: 85.27510748506413
|
|
|
- type: euclidean_pearson
|
|
|
value: 85.09886861540977
|
|
|
- type: euclidean_spearman
|
|
|
value: 85.62579245860887
|
|
|
- type: manhattan_pearson
|
|
|
value: 84.93017860464607
|
|
|
- type: manhattan_spearman
|
|
|
value: 85.5063988898453
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts22-crosslingual-sts
|
|
|
name: MTEB STS22 (en)
|
|
|
config: en
|
|
|
split: test
|
|
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 62.581573200584195
|
|
|
- type: cos_sim_spearman
|
|
|
value: 63.05503590247928
|
|
|
- type: euclidean_pearson
|
|
|
value: 63.652564812602094
|
|
|
- type: euclidean_spearman
|
|
|
value: 62.64811520876156
|
|
|
- type: manhattan_pearson
|
|
|
value: 63.506842893061076
|
|
|
- type: manhattan_spearman
|
|
|
value: 62.51289573046917
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts22-crosslingual-sts
|
|
|
name: MTEB STS22 (de)
|
|
|
config: de
|
|
|
split: test
|
|
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 48.2248801729127
|
|
|
- type: cos_sim_spearman
|
|
|
value: 56.5936604678561
|
|
|
- type: euclidean_pearson
|
|
|
value: 43.98149464089
|
|
|
- type: euclidean_spearman
|
|
|
value: 56.108561882423615
|
|
|
- type: manhattan_pearson
|
|
|
value: 43.86880305903564
|
|
|
- type: manhattan_spearman
|
|
|
value: 56.04671150510166
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts22-crosslingual-sts
|
|
|
name: MTEB STS22 (es)
|
|
|
config: es
|
|
|
split: test
|
|
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 55.17564527009831
|
|
|
- type: cos_sim_spearman
|
|
|
value: 64.57978560979488
|
|
|
- type: euclidean_pearson
|
|
|
value: 58.8818330154583
|
|
|
- type: euclidean_spearman
|
|
|
value: 64.99214839071281
|
|
|
- type: manhattan_pearson
|
|
|
value: 58.72671436121381
|
|
|
- type: manhattan_spearman
|
|
|
value: 65.10713416616109
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts22-crosslingual-sts
|
|
|
name: MTEB STS22 (pl)
|
|
|
config: pl
|
|
|
split: test
|
|
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 26.772131864023297
|
|
|
- type: cos_sim_spearman
|
|
|
value: 34.68200792408681
|
|
|
- type: euclidean_pearson
|
|
|
value: 16.68082419005441
|
|
|
- type: euclidean_spearman
|
|
|
value: 34.83099932652166
|
|
|
- type: manhattan_pearson
|
|
|
value: 16.52605949659529
|
|
|
- type: manhattan_spearman
|
|
|
value: 34.82075801399475
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts22-crosslingual-sts
|
|
|
name: MTEB STS22 (tr)
|
|
|
config: tr
|
|
|
split: test
|
|
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 54.42415189043831
|
|
|
- type: cos_sim_spearman
|
|
|
value: 63.54594264576758
|
|
|
- type: euclidean_pearson
|
|
|
value: 57.36577498297745
|
|
|
- type: euclidean_spearman
|
|
|
value: 63.111466379158074
|
|
|
- type: manhattan_pearson
|
|
|
value: 57.584543715873885
|
|
|
- type: manhattan_spearman
|
|
|
value: 63.22361054139183
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts22-crosslingual-sts
|
|
|
name: MTEB STS22 (ar)
|
|
|
config: ar
|
|
|
split: test
|
|
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 47.55216762405518
|
|
|
- type: cos_sim_spearman
|
|
|
value: 56.98670142896412
|
|
|
- type: euclidean_pearson
|
|
|
value: 50.15318757562699
|
|
|
- type: euclidean_spearman
|
|
|
value: 56.524941926541906
|
|
|
- type: manhattan_pearson
|
|
|
value: 49.955618528674904
|
|
|
- type: manhattan_spearman
|
|
|
value: 56.37102209240117
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts22-crosslingual-sts
|
|
|
name: MTEB STS22 (ru)
|
|
|
config: ru
|
|
|
split: test
|
|
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 49.20540980338571
|
|
|
- type: cos_sim_spearman
|
|
|
value: 59.9009453504406
|
|
|
- type: euclidean_pearson
|
|
|
value: 49.557749853620535
|
|
|
- type: euclidean_spearman
|
|
|
value: 59.76631621172456
|
|
|
- type: manhattan_pearson
|
|
|
value: 49.62340591181147
|
|
|
- type: manhattan_spearman
|
|
|
value: 59.94224880322436
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts22-crosslingual-sts
|
|
|
name: MTEB STS22 (zh)
|
|
|
config: zh
|
|
|
split: test
|
|
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 51.508169956576985
|
|
|
- type: cos_sim_spearman
|
|
|
value: 66.82461565306046
|
|
|
- type: euclidean_pearson
|
|
|
value: 56.2274426480083
|
|
|
- type: euclidean_spearman
|
|
|
value: 66.6775323848333
|
|
|
- type: manhattan_pearson
|
|
|
value: 55.98277796300661
|
|
|
- type: manhattan_spearman
|
|
|
value: 66.63669848497175
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts22-crosslingual-sts
|
|
|
name: MTEB STS22 (fr)
|
|
|
config: fr
|
|
|
split: test
|
|
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 72.86478788045507
|
|
|
- type: cos_sim_spearman
|
|
|
value: 76.7946552053193
|
|
|
- type: euclidean_pearson
|
|
|
value: 75.01598530490269
|
|
|
- type: euclidean_spearman
|
|
|
value: 76.83618917858281
|
|
|
- type: manhattan_pearson
|
|
|
value: 74.68337628304332
|
|
|
- type: manhattan_spearman
|
|
|
value: 76.57480204017773
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts22-crosslingual-sts
|
|
|
name: MTEB STS22 (de-en)
|
|
|
config: de-en
|
|
|
split: test
|
|
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 55.922619099401984
|
|
|
- type: cos_sim_spearman
|
|
|
value: 56.599362477240774
|
|
|
- type: euclidean_pearson
|
|
|
value: 56.68307052369783
|
|
|
- type: euclidean_spearman
|
|
|
value: 54.28760436777401
|
|
|
- type: manhattan_pearson
|
|
|
value: 56.67763566500681
|
|
|
- type: manhattan_spearman
|
|
|
value: 53.94619541711359
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts22-crosslingual-sts
|
|
|
name: MTEB STS22 (es-en)
|
|
|
config: es-en
|
|
|
split: test
|
|
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 66.74357206710913
|
|
|
- type: cos_sim_spearman
|
|
|
value: 72.5208244925311
|
|
|
- type: euclidean_pearson
|
|
|
value: 67.49254562186032
|
|
|
- type: euclidean_spearman
|
|
|
value: 72.02469076238683
|
|
|
- type: manhattan_pearson
|
|
|
value: 67.45251772238085
|
|
|
- type: manhattan_spearman
|
|
|
value: 72.05538819984538
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts22-crosslingual-sts
|
|
|
name: MTEB STS22 (it)
|
|
|
config: it
|
|
|
split: test
|
|
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 71.25734330033191
|
|
|
- type: cos_sim_spearman
|
|
|
value: 76.98349083946823
|
|
|
- type: euclidean_pearson
|
|
|
value: 73.71642838667736
|
|
|
- type: euclidean_spearman
|
|
|
value: 77.01715504651384
|
|
|
- type: manhattan_pearson
|
|
|
value: 73.61712711868105
|
|
|
- type: manhattan_spearman
|
|
|
value: 77.01392571153896
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts22-crosslingual-sts
|
|
|
name: MTEB STS22 (pl-en)
|
|
|
config: pl-en
|
|
|
split: test
|
|
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 63.18215462781212
|
|
|
- type: cos_sim_spearman
|
|
|
value: 65.54373266117607
|
|
|
- type: euclidean_pearson
|
|
|
value: 64.54126095439005
|
|
|
- type: euclidean_spearman
|
|
|
value: 65.30410369102711
|
|
|
- type: manhattan_pearson
|
|
|
value: 63.50332221148234
|
|
|
- type: manhattan_spearman
|
|
|
value: 64.3455878104313
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts22-crosslingual-sts
|
|
|
name: MTEB STS22 (zh-en)
|
|
|
config: zh-en
|
|
|
split: test
|
|
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 62.30509221440029
|
|
|
- type: cos_sim_spearman
|
|
|
value: 65.99582704642478
|
|
|
- type: euclidean_pearson
|
|
|
value: 63.43818859884195
|
|
|
- type: euclidean_spearman
|
|
|
value: 66.83172582815764
|
|
|
- type: manhattan_pearson
|
|
|
value: 63.055779168508764
|
|
|
- type: manhattan_spearman
|
|
|
value: 65.49585020501449
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts22-crosslingual-sts
|
|
|
name: MTEB STS22 (es-it)
|
|
|
config: es-it
|
|
|
split: test
|
|
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 59.587830825340404
|
|
|
- type: cos_sim_spearman
|
|
|
value: 68.93467614588089
|
|
|
- type: euclidean_pearson
|
|
|
value: 62.3073527367404
|
|
|
- type: euclidean_spearman
|
|
|
value: 69.69758171553175
|
|
|
- type: manhattan_pearson
|
|
|
value: 61.9074580815789
|
|
|
- type: manhattan_spearman
|
|
|
value: 69.57696375597865
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts22-crosslingual-sts
|
|
|
name: MTEB STS22 (de-fr)
|
|
|
config: de-fr
|
|
|
split: test
|
|
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 57.143220125577066
|
|
|
- type: cos_sim_spearman
|
|
|
value: 67.78857859159226
|
|
|
- type: euclidean_pearson
|
|
|
value: 55.58225107923733
|
|
|
- type: euclidean_spearman
|
|
|
value: 67.80662907184563
|
|
|
- type: manhattan_pearson
|
|
|
value: 56.24953502726514
|
|
|
- type: manhattan_spearman
|
|
|
value: 67.98262125431616
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts22-crosslingual-sts
|
|
|
name: MTEB STS22 (de-pl)
|
|
|
config: de-pl
|
|
|
split: test
|
|
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 21.826928900322066
|
|
|
- type: cos_sim_spearman
|
|
|
value: 49.578506634400405
|
|
|
- type: euclidean_pearson
|
|
|
value: 27.939890138843214
|
|
|
- type: euclidean_spearman
|
|
|
value: 52.71950519136242
|
|
|
- type: manhattan_pearson
|
|
|
value: 26.39878683847546
|
|
|
- type: manhattan_spearman
|
|
|
value: 47.54609580342499
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/sts22-crosslingual-sts
|
|
|
name: MTEB STS22 (fr-pl)
|
|
|
config: fr-pl
|
|
|
split: test
|
|
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 57.27603854632001
|
|
|
- type: cos_sim_spearman
|
|
|
value: 50.709255283710995
|
|
|
- type: euclidean_pearson
|
|
|
value: 59.5419024445929
|
|
|
- type: euclidean_spearman
|
|
|
value: 50.709255283710995
|
|
|
- type: manhattan_pearson
|
|
|
value: 59.03256832438492
|
|
|
- type: manhattan_spearman
|
|
|
value: 61.97797868009122
|
|
|
- task:
|
|
|
type: STS
|
|
|
dataset:
|
|
|
type: mteb/stsbenchmark-sts
|
|
|
name: MTEB STSBenchmark
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 85.00757054859712
|
|
|
- type: cos_sim_spearman
|
|
|
value: 87.29283629622222
|
|
|
- type: euclidean_pearson
|
|
|
value: 86.54824171775536
|
|
|
- type: euclidean_spearman
|
|
|
value: 87.24364730491402
|
|
|
- type: manhattan_pearson
|
|
|
value: 86.5062156915074
|
|
|
- type: manhattan_spearman
|
|
|
value: 87.15052170378574
|
|
|
- task:
|
|
|
type: Reranking
|
|
|
dataset:
|
|
|
type: mteb/scidocs-reranking
|
|
|
name: MTEB SciDocsRR
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab
|
|
|
metrics:
|
|
|
- type: map
|
|
|
value: 82.03549357197389
|
|
|
- type: mrr
|
|
|
value: 95.05437645143527
|
|
|
- task:
|
|
|
type: Retrieval
|
|
|
dataset:
|
|
|
type: scifact
|
|
|
name: MTEB SciFact
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: None
|
|
|
metrics:
|
|
|
- type: map_at_1
|
|
|
value: 57.260999999999996
|
|
|
- type: map_at_10
|
|
|
value: 66.259
|
|
|
- type: map_at_100
|
|
|
value: 66.884
|
|
|
- type: map_at_1000
|
|
|
value: 66.912
|
|
|
- type: map_at_3
|
|
|
value: 63.685
|
|
|
- type: map_at_5
|
|
|
value: 65.35499999999999
|
|
|
- type: mrr_at_1
|
|
|
value: 60.333000000000006
|
|
|
- type: mrr_at_10
|
|
|
value: 67.5
|
|
|
- type: mrr_at_100
|
|
|
value: 68.013
|
|
|
- type: mrr_at_1000
|
|
|
value: 68.038
|
|
|
- type: mrr_at_3
|
|
|
value: 65.61099999999999
|
|
|
- type: mrr_at_5
|
|
|
value: 66.861
|
|
|
- type: ndcg_at_1
|
|
|
value: 60.333000000000006
|
|
|
- type: ndcg_at_10
|
|
|
value: 70.41
|
|
|
- type: ndcg_at_100
|
|
|
value: 73.10600000000001
|
|
|
- type: ndcg_at_1000
|
|
|
value: 73.846
|
|
|
- type: ndcg_at_3
|
|
|
value: 66.133
|
|
|
- type: ndcg_at_5
|
|
|
value: 68.499
|
|
|
- type: precision_at_1
|
|
|
value: 60.333000000000006
|
|
|
- type: precision_at_10
|
|
|
value: 9.232999999999999
|
|
|
- type: precision_at_100
|
|
|
value: 1.0630000000000002
|
|
|
- type: precision_at_1000
|
|
|
value: 0.11299999999999999
|
|
|
- type: precision_at_3
|
|
|
value: 25.667
|
|
|
- type: precision_at_5
|
|
|
value: 17.067
|
|
|
- type: recall_at_1
|
|
|
value: 57.260999999999996
|
|
|
- type: recall_at_10
|
|
|
value: 81.94399999999999
|
|
|
- type: recall_at_100
|
|
|
value: 93.867
|
|
|
- type: recall_at_1000
|
|
|
value: 99.667
|
|
|
- type: recall_at_3
|
|
|
value: 70.339
|
|
|
- type: recall_at_5
|
|
|
value: 76.25
|
|
|
- task:
|
|
|
type: PairClassification
|
|
|
dataset:
|
|
|
type: mteb/sprintduplicatequestions-pairclassification
|
|
|
name: MTEB SprintDuplicateQuestions
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46
|
|
|
metrics:
|
|
|
- type: cos_sim_accuracy
|
|
|
value: 99.74356435643564
|
|
|
- type: cos_sim_ap
|
|
|
value: 93.13411948212683
|
|
|
- type: cos_sim_f1
|
|
|
value: 86.80521991300147
|
|
|
- type: cos_sim_precision
|
|
|
value: 84.00374181478017
|
|
|
- type: cos_sim_recall
|
|
|
value: 89.8
|
|
|
- type: dot_accuracy
|
|
|
value: 99.67920792079208
|
|
|
- type: dot_ap
|
|
|
value: 89.27277565444479
|
|
|
- type: dot_f1
|
|
|
value: 83.9276990718124
|
|
|
- type: dot_precision
|
|
|
value: 82.04393505253104
|
|
|
- type: dot_recall
|
|
|
value: 85.9
|
|
|
- type: euclidean_accuracy
|
|
|
value: 99.74257425742574
|
|
|
- type: euclidean_ap
|
|
|
value: 93.17993008259062
|
|
|
- type: euclidean_f1
|
|
|
value: 86.69396110542476
|
|
|
- type: euclidean_precision
|
|
|
value: 88.78406708595388
|
|
|
- type: euclidean_recall
|
|
|
value: 84.7
|
|
|
- type: manhattan_accuracy
|
|
|
value: 99.74257425742574
|
|
|
- type: manhattan_ap
|
|
|
value: 93.14413755550099
|
|
|
- type: manhattan_f1
|
|
|
value: 86.82483594144371
|
|
|
- type: manhattan_precision
|
|
|
value: 87.66564729867483
|
|
|
- type: manhattan_recall
|
|
|
value: 86
|
|
|
- type: max_accuracy
|
|
|
value: 99.74356435643564
|
|
|
- type: max_ap
|
|
|
value: 93.17993008259062
|
|
|
- type: max_f1
|
|
|
value: 86.82483594144371
|
|
|
- task:
|
|
|
type: Clustering
|
|
|
dataset:
|
|
|
type: mteb/stackexchange-clustering
|
|
|
name: MTEB StackExchangeClustering
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259
|
|
|
metrics:
|
|
|
- type: v_measure
|
|
|
value: 57.525863806168566
|
|
|
- task:
|
|
|
type: Clustering
|
|
|
dataset:
|
|
|
type: mteb/stackexchange-clustering-p2p
|
|
|
name: MTEB StackExchangeClusteringP2P
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: 815ca46b2622cec33ccafc3735d572c266efdb44
|
|
|
metrics:
|
|
|
- type: v_measure
|
|
|
value: 32.68850574423839
|
|
|
- task:
|
|
|
type: Reranking
|
|
|
dataset:
|
|
|
type: mteb/stackoverflowdupquestions-reranking
|
|
|
name: MTEB StackOverflowDupQuestions
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: e185fbe320c72810689fc5848eb6114e1ef5ec69
|
|
|
metrics:
|
|
|
- type: map
|
|
|
value: 49.71580650644033
|
|
|
- type: mrr
|
|
|
value: 50.50971903913081
|
|
|
- task:
|
|
|
type: Summarization
|
|
|
dataset:
|
|
|
type: mteb/summeval
|
|
|
name: MTEB SummEval
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c
|
|
|
metrics:
|
|
|
- type: cos_sim_pearson
|
|
|
value: 29.152190498799484
|
|
|
- type: cos_sim_spearman
|
|
|
value: 29.686180371952727
|
|
|
- type: dot_pearson
|
|
|
value: 27.248664793816342
|
|
|
- type: dot_spearman
|
|
|
value: 28.37748983721745
|
|
|
- task:
|
|
|
type: Retrieval
|
|
|
dataset:
|
|
|
type: trec-covid
|
|
|
name: MTEB TRECCOVID
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: None
|
|
|
metrics:
|
|
|
- type: map_at_1
|
|
|
value: 0.20400000000000001
|
|
|
- type: map_at_10
|
|
|
value: 1.6209999999999998
|
|
|
- type: map_at_100
|
|
|
value: 9.690999999999999
|
|
|
- type: map_at_1000
|
|
|
value: 23.733
|
|
|
- type: map_at_3
|
|
|
value: 0.575
|
|
|
- type: map_at_5
|
|
|
value: 0.885
|
|
|
- type: mrr_at_1
|
|
|
value: 78
|
|
|
- type: mrr_at_10
|
|
|
value: 86.56700000000001
|
|
|
- type: mrr_at_100
|
|
|
value: 86.56700000000001
|
|
|
- type: mrr_at_1000
|
|
|
value: 86.56700000000001
|
|
|
- type: mrr_at_3
|
|
|
value: 85.667
|
|
|
- type: mrr_at_5
|
|
|
value: 86.56700000000001
|
|
|
- type: ndcg_at_1
|
|
|
value: 76
|
|
|
- type: ndcg_at_10
|
|
|
value: 71.326
|
|
|
- type: ndcg_at_100
|
|
|
value: 54.208999999999996
|
|
|
- type: ndcg_at_1000
|
|
|
value: 49.252
|
|
|
- type: ndcg_at_3
|
|
|
value: 74.235
|
|
|
- type: ndcg_at_5
|
|
|
value: 73.833
|
|
|
- type: precision_at_1
|
|
|
value: 78
|
|
|
- type: precision_at_10
|
|
|
value: 74.8
|
|
|
- type: precision_at_100
|
|
|
value: 55.50000000000001
|
|
|
- type: precision_at_1000
|
|
|
value: 21.836
|
|
|
- type: precision_at_3
|
|
|
value: 78
|
|
|
- type: precision_at_5
|
|
|
value: 78
|
|
|
- type: recall_at_1
|
|
|
value: 0.20400000000000001
|
|
|
- type: recall_at_10
|
|
|
value: 1.894
|
|
|
- type: recall_at_100
|
|
|
value: 13.245999999999999
|
|
|
- type: recall_at_1000
|
|
|
value: 46.373
|
|
|
- type: recall_at_3
|
|
|
value: 0.613
|
|
|
- type: recall_at_5
|
|
|
value: 0.991
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (sqi-eng)
|
|
|
config: sqi-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 95.89999999999999
|
|
|
- type: f1
|
|
|
value: 94.69999999999999
|
|
|
- type: precision
|
|
|
value: 94.11666666666667
|
|
|
- type: recall
|
|
|
value: 95.89999999999999
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (fry-eng)
|
|
|
config: fry-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 68.20809248554913
|
|
|
- type: f1
|
|
|
value: 63.431048720066066
|
|
|
- type: precision
|
|
|
value: 61.69143958161298
|
|
|
- type: recall
|
|
|
value: 68.20809248554913
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (kur-eng)
|
|
|
config: kur-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 71.21951219512195
|
|
|
- type: f1
|
|
|
value: 66.82926829268293
|
|
|
- type: precision
|
|
|
value: 65.1260162601626
|
|
|
- type: recall
|
|
|
value: 71.21951219512195
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (tur-eng)
|
|
|
config: tur-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 97.2
|
|
|
- type: f1
|
|
|
value: 96.26666666666667
|
|
|
- type: precision
|
|
|
value: 95.8
|
|
|
- type: recall
|
|
|
value: 97.2
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (deu-eng)
|
|
|
config: deu-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 99.3
|
|
|
- type: f1
|
|
|
value: 99.06666666666666
|
|
|
- type: precision
|
|
|
value: 98.95
|
|
|
- type: recall
|
|
|
value: 99.3
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (nld-eng)
|
|
|
config: nld-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 97.39999999999999
|
|
|
- type: f1
|
|
|
value: 96.63333333333333
|
|
|
- type: precision
|
|
|
value: 96.26666666666668
|
|
|
- type: recall
|
|
|
value: 97.39999999999999
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (ron-eng)
|
|
|
config: ron-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 96
|
|
|
- type: f1
|
|
|
value: 94.86666666666666
|
|
|
- type: precision
|
|
|
value: 94.31666666666668
|
|
|
- type: recall
|
|
|
value: 96
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (ang-eng)
|
|
|
config: ang-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 47.01492537313433
|
|
|
- type: f1
|
|
|
value: 40.178867566927266
|
|
|
- type: precision
|
|
|
value: 38.179295828549556
|
|
|
- type: recall
|
|
|
value: 47.01492537313433
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (ido-eng)
|
|
|
config: ido-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 86.5
|
|
|
- type: f1
|
|
|
value: 83.62537480063796
|
|
|
- type: precision
|
|
|
value: 82.44555555555554
|
|
|
- type: recall
|
|
|
value: 86.5
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (jav-eng)
|
|
|
config: jav-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 80.48780487804879
|
|
|
- type: f1
|
|
|
value: 75.45644599303138
|
|
|
- type: precision
|
|
|
value: 73.37398373983739
|
|
|
- type: recall
|
|
|
value: 80.48780487804879
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (isl-eng)
|
|
|
config: isl-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 93.7
|
|
|
- type: f1
|
|
|
value: 91.95666666666666
|
|
|
- type: precision
|
|
|
value: 91.125
|
|
|
- type: recall
|
|
|
value: 93.7
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (slv-eng)
|
|
|
config: slv-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 91.73754556500607
|
|
|
- type: f1
|
|
|
value: 89.65168084244632
|
|
|
- type: precision
|
|
|
value: 88.73025516403402
|
|
|
- type: recall
|
|
|
value: 91.73754556500607
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (cym-eng)
|
|
|
config: cym-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 81.04347826086956
|
|
|
- type: f1
|
|
|
value: 76.2128364389234
|
|
|
- type: precision
|
|
|
value: 74.2
|
|
|
- type: recall
|
|
|
value: 81.04347826086956
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (kaz-eng)
|
|
|
config: kaz-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 83.65217391304348
|
|
|
- type: f1
|
|
|
value: 79.4376811594203
|
|
|
- type: precision
|
|
|
value: 77.65797101449274
|
|
|
- type: recall
|
|
|
value: 83.65217391304348
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (est-eng)
|
|
|
config: est-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 87.5
|
|
|
- type: f1
|
|
|
value: 85.02690476190476
|
|
|
- type: precision
|
|
|
value: 83.96261904761904
|
|
|
- type: recall
|
|
|
value: 87.5
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (heb-eng)
|
|
|
config: heb-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 89.3
|
|
|
- type: f1
|
|
|
value: 86.52333333333333
|
|
|
- type: precision
|
|
|
value: 85.22833333333332
|
|
|
- type: recall
|
|
|
value: 89.3
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (gla-eng)
|
|
|
config: gla-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 65.01809408926418
|
|
|
- type: f1
|
|
|
value: 59.00594446432805
|
|
|
- type: precision
|
|
|
value: 56.827215807915444
|
|
|
- type: recall
|
|
|
value: 65.01809408926418
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (mar-eng)
|
|
|
config: mar-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 91.2
|
|
|
- type: f1
|
|
|
value: 88.58
|
|
|
- type: precision
|
|
|
value: 87.33333333333334
|
|
|
- type: recall
|
|
|
value: 91.2
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (lat-eng)
|
|
|
config: lat-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 59.199999999999996
|
|
|
- type: f1
|
|
|
value: 53.299166276284915
|
|
|
- type: precision
|
|
|
value: 51.3383908045977
|
|
|
- type: recall
|
|
|
value: 59.199999999999996
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (bel-eng)
|
|
|
config: bel-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 93.2
|
|
|
- type: f1
|
|
|
value: 91.2
|
|
|
- type: precision
|
|
|
value: 90.25
|
|
|
- type: recall
|
|
|
value: 93.2
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (pms-eng)
|
|
|
config: pms-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 64.76190476190476
|
|
|
- type: f1
|
|
|
value: 59.867110667110666
|
|
|
- type: precision
|
|
|
value: 58.07390192653351
|
|
|
- type: recall
|
|
|
value: 64.76190476190476
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (gle-eng)
|
|
|
config: gle-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 76.2
|
|
|
- type: f1
|
|
|
value: 71.48147546897547
|
|
|
- type: precision
|
|
|
value: 69.65409090909091
|
|
|
- type: recall
|
|
|
value: 76.2
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (pes-eng)
|
|
|
config: pes-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 93.8
|
|
|
- type: f1
|
|
|
value: 92.14
|
|
|
- type: precision
|
|
|
value: 91.35833333333333
|
|
|
- type: recall
|
|
|
value: 93.8
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (nob-eng)
|
|
|
config: nob-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 97.89999999999999
|
|
|
- type: f1
|
|
|
value: 97.2
|
|
|
- type: precision
|
|
|
value: 96.85000000000001
|
|
|
- type: recall
|
|
|
value: 97.89999999999999
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (bul-eng)
|
|
|
config: bul-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 94.6
|
|
|
- type: f1
|
|
|
value: 92.93333333333334
|
|
|
- type: precision
|
|
|
value: 92.13333333333333
|
|
|
- type: recall
|
|
|
value: 94.6
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (cbk-eng)
|
|
|
config: cbk-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 74.1
|
|
|
- type: f1
|
|
|
value: 69.14817460317461
|
|
|
- type: precision
|
|
|
value: 67.2515873015873
|
|
|
- type: recall
|
|
|
value: 74.1
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (hun-eng)
|
|
|
config: hun-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 95.19999999999999
|
|
|
- type: f1
|
|
|
value: 94.01333333333335
|
|
|
- type: precision
|
|
|
value: 93.46666666666667
|
|
|
- type: recall
|
|
|
value: 95.19999999999999
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (uig-eng)
|
|
|
config: uig-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 76.9
|
|
|
- type: f1
|
|
|
value: 72.07523809523809
|
|
|
- type: precision
|
|
|
value: 70.19777777777779
|
|
|
- type: recall
|
|
|
value: 76.9
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (rus-eng)
|
|
|
config: rus-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 94.1
|
|
|
- type: f1
|
|
|
value: 92.31666666666666
|
|
|
- type: precision
|
|
|
value: 91.43333333333332
|
|
|
- type: recall
|
|
|
value: 94.1
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (spa-eng)
|
|
|
config: spa-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 97.8
|
|
|
- type: f1
|
|
|
value: 97.1
|
|
|
- type: precision
|
|
|
value: 96.76666666666668
|
|
|
- type: recall
|
|
|
value: 97.8
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (hye-eng)
|
|
|
config: hye-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 92.85714285714286
|
|
|
- type: f1
|
|
|
value: 90.92093441150045
|
|
|
- type: precision
|
|
|
value: 90.00449236298293
|
|
|
- type: recall
|
|
|
value: 92.85714285714286
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (tel-eng)
|
|
|
config: tel-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 93.16239316239316
|
|
|
- type: f1
|
|
|
value: 91.33903133903132
|
|
|
- type: precision
|
|
|
value: 90.56267806267806
|
|
|
- type: recall
|
|
|
value: 93.16239316239316
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (afr-eng)
|
|
|
config: afr-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 92.4
|
|
|
- type: f1
|
|
|
value: 90.25666666666666
|
|
|
- type: precision
|
|
|
value: 89.25833333333334
|
|
|
- type: recall
|
|
|
value: 92.4
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (mon-eng)
|
|
|
config: mon-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 90.22727272727272
|
|
|
- type: f1
|
|
|
value: 87.53030303030303
|
|
|
- type: precision
|
|
|
value: 86.37121212121211
|
|
|
- type: recall
|
|
|
value: 90.22727272727272
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (arz-eng)
|
|
|
config: arz-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 79.03563941299791
|
|
|
- type: f1
|
|
|
value: 74.7349505840072
|
|
|
- type: precision
|
|
|
value: 72.9035639412998
|
|
|
- type: recall
|
|
|
value: 79.03563941299791
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (hrv-eng)
|
|
|
config: hrv-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 97
|
|
|
- type: f1
|
|
|
value: 96.15
|
|
|
- type: precision
|
|
|
value: 95.76666666666668
|
|
|
- type: recall
|
|
|
value: 97
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (nov-eng)
|
|
|
config: nov-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 76.26459143968872
|
|
|
- type: f1
|
|
|
value: 71.55642023346303
|
|
|
- type: precision
|
|
|
value: 69.7544932369835
|
|
|
- type: recall
|
|
|
value: 76.26459143968872
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (gsw-eng)
|
|
|
config: gsw-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 58.119658119658126
|
|
|
- type: f1
|
|
|
value: 51.65242165242165
|
|
|
- type: precision
|
|
|
value: 49.41768108434775
|
|
|
- type: recall
|
|
|
value: 58.119658119658126
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (nds-eng)
|
|
|
config: nds-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 74.3
|
|
|
- type: f1
|
|
|
value: 69.52055555555555
|
|
|
- type: precision
|
|
|
value: 67.7574938949939
|
|
|
- type: recall
|
|
|
value: 74.3
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (ukr-eng)
|
|
|
config: ukr-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 94.8
|
|
|
- type: f1
|
|
|
value: 93.31666666666666
|
|
|
- type: precision
|
|
|
value: 92.60000000000001
|
|
|
- type: recall
|
|
|
value: 94.8
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (uzb-eng)
|
|
|
config: uzb-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 76.63551401869158
|
|
|
- type: f1
|
|
|
value: 72.35202492211837
|
|
|
- type: precision
|
|
|
value: 70.60358255451713
|
|
|
- type: recall
|
|
|
value: 76.63551401869158
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (lit-eng)
|
|
|
config: lit-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 90.4
|
|
|
- type: f1
|
|
|
value: 88.4811111111111
|
|
|
- type: precision
|
|
|
value: 87.7452380952381
|
|
|
- type: recall
|
|
|
value: 90.4
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (ina-eng)
|
|
|
config: ina-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 95
|
|
|
- type: f1
|
|
|
value: 93.60666666666667
|
|
|
- type: precision
|
|
|
value: 92.975
|
|
|
- type: recall
|
|
|
value: 95
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (lfn-eng)
|
|
|
config: lfn-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 67.2
|
|
|
- type: f1
|
|
|
value: 63.01595782872099
|
|
|
- type: precision
|
|
|
value: 61.596587301587306
|
|
|
- type: recall
|
|
|
value: 67.2
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (zsm-eng)
|
|
|
config: zsm-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 95.7
|
|
|
- type: f1
|
|
|
value: 94.52999999999999
|
|
|
- type: precision
|
|
|
value: 94
|
|
|
- type: recall
|
|
|
value: 95.7
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (ita-eng)
|
|
|
config: ita-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 94.6
|
|
|
- type: f1
|
|
|
value: 93.28999999999999
|
|
|
- type: precision
|
|
|
value: 92.675
|
|
|
- type: recall
|
|
|
value: 94.6
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (cmn-eng)
|
|
|
config: cmn-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 96.39999999999999
|
|
|
- type: f1
|
|
|
value: 95.28333333333333
|
|
|
- type: precision
|
|
|
value: 94.75
|
|
|
- type: recall
|
|
|
value: 96.39999999999999
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (lvs-eng)
|
|
|
config: lvs-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 91.9
|
|
|
- type: f1
|
|
|
value: 89.83
|
|
|
- type: precision
|
|
|
value: 88.92
|
|
|
- type: recall
|
|
|
value: 91.9
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (glg-eng)
|
|
|
config: glg-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 94.69999999999999
|
|
|
- type: f1
|
|
|
value: 93.34222222222223
|
|
|
- type: precision
|
|
|
value: 92.75416666666668
|
|
|
- type: recall
|
|
|
value: 94.69999999999999
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (ceb-eng)
|
|
|
config: ceb-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 60.333333333333336
|
|
|
- type: f1
|
|
|
value: 55.31203703703703
|
|
|
- type: precision
|
|
|
value: 53.39971108326371
|
|
|
- type: recall
|
|
|
value: 60.333333333333336
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (bre-eng)
|
|
|
config: bre-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 12.9
|
|
|
- type: f1
|
|
|
value: 11.099861903031458
|
|
|
- type: precision
|
|
|
value: 10.589187932631877
|
|
|
- type: recall
|
|
|
value: 12.9
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (ben-eng)
|
|
|
config: ben-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 86.7
|
|
|
- type: f1
|
|
|
value: 83.0152380952381
|
|
|
- type: precision
|
|
|
value: 81.37833333333333
|
|
|
- type: recall
|
|
|
value: 86.7
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (swg-eng)
|
|
|
config: swg-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 63.39285714285714
|
|
|
- type: f1
|
|
|
value: 56.832482993197274
|
|
|
- type: precision
|
|
|
value: 54.56845238095237
|
|
|
- type: recall
|
|
|
value: 63.39285714285714
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (arq-eng)
|
|
|
config: arq-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 48.73765093304062
|
|
|
- type: f1
|
|
|
value: 41.555736920720456
|
|
|
- type: precision
|
|
|
value: 39.06874531737319
|
|
|
- type: recall
|
|
|
value: 48.73765093304062
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (kab-eng)
|
|
|
config: kab-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 41.099999999999994
|
|
|
- type: f1
|
|
|
value: 36.540165945165946
|
|
|
- type: precision
|
|
|
value: 35.05175685425686
|
|
|
- type: recall
|
|
|
value: 41.099999999999994
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (fra-eng)
|
|
|
config: fra-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 94.89999999999999
|
|
|
- type: f1
|
|
|
value: 93.42333333333333
|
|
|
- type: precision
|
|
|
value: 92.75833333333333
|
|
|
- type: recall
|
|
|
value: 94.89999999999999
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (por-eng)
|
|
|
config: por-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 94.89999999999999
|
|
|
- type: f1
|
|
|
value: 93.63333333333334
|
|
|
- type: precision
|
|
|
value: 93.01666666666665
|
|
|
- type: recall
|
|
|
value: 94.89999999999999
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (tat-eng)
|
|
|
config: tat-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 77.9
|
|
|
- type: f1
|
|
|
value: 73.64833333333334
|
|
|
- type: precision
|
|
|
value: 71.90282106782105
|
|
|
- type: recall
|
|
|
value: 77.9
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (oci-eng)
|
|
|
config: oci-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 59.4
|
|
|
- type: f1
|
|
|
value: 54.90521367521367
|
|
|
- type: precision
|
|
|
value: 53.432840025471606
|
|
|
- type: recall
|
|
|
value: 59.4
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (pol-eng)
|
|
|
config: pol-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 97.39999999999999
|
|
|
- type: f1
|
|
|
value: 96.6
|
|
|
- type: precision
|
|
|
value: 96.2
|
|
|
- type: recall
|
|
|
value: 97.39999999999999
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (war-eng)
|
|
|
config: war-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 67.2
|
|
|
- type: f1
|
|
|
value: 62.25926129426129
|
|
|
- type: precision
|
|
|
value: 60.408376623376626
|
|
|
- type: recall
|
|
|
value: 67.2
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (aze-eng)
|
|
|
config: aze-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 90.2
|
|
|
- type: f1
|
|
|
value: 87.60666666666667
|
|
|
- type: precision
|
|
|
value: 86.45277777777778
|
|
|
- type: recall
|
|
|
value: 90.2
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (vie-eng)
|
|
|
config: vie-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 97.7
|
|
|
- type: f1
|
|
|
value: 97
|
|
|
- type: precision
|
|
|
value: 96.65
|
|
|
- type: recall
|
|
|
value: 97.7
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (nno-eng)
|
|
|
config: nno-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 93.2
|
|
|
- type: f1
|
|
|
value: 91.39746031746031
|
|
|
- type: precision
|
|
|
value: 90.6125
|
|
|
- type: recall
|
|
|
value: 93.2
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (cha-eng)
|
|
|
config: cha-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 32.11678832116788
|
|
|
- type: f1
|
|
|
value: 27.210415386260234
|
|
|
- type: precision
|
|
|
value: 26.20408990846947
|
|
|
- type: recall
|
|
|
value: 32.11678832116788
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (mhr-eng)
|
|
|
config: mhr-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 8.5
|
|
|
- type: f1
|
|
|
value: 6.787319277832475
|
|
|
- type: precision
|
|
|
value: 6.3452094433344435
|
|
|
- type: recall
|
|
|
value: 8.5
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (dan-eng)
|
|
|
config: dan-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 96.1
|
|
|
- type: f1
|
|
|
value: 95.08
|
|
|
- type: precision
|
|
|
value: 94.61666666666667
|
|
|
- type: recall
|
|
|
value: 96.1
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (ell-eng)
|
|
|
config: ell-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 95.3
|
|
|
- type: f1
|
|
|
value: 93.88333333333333
|
|
|
- type: precision
|
|
|
value: 93.18333333333332
|
|
|
- type: recall
|
|
|
value: 95.3
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (amh-eng)
|
|
|
config: amh-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 85.11904761904762
|
|
|
- type: f1
|
|
|
value: 80.69444444444444
|
|
|
- type: precision
|
|
|
value: 78.72023809523809
|
|
|
- type: recall
|
|
|
value: 85.11904761904762
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (pam-eng)
|
|
|
config: pam-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 11.1
|
|
|
- type: f1
|
|
|
value: 9.276381801735853
|
|
|
- type: precision
|
|
|
value: 8.798174603174601
|
|
|
- type: recall
|
|
|
value: 11.1
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (hsb-eng)
|
|
|
config: hsb-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 63.56107660455487
|
|
|
- type: f1
|
|
|
value: 58.70433569191332
|
|
|
- type: precision
|
|
|
value: 56.896926581464015
|
|
|
- type: recall
|
|
|
value: 63.56107660455487
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (srp-eng)
|
|
|
config: srp-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 94.69999999999999
|
|
|
- type: f1
|
|
|
value: 93.10000000000001
|
|
|
- type: precision
|
|
|
value: 92.35
|
|
|
- type: recall
|
|
|
value: 94.69999999999999
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (epo-eng)
|
|
|
config: epo-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 96.8
|
|
|
- type: f1
|
|
|
value: 96.01222222222222
|
|
|
- type: precision
|
|
|
value: 95.67083333333332
|
|
|
- type: recall
|
|
|
value: 96.8
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (kzj-eng)
|
|
|
config: kzj-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 9.2
|
|
|
- type: f1
|
|
|
value: 7.911555250305249
|
|
|
- type: precision
|
|
|
value: 7.631246556216846
|
|
|
- type: recall
|
|
|
value: 9.2
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (awa-eng)
|
|
|
config: awa-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 77.48917748917748
|
|
|
- type: f1
|
|
|
value: 72.27375798804371
|
|
|
- type: precision
|
|
|
value: 70.14430014430013
|
|
|
- type: recall
|
|
|
value: 77.48917748917748
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (fao-eng)
|
|
|
config: fao-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 77.09923664122137
|
|
|
- type: f1
|
|
|
value: 72.61541257724463
|
|
|
- type: precision
|
|
|
value: 70.8998380754106
|
|
|
- type: recall
|
|
|
value: 77.09923664122137
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (mal-eng)
|
|
|
config: mal-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 98.2532751091703
|
|
|
- type: f1
|
|
|
value: 97.69529354682193
|
|
|
- type: precision
|
|
|
value: 97.42843279961184
|
|
|
- type: recall
|
|
|
value: 98.2532751091703
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (ile-eng)
|
|
|
config: ile-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 82.8
|
|
|
- type: f1
|
|
|
value: 79.14672619047619
|
|
|
- type: precision
|
|
|
value: 77.59489247311828
|
|
|
- type: recall
|
|
|
value: 82.8
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (bos-eng)
|
|
|
config: bos-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 94.35028248587571
|
|
|
- type: f1
|
|
|
value: 92.86252354048965
|
|
|
- type: precision
|
|
|
value: 92.2080979284369
|
|
|
- type: recall
|
|
|
value: 94.35028248587571
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (cor-eng)
|
|
|
config: cor-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 8.5
|
|
|
- type: f1
|
|
|
value: 6.282429263935621
|
|
|
- type: precision
|
|
|
value: 5.783274240739785
|
|
|
- type: recall
|
|
|
value: 8.5
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (cat-eng)
|
|
|
config: cat-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 92.7
|
|
|
- type: f1
|
|
|
value: 91.025
|
|
|
- type: precision
|
|
|
value: 90.30428571428571
|
|
|
- type: recall
|
|
|
value: 92.7
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (eus-eng)
|
|
|
config: eus-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 81
|
|
|
- type: f1
|
|
|
value: 77.8232380952381
|
|
|
- type: precision
|
|
|
value: 76.60194444444444
|
|
|
- type: recall
|
|
|
value: 81
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (yue-eng)
|
|
|
config: yue-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 91
|
|
|
- type: f1
|
|
|
value: 88.70857142857142
|
|
|
- type: precision
|
|
|
value: 87.7
|
|
|
- type: recall
|
|
|
value: 91
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (swe-eng)
|
|
|
config: swe-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 96.39999999999999
|
|
|
- type: f1
|
|
|
value: 95.3
|
|
|
- type: precision
|
|
|
value: 94.76666666666667
|
|
|
- type: recall
|
|
|
value: 96.39999999999999
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (dtp-eng)
|
|
|
config: dtp-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 8.1
|
|
|
- type: f1
|
|
|
value: 7.001008218834307
|
|
|
- type: precision
|
|
|
value: 6.708329562594269
|
|
|
- type: recall
|
|
|
value: 8.1
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (kat-eng)
|
|
|
config: kat-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 87.1313672922252
|
|
|
- type: f1
|
|
|
value: 84.09070598748882
|
|
|
- type: precision
|
|
|
value: 82.79171454104429
|
|
|
- type: recall
|
|
|
value: 87.1313672922252
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (jpn-eng)
|
|
|
config: jpn-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 96.39999999999999
|
|
|
- type: f1
|
|
|
value: 95.28333333333333
|
|
|
- type: precision
|
|
|
value: 94.73333333333332
|
|
|
- type: recall
|
|
|
value: 96.39999999999999
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (csb-eng)
|
|
|
config: csb-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 42.29249011857708
|
|
|
- type: f1
|
|
|
value: 36.981018542283365
|
|
|
- type: precision
|
|
|
value: 35.415877813576024
|
|
|
- type: recall
|
|
|
value: 42.29249011857708
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (xho-eng)
|
|
|
config: xho-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 83.80281690140845
|
|
|
- type: f1
|
|
|
value: 80.86854460093896
|
|
|
- type: precision
|
|
|
value: 79.60093896713614
|
|
|
- type: recall
|
|
|
value: 83.80281690140845
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (orv-eng)
|
|
|
config: orv-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 45.26946107784431
|
|
|
- type: f1
|
|
|
value: 39.80235464678088
|
|
|
- type: precision
|
|
|
value: 38.14342660001342
|
|
|
- type: recall
|
|
|
value: 45.26946107784431
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (ind-eng)
|
|
|
config: ind-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 94.3
|
|
|
- type: f1
|
|
|
value: 92.9
|
|
|
- type: precision
|
|
|
value: 92.26666666666668
|
|
|
- type: recall
|
|
|
value: 94.3
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (tuk-eng)
|
|
|
config: tuk-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 37.93103448275862
|
|
|
- type: f1
|
|
|
value: 33.15192743764172
|
|
|
- type: precision
|
|
|
value: 31.57456528146183
|
|
|
- type: recall
|
|
|
value: 37.93103448275862
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (max-eng)
|
|
|
config: max-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 69.01408450704226
|
|
|
- type: f1
|
|
|
value: 63.41549295774648
|
|
|
- type: precision
|
|
|
value: 61.342778895595806
|
|
|
- type: recall
|
|
|
value: 69.01408450704226
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (swh-eng)
|
|
|
config: swh-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 76.66666666666667
|
|
|
- type: f1
|
|
|
value: 71.60705960705961
|
|
|
- type: precision
|
|
|
value: 69.60683760683762
|
|
|
- type: recall
|
|
|
value: 76.66666666666667
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (hin-eng)
|
|
|
config: hin-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 95.8
|
|
|
- type: f1
|
|
|
value: 94.48333333333333
|
|
|
- type: precision
|
|
|
value: 93.83333333333333
|
|
|
- type: recall
|
|
|
value: 95.8
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (dsb-eng)
|
|
|
config: dsb-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 52.81837160751566
|
|
|
- type: f1
|
|
|
value: 48.435977731384824
|
|
|
- type: precision
|
|
|
value: 47.11291973845539
|
|
|
- type: recall
|
|
|
value: 52.81837160751566
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (ber-eng)
|
|
|
config: ber-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 44.9
|
|
|
- type: f1
|
|
|
value: 38.88962621607783
|
|
|
- type: precision
|
|
|
value: 36.95936507936508
|
|
|
- type: recall
|
|
|
value: 44.9
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (tam-eng)
|
|
|
config: tam-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 90.55374592833876
|
|
|
- type: f1
|
|
|
value: 88.22553125484721
|
|
|
- type: precision
|
|
|
value: 87.26927252985884
|
|
|
- type: recall
|
|
|
value: 90.55374592833876
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (slk-eng)
|
|
|
config: slk-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 94.6
|
|
|
- type: f1
|
|
|
value: 93.13333333333333
|
|
|
- type: precision
|
|
|
value: 92.45333333333333
|
|
|
- type: recall
|
|
|
value: 94.6
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (tgl-eng)
|
|
|
config: tgl-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 93.7
|
|
|
- type: f1
|
|
|
value: 91.99666666666667
|
|
|
- type: precision
|
|
|
value: 91.26666666666668
|
|
|
- type: recall
|
|
|
value: 93.7
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (ast-eng)
|
|
|
config: ast-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 85.03937007874016
|
|
|
- type: f1
|
|
|
value: 81.75853018372703
|
|
|
- type: precision
|
|
|
value: 80.34120734908137
|
|
|
- type: recall
|
|
|
value: 85.03937007874016
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (mkd-eng)
|
|
|
config: mkd-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 88.3
|
|
|
- type: f1
|
|
|
value: 85.5
|
|
|
- type: precision
|
|
|
value: 84.25833333333334
|
|
|
- type: recall
|
|
|
value: 88.3
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (khm-eng)
|
|
|
config: khm-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 65.51246537396122
|
|
|
- type: f1
|
|
|
value: 60.02297410192148
|
|
|
- type: precision
|
|
|
value: 58.133467727289236
|
|
|
- type: recall
|
|
|
value: 65.51246537396122
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (ces-eng)
|
|
|
config: ces-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 96
|
|
|
- type: f1
|
|
|
value: 94.89
|
|
|
- type: precision
|
|
|
value: 94.39166666666667
|
|
|
- type: recall
|
|
|
value: 96
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (tzl-eng)
|
|
|
config: tzl-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 57.692307692307686
|
|
|
- type: f1
|
|
|
value: 53.162393162393165
|
|
|
- type: precision
|
|
|
value: 51.70673076923077
|
|
|
- type: recall
|
|
|
value: 57.692307692307686
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (urd-eng)
|
|
|
config: urd-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 91.60000000000001
|
|
|
- type: f1
|
|
|
value: 89.21190476190475
|
|
|
- type: precision
|
|
|
value: 88.08666666666667
|
|
|
- type: recall
|
|
|
value: 91.60000000000001
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (ara-eng)
|
|
|
config: ara-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 88
|
|
|
- type: f1
|
|
|
value: 85.47
|
|
|
- type: precision
|
|
|
value: 84.43266233766234
|
|
|
- type: recall
|
|
|
value: 88
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (kor-eng)
|
|
|
config: kor-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 92.7
|
|
|
- type: f1
|
|
|
value: 90.64999999999999
|
|
|
- type: precision
|
|
|
value: 89.68333333333332
|
|
|
- type: recall
|
|
|
value: 92.7
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (yid-eng)
|
|
|
config: yid-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 80.30660377358491
|
|
|
- type: f1
|
|
|
value: 76.33044137466307
|
|
|
- type: precision
|
|
|
value: 74.78970125786164
|
|
|
- type: recall
|
|
|
value: 80.30660377358491
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (fin-eng)
|
|
|
config: fin-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 96.39999999999999
|
|
|
- type: f1
|
|
|
value: 95.44
|
|
|
- type: precision
|
|
|
value: 94.99166666666666
|
|
|
- type: recall
|
|
|
value: 96.39999999999999
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (tha-eng)
|
|
|
config: tha-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 96.53284671532847
|
|
|
- type: f1
|
|
|
value: 95.37712895377129
|
|
|
- type: precision
|
|
|
value: 94.7992700729927
|
|
|
- type: recall
|
|
|
value: 96.53284671532847
|
|
|
- task:
|
|
|
type: BitextMining
|
|
|
dataset:
|
|
|
type: mteb/tatoeba-bitext-mining
|
|
|
name: MTEB Tatoeba (wuu-eng)
|
|
|
config: wuu-eng
|
|
|
split: test
|
|
|
revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 89
|
|
|
- type: f1
|
|
|
value: 86.23190476190476
|
|
|
- type: precision
|
|
|
value: 85.035
|
|
|
- type: recall
|
|
|
value: 89
|
|
|
- task:
|
|
|
type: Retrieval
|
|
|
dataset:
|
|
|
type: webis-touche2020
|
|
|
name: MTEB Touche2020
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: None
|
|
|
metrics:
|
|
|
- type: map_at_1
|
|
|
value: 2.585
|
|
|
- type: map_at_10
|
|
|
value: 9.012
|
|
|
- type: map_at_100
|
|
|
value: 14.027000000000001
|
|
|
- type: map_at_1000
|
|
|
value: 15.565000000000001
|
|
|
- type: map_at_3
|
|
|
value: 5.032
|
|
|
- type: map_at_5
|
|
|
value: 6.657
|
|
|
- type: mrr_at_1
|
|
|
value: 28.571
|
|
|
- type: mrr_at_10
|
|
|
value: 45.377
|
|
|
- type: mrr_at_100
|
|
|
value: 46.119
|
|
|
- type: mrr_at_1000
|
|
|
value: 46.127
|
|
|
- type: mrr_at_3
|
|
|
value: 41.156
|
|
|
- type: mrr_at_5
|
|
|
value: 42.585
|
|
|
- type: ndcg_at_1
|
|
|
value: 27.551
|
|
|
- type: ndcg_at_10
|
|
|
value: 23.395
|
|
|
- type: ndcg_at_100
|
|
|
value: 33.342
|
|
|
- type: ndcg_at_1000
|
|
|
value: 45.523
|
|
|
- type: ndcg_at_3
|
|
|
value: 25.158
|
|
|
- type: ndcg_at_5
|
|
|
value: 23.427
|
|
|
- type: precision_at_1
|
|
|
value: 28.571
|
|
|
- type: precision_at_10
|
|
|
value: 21.429000000000002
|
|
|
- type: precision_at_100
|
|
|
value: 6.714
|
|
|
- type: precision_at_1000
|
|
|
value: 1.473
|
|
|
- type: precision_at_3
|
|
|
value: 27.211000000000002
|
|
|
- type: precision_at_5
|
|
|
value: 24.490000000000002
|
|
|
- type: recall_at_1
|
|
|
value: 2.585
|
|
|
- type: recall_at_10
|
|
|
value: 15.418999999999999
|
|
|
- type: recall_at_100
|
|
|
value: 42.485
|
|
|
- type: recall_at_1000
|
|
|
value: 79.536
|
|
|
- type: recall_at_3
|
|
|
value: 6.239999999999999
|
|
|
- type: recall_at_5
|
|
|
value: 8.996
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/toxic_conversations_50k
|
|
|
name: MTEB ToxicConversationsClassification
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 71.3234
|
|
|
- type: ap
|
|
|
value: 14.361688653847423
|
|
|
- type: f1
|
|
|
value: 54.819068624319044
|
|
|
- task:
|
|
|
type: Classification
|
|
|
dataset:
|
|
|
type: mteb/tweet_sentiment_extraction
|
|
|
name: MTEB TweetSentimentExtractionClassification
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: d604517c81ca91fe16a244d1248fc021f9ecee7a
|
|
|
metrics:
|
|
|
- type: accuracy
|
|
|
value: 61.97792869269949
|
|
|
- type: f1
|
|
|
value: 62.28965628513728
|
|
|
- task:
|
|
|
type: Clustering
|
|
|
dataset:
|
|
|
type: mteb/twentynewsgroups-clustering
|
|
|
name: MTEB TwentyNewsgroupsClustering
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: 6125ec4e24fa026cec8a478383ee943acfbd5449
|
|
|
metrics:
|
|
|
- type: v_measure
|
|
|
value: 38.90540145385218
|
|
|
- task:
|
|
|
type: PairClassification
|
|
|
dataset:
|
|
|
type: mteb/twittersemeval2015-pairclassification
|
|
|
name: MTEB TwitterSemEval2015
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1
|
|
|
metrics:
|
|
|
- type: cos_sim_accuracy
|
|
|
value: 86.53513739047506
|
|
|
- type: cos_sim_ap
|
|
|
value: 75.27741586677557
|
|
|
- type: cos_sim_f1
|
|
|
value: 69.18792902473774
|
|
|
- type: cos_sim_precision
|
|
|
value: 67.94708725515136
|
|
|
- type: cos_sim_recall
|
|
|
value: 70.47493403693932
|
|
|
- type: dot_accuracy
|
|
|
value: 84.7052512368123
|
|
|
- type: dot_ap
|
|
|
value: 69.36075482849378
|
|
|
- type: dot_f1
|
|
|
value: 64.44688376631296
|
|
|
- type: dot_precision
|
|
|
value: 59.92288500793831
|
|
|
- type: dot_recall
|
|
|
value: 69.70976253298153
|
|
|
- type: euclidean_accuracy
|
|
|
value: 86.60666388508076
|
|
|
- type: euclidean_ap
|
|
|
value: 75.47512772621097
|
|
|
- type: euclidean_f1
|
|
|
value: 69.413872536473
|
|
|
- type: euclidean_precision
|
|
|
value: 67.39562624254472
|
|
|
- type: euclidean_recall
|
|
|
value: 71.55672823218997
|
|
|
- type: manhattan_accuracy
|
|
|
value: 86.52917684925792
|
|
|
- type: manhattan_ap
|
|
|
value: 75.34000110496703
|
|
|
- type: manhattan_f1
|
|
|
value: 69.28489190226429
|
|
|
- type: manhattan_precision
|
|
|
value: 67.24608889992551
|
|
|
- type: manhattan_recall
|
|
|
value: 71.45118733509234
|
|
|
- type: max_accuracy
|
|
|
value: 86.60666388508076
|
|
|
- type: max_ap
|
|
|
value: 75.47512772621097
|
|
|
- type: max_f1
|
|
|
value: 69.413872536473
|
|
|
- task:
|
|
|
type: PairClassification
|
|
|
dataset:
|
|
|
type: mteb/twitterurlcorpus-pairclassification
|
|
|
name: MTEB TwitterURLCorpus
|
|
|
config: default
|
|
|
split: test
|
|
|
revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf
|
|
|
metrics:
|
|
|
- type: cos_sim_accuracy
|
|
|
value: 89.01695967710637
|
|
|
- type: cos_sim_ap
|
|
|
value: 85.8298270742901
|
|
|
- type: cos_sim_f1
|
|
|
value: 78.46988128389272
|
|
|
- type: cos_sim_precision
|
|
|
value: 74.86017897091722
|
|
|
- type: cos_sim_recall
|
|
|
value: 82.44533415460425
|
|
|
- type: dot_accuracy
|
|
|
value: 88.19420188613343
|
|
|
- type: dot_ap
|
|
|
value: 83.82679165901324
|
|
|
- type: dot_f1
|
|
|
value: 76.55833777304208
|
|
|
- type: dot_precision
|
|
|
value: 75.6884875846501
|
|
|
- type: dot_recall
|
|
|
value: 77.44841392054204
|
|
|
- type: euclidean_accuracy
|
|
|
value: 89.03054294252338
|
|
|
- type: euclidean_ap
|
|
|
value: 85.89089555185325
|
|
|
- type: euclidean_f1
|
|
|
value: 78.62997658079624
|
|
|
- type: euclidean_precision
|
|
|
value: 74.92329149232914
|
|
|
- type: euclidean_recall
|
|
|
value: 82.72251308900523
|
|
|
- type: manhattan_accuracy
|
|
|
value: 89.0266620095471
|
|
|
- type: manhattan_ap
|
|
|
value: 85.86458997929147
|
|
|
- type: manhattan_f1
|
|
|
value: 78.50685331000291
|
|
|
- type: manhattan_precision
|
|
|
value: 74.5499861534201
|
|
|
- type: manhattan_recall
|
|
|
value: 82.90729904527257
|
|
|
- type: max_accuracy
|
|
|
value: 89.03054294252338
|
|
|
- type: max_ap
|
|
|
value: 85.89089555185325
|
|
|
- type: max_f1
|
|
|
value: 78.62997658079624
|
|
|
language:
|
|
|
- multilingual
|
|
|
- af
|
|
|
- am
|
|
|
- ar
|
|
|
- as
|
|
|
- az
|
|
|
- be
|
|
|
- bg
|
|
|
- bn
|
|
|
- br
|
|
|
- bs
|
|
|
- ca
|
|
|
- cs
|
|
|
- cy
|
|
|
- da
|
|
|
- de
|
|
|
- el
|
|
|
- en
|
|
|
- eo
|
|
|
- es
|
|
|
- et
|
|
|
- eu
|
|
|
- fa
|
|
|
- fi
|
|
|
- fr
|
|
|
- fy
|
|
|
- ga
|
|
|
- gd
|
|
|
- gl
|
|
|
- gu
|
|
|
- ha
|
|
|
- he
|
|
|
- hi
|
|
|
- hr
|
|
|
- hu
|
|
|
- hy
|
|
|
- id
|
|
|
- is
|
|
|
- it
|
|
|
- ja
|
|
|
- jv
|
|
|
- ka
|
|
|
- kk
|
|
|
- km
|
|
|
- kn
|
|
|
- ko
|
|
|
- ku
|
|
|
- ky
|
|
|
- la
|
|
|
- lo
|
|
|
- lt
|
|
|
- lv
|
|
|
- mg
|
|
|
- mk
|
|
|
- ml
|
|
|
- mn
|
|
|
- mr
|
|
|
- ms
|
|
|
- my
|
|
|
- ne
|
|
|
- nl
|
|
|
- 'no'
|
|
|
- om
|
|
|
- or
|
|
|
- pa
|
|
|
- pl
|
|
|
- ps
|
|
|
- pt
|
|
|
- ro
|
|
|
- ru
|
|
|
- sa
|
|
|
- sd
|
|
|
- si
|
|
|
- sk
|
|
|
- sl
|
|
|
- so
|
|
|
- sq
|
|
|
- sr
|
|
|
- su
|
|
|
- sv
|
|
|
- sw
|
|
|
- ta
|
|
|
- te
|
|
|
- th
|
|
|
- tl
|
|
|
- tr
|
|
|
- ug
|
|
|
- uk
|
|
|
- ur
|
|
|
- uz
|
|
|
- vi
|
|
|
- xh
|
|
|
- yi
|
|
|
- zh
|
|
|
license: mit
|
|
|
---
|
|
|
|
|
|
## Multilingual-E5-large
|
|
|
|
|
|
[Multilingual E5 Text Embeddings: A Technical Report](https://arxiv.org/pdf/2402.05672).
|
|
|
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei, arXiv 2024
|
|
|
|
|
|
This model has 24 layers and the embedding size is 1024.
|
|
|
|
|
|
## Usage
|
|
|
|
|
|
Below is an example to encode queries and passages from the MS-MARCO passage ranking dataset.
|
|
|
|
|
|
```python
|
|
|
import torch.nn.functional as F
|
|
|
|
|
|
from torch import Tensor
|
|
|
from transformers import AutoTokenizer, AutoModel
|
|
|
|
|
|
|
|
|
def average_pool(last_hidden_states: Tensor,
|
|
|
attention_mask: Tensor) -> Tensor:
|
|
|
last_hidden = last_hidden_states.masked_fill(~attention_mask[..., None].bool(), 0.0)
|
|
|
return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None]
|
|
|
|
|
|
|
|
|
# Each input text should start with "query: " or "passage: ", even for non-English texts.
|
|
|
# For tasks other than retrieval, you can simply use the "query: " prefix.
|
|
|
input_texts = ['query: how much protein should a female eat',
|
|
|
'query: 南瓜的家常做法',
|
|
|
"passage: As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.",
|
|
|
"passage: 1.清炒南瓜丝 原料:嫩南瓜半个 调料:葱、盐、白糖、鸡精 做法: 1、南瓜用刀薄薄的削去表面一层皮,用勺子刮去瓤 2、擦成细丝(没有擦菜板就用刀慢慢切成细丝) 3、锅烧热放油,入葱花煸出香味 4、入南瓜丝快速翻炒一分钟左右,放盐、一点白糖和鸡精调味出锅 2.香葱炒南瓜 原料:南瓜1只 调料:香葱、蒜末、橄榄油、盐 做法: 1、将南瓜去皮,切成片 2、油锅8成热后,将蒜末放入爆香 3、爆香后,将南瓜片放入,翻炒 4、在翻炒的同时,可以不时地往锅里加水,但不要太多 5、放入盐,炒匀 6、南瓜差不多软和绵了之后,就可以关火 7、撒入香葱,即可出锅"]
|
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained('intfloat/multilingual-e5-large')
|
|
|
model = AutoModel.from_pretrained('intfloat/multilingual-e5-large')
|
|
|
|
|
|
# Tokenize the input texts
|
|
|
batch_dict = tokenizer(input_texts, max_length=512, padding=True, truncation=True, return_tensors='pt')
|
|
|
|
|
|
outputs = model(**batch_dict)
|
|
|
embeddings = average_pool(outputs.last_hidden_state, batch_dict['attention_mask'])
|
|
|
|
|
|
# normalize embeddings
|
|
|
embeddings = F.normalize(embeddings, p=2, dim=1)
|
|
|
scores = (embeddings[:2] @ embeddings[2:].T) * 100
|
|
|
print(scores.tolist())
|
|
|
```
|
|
|
|
|
|
## Supported Languages
|
|
|
|
|
|
This model is initialized from [xlm-roberta-large](https://huggingface.co/xlm-roberta-large)
|
|
|
and continually trained on a mixture of multilingual datasets.
|
|
|
It supports 100 languages from xlm-roberta,
|
|
|
but low-resource languages may see performance degradation.
|
|
|
|
|
|
## Training Details
|
|
|
|
|
|
**Initialization**: [xlm-roberta-large](https://huggingface.co/xlm-roberta-large)
|
|
|
|
|
|
**First stage**: contrastive pre-training with weak supervision
|
|
|
|
|
|
| Dataset | Weak supervision | # of text pairs |
|
|
|
|--------------------------------------------------------------------------------------------------------|---------------------------------------|-----------------|
|
|
|
| Filtered [mC4](https://huggingface.co/datasets/mc4) | (title, page content) | 1B |
|
|
|
| [CC News](https://huggingface.co/datasets/intfloat/multilingual_cc_news) | (title, news content) | 400M |
|
|
|
| [NLLB](https://huggingface.co/datasets/allenai/nllb) | translation pairs | 2.4B |
|
|
|
| [Wikipedia](https://huggingface.co/datasets/intfloat/wikipedia) | (hierarchical section title, passage) | 150M |
|
|
|
| Filtered [Reddit](https://www.reddit.com/) | (comment, response) | 800M |
|
|
|
| [S2ORC](https://github.com/allenai/s2orc) | (title, abstract) and citation pairs | 100M |
|
|
|
| [Stackexchange](https://stackexchange.com/) | (question, answer) | 50M |
|
|
|
| [xP3](https://huggingface.co/datasets/bigscience/xP3) | (input prompt, response) | 80M |
|
|
|
| [Miscellaneous unsupervised SBERT data](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) | - | 10M |
|
|
|
|
|
|
**Second stage**: supervised fine-tuning
|
|
|
|
|
|
| Dataset | Language | # of text pairs |
|
|
|
|----------------------------------------------------------------------------------------|--------------|-----------------|
|
|
|
| [MS MARCO](https://microsoft.github.io/msmarco/) | English | 500k |
|
|
|
| [NQ](https://github.com/facebookresearch/DPR) | English | 70k |
|
|
|
| [Trivia QA](https://github.com/facebookresearch/DPR) | English | 60k |
|
|
|
| [NLI from SimCSE](https://github.com/princeton-nlp/SimCSE) | English | <300k |
|
|
|
| [ELI5](https://huggingface.co/datasets/eli5) | English | 500k |
|
|
|
| [DuReader Retrieval](https://github.com/baidu/DuReader/tree/master/DuReader-Retrieval) | Chinese | 86k |
|
|
|
| [KILT Fever](https://huggingface.co/datasets/kilt_tasks) | English | 70k |
|
|
|
| [KILT HotpotQA](https://huggingface.co/datasets/kilt_tasks) | English | 70k |
|
|
|
| [SQuAD](https://huggingface.co/datasets/squad) | English | 87k |
|
|
|
| [Quora](https://huggingface.co/datasets/quora) | English | 150k |
|
|
|
| [Mr. TyDi](https://huggingface.co/datasets/castorini/mr-tydi) | 11 languages | 50k |
|
|
|
| [MIRACL](https://huggingface.co/datasets/miracl/miracl) | 16 languages | 40k |
|
|
|
|
|
|
For all labeled datasets, we only use its training set for fine-tuning.
|
|
|
|
|
|
For other training details, please refer to our paper at [https://arxiv.org/pdf/2402.05672](https://arxiv.org/pdf/2402.05672).
|
|
|
|
|
|
## Benchmark Results on [Mr. TyDi](https://arxiv.org/abs/2108.08787)
|
|
|
|
|
|
| Model | Avg MRR@10 | | ar | bn | en | fi | id | ja | ko | ru | sw | te | th |
|
|
|
|-----------------------|------------|-------|------| --- | --- | --- | --- | --- | --- | --- |------| --- | --- |
|
|
|
| BM25 | 33.3 | | 36.7 | 41.3 | 15.1 | 28.8 | 38.2 | 21.7 | 28.1 | 32.9 | 39.6 | 42.4 | 41.7 |
|
|
|
| mDPR | 16.7 | | 26.0 | 25.8 | 16.2 | 11.3 | 14.6 | 18.1 | 21.9 | 18.5 | 7.3 | 10.6 | 13.5 |
|
|
|
| BM25 + mDPR | 41.7 | | 49.1 | 53.5 | 28.4 | 36.5 | 45.5 | 35.5 | 36.2 | 42.7 | 40.5 | 42.0 | 49.2 |
|
|
|
| | |
|
|
|
| multilingual-e5-small | 64.4 | | 71.5 | 66.3 | 54.5 | 57.7 | 63.2 | 55.4 | 54.3 | 60.8 | 65.4 | 89.1 | 70.1 |
|
|
|
| multilingual-e5-base | 65.9 | | 72.3 | 65.0 | 58.5 | 60.8 | 64.9 | 56.6 | 55.8 | 62.7 | 69.0 | 86.6 | 72.7 |
|
|
|
| multilingual-e5-large | **70.5** | | 77.5 | 73.2 | 60.8 | 66.8 | 68.5 | 62.5 | 61.6 | 65.8 | 72.7 | 90.2 | 76.2 |
|
|
|
|
|
|
## MTEB Benchmark Evaluation
|
|
|
|
|
|
Check out [unilm/e5](https://github.com/microsoft/unilm/tree/master/e5) to reproduce evaluation results
|
|
|
on the [BEIR](https://arxiv.org/abs/2104.08663) and [MTEB benchmark](https://arxiv.org/abs/2210.07316).
|
|
|
|
|
|
## Support for Sentence Transformers
|
|
|
|
|
|
Below is an example for usage with sentence_transformers.
|
|
|
```python
|
|
|
from sentence_transformers import SentenceTransformer
|
|
|
model = SentenceTransformer('intfloat/multilingual-e5-large')
|
|
|
input_texts = [
|
|
|
'query: how much protein should a female eat',
|
|
|
'query: 南瓜的家常做法',
|
|
|
"passage: As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 i s 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or traini ng for a marathon. Check out the chart below to see how much protein you should be eating each day.",
|
|
|
"passage: 1.清炒南瓜丝 原料:嫩南瓜半个 调料:葱、盐、白糖、鸡精 做法: 1、南瓜用刀薄薄的削去表面一层皮 ,用勺子刮去瓤 2、擦成细丝(没有擦菜板就用刀慢慢切成细丝) 3、锅烧热放油,入葱花煸出香味 4、入南瓜丝快速翻炒一分钟左右, 放盐、一点白糖和鸡精调味出锅 2.香葱炒南瓜 原料:南瓜1只 调料:香葱、蒜末、橄榄油、盐 做法: 1、将南瓜去皮,切成片 2、油 锅8成热后,将蒜末放入爆香 3、爆香后,将南瓜片放入,翻炒 4、在翻炒的同时,可以不时地往锅里加水,但不要太多 5、放入盐,炒匀 6、南瓜差不多软和绵了之后,就可以关火 7、撒入香葱,即可出锅"
|
|
|
]
|
|
|
embeddings = model.encode(input_texts, normalize_embeddings=True)
|
|
|
```
|
|
|
|
|
|
Package requirements
|
|
|
|
|
|
`pip install sentence_transformers~=2.2.2`
|
|
|
|
|
|
Contributors: [michaelfeil](https://huggingface.co/michaelfeil)
|
|
|
|
|
|
## FAQ
|
|
|
|
|
|
**1. Do I need to add the prefix "query: " and "passage: " to input texts?**
|
|
|
|
|
|
Yes, this is how the model is trained, otherwise you will see a performance degradation.
|
|
|
|
|
|
Here are some rules of thumb:
|
|
|
- Use "query: " and "passage: " correspondingly for asymmetric tasks such as passage retrieval in open QA, ad-hoc information retrieval.
|
|
|
|
|
|
- Use "query: " prefix for symmetric tasks such as semantic similarity, bitext mining, paraphrase retrieval.
|
|
|
|
|
|
- Use "query: " prefix if you want to use embeddings as features, such as linear probing classification, clustering.
|
|
|
|
|
|
**2. Why are my reproduced results slightly different from reported in the model card?**
|
|
|
|
|
|
Different versions of `transformers` and `pytorch` could cause negligible but non-zero performance differences.
|
|
|
|
|
|
**3. Why does the cosine similarity scores distribute around 0.7 to 1.0?**
|
|
|
|
|
|
This is a known and expected behavior as we use a low temperature 0.01 for InfoNCE contrastive loss.
|
|
|
|
|
|
For text embedding tasks like text retrieval or semantic similarity,
|
|
|
what matters is the relative order of the scores instead of the absolute values,
|
|
|
so this should not be an issue.
|
|
|
|
|
|
## Citation
|
|
|
|
|
|
If you find our paper or models helpful, please consider cite as follows:
|
|
|
|
|
|
```
|
|
|
@article{wang2024multilingual,
|
|
|
title={Multilingual E5 Text Embeddings: A Technical Report},
|
|
|
author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Yang, Linjun and Majumder, Rangan and Wei, Furu},
|
|
|
journal={arXiv preprint arXiv:2402.05672},
|
|
|
year={2024}
|
|
|
}
|
|
|
```
|
|
|
|
|
|
## Limitations
|
|
|
|
|
|
Long texts will be truncated to at most 512 tokens.
|
|
|
|
