rahulseetharaman/reranker-bert-uncased_L-12_H-768_A-12-msmarco-bce

CrossEncoder based on bansalaman18/bert-uncased_L-12_H-768_A-12

This is a Cross Encoder model finetuned from bansalaman18/bert-uncased_L-12_H-768_A-12 on the msmarco dataset using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: bansalaman18/bert-uncased_L-12_H-768_A-12
• Maximum Sequence Length: 512 tokens
• Number of Output Labels: 1 label
• Training Dataset:
  • msmarco
• Language: en

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("rahulseetharaman/reranker-bert-uncased_L-12_H-768_A-12-msmarco-bce")
# Get scores for pairs of texts
pairs = [
    ['are solar pool covers worth it', 'If you are using Onga pool pumps or Hurlcon pool pumps, then you need not worry about them getting overheated for they are one of the best pool pumps available on the market. If you want to know about What causes a pool pump to overheat so please visit here onga pool pumps.'],
    ['how much do Customer Service Agent: Ticketing/Gate make in general', '$41,000. Average Airport Customer Service Ticketing Gate Agent salaries for job postings in Houston, TX are 13% higher than average Airport Customer Service Ticketing Gate Agent salaries for job postings nationwide.verage Airport Customer Service Ticketing Gate Agent salaries for job postings in Houston, TX are 13% higher than average Airport Customer Service Ticketing Gate Agent salaries for job postings nationwide.'],
    ['what is adverse selection economics', 'The last first woman to win the Nobel in her category was Elinor Ostrom, who shared the 2009 economics prize for her groundbreaking analysis of common property. The wait was so long for a woman economics laureate in part because that prize wasnâ\x80\x99t established until 1969.'],
    ['where do newts live', 'Newts can be found living in North America, Europe and Asia. They are not found in Australia or Africa. In fact there are no species of salamander that live in Australia and only a few found in Northern Africa. Seven species of newt live in Europe.'],
    ['define: rolling hourly average', 'An example of two moving average curves. In statistics, a moving average (rolling average or running average) is a calculation to analyze data points by creating series of averages of different subsets of the full data set. It is also called a moving mean (MM) or rolling mean and is a type of finite impulse response filter.'],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'are solar pool covers worth it',
    [
        'If you are using Onga pool pumps or Hurlcon pool pumps, then you need not worry about them getting overheated for they are one of the best pool pumps available on the market. If you want to know about What causes a pool pump to overheat so please visit here onga pool pumps.',
        '$41,000. Average Airport Customer Service Ticketing Gate Agent salaries for job postings in Houston, TX are 13% higher than average Airport Customer Service Ticketing Gate Agent salaries for job postings nationwide.verage Airport Customer Service Ticketing Gate Agent salaries for job postings in Houston, TX are 13% higher than average Airport Customer Service Ticketing Gate Agent salaries for job postings nationwide.',
        'The last first woman to win the Nobel in her category was Elinor Ostrom, who shared the 2009 economics prize for her groundbreaking analysis of common property. The wait was so long for a woman economics laureate in part because that prize wasnâ\x80\x99t established until 1969.',
        'Newts can be found living in North America, Europe and Asia. They are not found in Australia or Africa. In fact there are no species of salamander that live in Australia and only a few found in Northern Africa. Seven species of newt live in Europe.',
        'An example of two moving average curves. In statistics, a moving average (rolling average or running average) is a calculation to analyze data points by creating series of averages of different subsets of the full data set. It is also called a moving mean (MM) or rolling mean and is a type of finite impulse response filter.',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Evaluation

Metrics

Cross Encoder Reranking

• Datasets: NanoMSMARCO_R100, NanoNFCorpus_R100 and NanoNQ_R100
• Evaluated with CrossEncoderRerankingEvaluator with these parameters:

  {
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Metric	NanoMSMARCO_R100	NanoNFCorpus_R100	NanoNQ_R100
map	0.0574 (-0.4322)	0.3105 (+0.0495)	0.0552 (-0.3644)
mrr@10	0.0327 (-0.4448)	0.4762 (-0.0236)	0.0285 (-0.3982)
ndcg@10	0.0435 (-0.4969)	0.3018 (-0.0233)	0.0398 (-0.4608)

Cross Encoder Nano BEIR

• Dataset: NanoBEIR_R100_mean
• Evaluated with CrossEncoderNanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nfcorpus",
          "nq"
      ],
      "rerank_k": 100,
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Metric	Value
map	0.1410 (-0.2491)
mrr@10	0.1791 (-0.2889)
ndcg@10	0.1284 (-0.3270)

Training Details

Training Dataset

msmarco

• Dataset: msmarco at 9e329ed
• Size: 90,000 training samples
• Columns: query, passage, and score
• Approximate statistics based on the first 1000 samples:

  	query	passage	score
  type	string	string	float
  details	min: 7 characters mean: 33.59 characters max: 164 characters	min: 49 characters mean: 340.88 characters max: 1018 characters	min: 0.0 mean: 0.53 max: 1.0

• Samples:

  query	passage	score
  fantomcoin current price	The current Average monthly rental price per square meter for a studio property in Pretoria / Tshwane on Gumtree is R 47.	0.0
  ddp price definition	Delivered Duty Paid - DDP. Loading the player... What does 'Delivered Duty Paid - DDP' mean. Delivered duty paid (DDP) is a transaction where the seller pays for the total costs associated with transporting goods and is fully responsible for the goods until they are received and transferred to the buyer.	1.0
  what is neil diamond's hometown	Oct 6, 2014 8:00 am ET. Brooklyn native Neil Diamond played his first-ever hometown show last week with a 10-song set at Erasmus Hall High School, where he sang in the choir during the two years he was a student there. Speakeasy today premieres a clip of Diamond performing the new song “Something Blue” at that concert.	1.0

• Loss: BinaryCrossEntropyLoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "pos_weight": null
  }
  

Evaluation Dataset

msmarco

• Dataset: msmarco at 9e329ed
• Size: 10,000 evaluation samples
• Columns: query, passage, and score
• Approximate statistics based on the first 1000 samples:

  	query	passage	score
  type	string	string	float
  details	min: 9 characters mean: 34.17 characters max: 146 characters	min: 83 characters mean: 349.58 characters max: 974 characters	min: 0.0 mean: 0.51 max: 1.0

• Samples:

  query	passage	score
  are solar pool covers worth it	If you are using Onga pool pumps or Hurlcon pool pumps, then you need not worry about them getting overheated for they are one of the best pool pumps available on the market. If you want to know about What causes a pool pump to overheat so please visit here onga pool pumps.	0.0
  how much do Customer Service Agent: Ticketing/Gate make in general	$41,000. Average Airport Customer Service Ticketing Gate Agent salaries for job postings in Houston, TX are 13% higher than average Airport Customer Service Ticketing Gate Agent salaries for job postings nationwide.verage Airport Customer Service Ticketing Gate Agent salaries for job postings in Houston, TX are 13% higher than average Airport Customer Service Ticketing Gate Agent salaries for job postings nationwide.	1.0
  what is adverse selection economics	The last first woman to win the Nobel in her category was Elinor Ostrom, who shared the 2009 economics prize for her groundbreaking analysis of common property. The wait was so long for a woman economics laureate in part because that prize wasn’t established until 1969.	0.0

• Loss: BinaryCrossEntropyLoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "pos_weight": null
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 4
• warmup_ratio: 0.1
• seed: 12
• bf16: True
• dataloader_num_workers: 4
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 12
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 4
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	NanoMSMARCO_R100_ndcg@10	NanoNFCorpus_R100_ndcg@10	NanoNQ_R100_ndcg@10	NanoBEIR_R100_mean_ndcg@10
-1	-1	-	-	0.0355 (-0.5049)	0.2692 (-0.0558)	0.0305 (-0.4701)	0.1118 (-0.3436)
0.0002	1	0.6151	-	-	-	-	-
0.1778	1000	0.6964	0.7084	0.0435 (-0.4969)	0.3018 (-0.0233)	0.0398 (-0.4608)	0.1284 (-0.3270)
0.3556	2000	0.6961	0.7105	0.0531 (-0.4874)	0.2442 (-0.0808)	0.0330 (-0.4677)	0.1101 (-0.3453)
0.5333	3000	0.6991	0.6928	0.0321 (-0.5083)	0.2538 (-0.0712)	0.0191 (-0.4815)	0.1017 (-0.3537)
0.7111	4000	0.6974	0.6974	0.0659 (-0.4745)	0.2469 (-0.0781)	0.0572 (-0.4434)	0.1234 (-0.3320)
0.8889	5000	0.6982	0.6943	0.0285 (-0.5120)	0.2260 (-0.0990)	0.0275 (-0.4732)	0.0940 (-0.3614)
1.0667	6000	0.6975	0.6974	0.0769 (-0.4635)	0.2280 (-0.0970)	0.0510 (-0.4496)	0.1187 (-0.3367)
1.2444	7000	0.6973	0.6939	0.0535 (-0.4869)	0.2394 (-0.0856)	0.0493 (-0.4514)	0.1141 (-0.3413)
1.4222	8000	0.6975	0.6929	0.0385 (-0.5019)	0.2416 (-0.0835)	0.0564 (-0.4442)	0.1122 (-0.3432)
1.6	9000	0.6972	0.6940	0.0611 (-0.4793)	0.2568 (-0.0682)	0.0337 (-0.4669)	0.1172 (-0.3382)
1.7778	10000	0.6966	0.6937	0.0400 (-0.5004)	0.2552 (-0.0699)	0.0595 (-0.4412)	0.1182 (-0.3372)
1.9556	11000	0.6964	0.6935	0.0199 (-0.5205)	0.2658 (-0.0592)	0.0202 (-0.4804)	0.1020 (-0.3534)
2.1333	12000	0.6964	0.6934	0.0427 (-0.4978)	0.2532 (-0.0719)	0.0476 (-0.4531)	0.1145 (-0.3409)
2.3111	13000	0.6963	0.6932	0.0459 (-0.4945)	0.2495 (-0.0756)	0.0344 (-0.4663)	0.1099 (-0.3455)
2.4889	14000	0.6958	0.6929	0.0380 (-0.5025)	0.2315 (-0.0935)	0.0155 (-0.4851)	0.0950 (-0.3604)
2.6667	15000	0.6959	0.6930	0.0480 (-0.4924)	0.2505 (-0.0745)	0.0328 (-0.4678)	0.1104 (-0.3449)
2.8444	16000	0.6954	0.6942	0.0345 (-0.5059)	0.2398 (-0.0853)	0.0209 (-0.4798)	0.0984 (-0.3570)
3.0222	17000	0.6956	0.6935	0.0708 (-0.4696)	0.2289 (-0.0961)	0.0565 (-0.4441)	0.1187 (-0.3366)
3.2	18000	0.6956	0.6945	0.0460 (-0.4944)	0.2644 (-0.0606)	0.0269 (-0.4738)	0.1124 (-0.3429)
3.3778	19000	0.6954	0.6946	0.0366 (-0.5038)	0.2576 (-0.0675)	0.0230 (-0.4777)	0.1057 (-0.3496)
3.5556	20000	0.6954	0.6936	0.0316 (-0.5088)	0.2317 (-0.0933)	0.0309 (-0.4698)	0.0981 (-0.3573)
3.7333	21000	0.6948	0.6929	0.0393 (-0.5011)	0.2546 (-0.0705)	0.0452 (-0.4555)	0.1130 (-0.3423)
3.9111	22000	0.6952	0.6937	0.0432 (-0.4973)	0.2617 (-0.0633)	0.0372 (-0.4634)	0.1140 (-0.3413)
-1	-1	-	-	0.0435 (-0.4969)	0.3018 (-0.0233)	0.0398 (-0.4608)	0.1284 (-0.3270)

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.18
• Sentence Transformers: 5.0.0
• Transformers: 4.56.0.dev0
• PyTorch: 2.7.1+cu126
• Accelerate: 1.9.0
• Datasets: 4.0.0
• Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}