ayushexel/emb-all-MiniLM-L6-v2-squad-9-epochs

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("ayushexel/emb-all-MiniLM-L6-v2-squad-9-epochs")
# Run inference
sentences = [
    'What is the extinction caused by the splitting of Pangaea called?',
    'The Late Jurassic spans from 163 million to 145 million years ago. The Late Jurassic featured a massive extinction of sauropods and Ichthyosaurs due to the separation of Pangaea into Laurasia and Gondwana in an extinction known as the Jurassic-Cretaceous extinction. Sea levels rose, destroying fern prairies and creating shallows in its wake. Ichthyosaurs went extinct whereas sauropods, as a whole, did not die out in the Jurassic; in fact, some species, like the Titanosaurus, lived up to the K-T extinction. The increase in sea-levels opened up the Atlantic sea way which would continue to get larger over time. The divided world would give opportunity for the diversification of new dinosaurs.',
    "The era began in the wake of the Permian–Triassic extinction event, the largest well-documented mass extinction in Earth's history, and ended with the Cretaceous–Paleogene extinction event, another mass extinction which is known for having killed off non-avian dinosaurs, as well as other plant and animal species. The Mesozoic was a time of significant tectonic, climate and evolutionary activity. The era witnessed the gradual rifting of the supercontinent Pangaea into separate landmasses that would eventually move into their current positions. The climate of the Mesozoic was varied, alternating between warming and cooling periods. Overall, however, the Earth was hotter than it is today. Non-avian dinosaurs appeared in the Late Triassic and became the dominant terrestrial vertebrates early in the Jurassic, occupying this position for about 135 million years until their demise at the end of the Cretaceous. Birds first appeared in the Jurassic, having evolved from a branch of theropod dinosaurs. The first mammals also appeared during the Mesozoic, but would remain small—less than 15 kg (33 lb)—until the Cenozoic.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

• Dataset: gooqa-dev
• Evaluated with TripletEvaluator

Metric	Value
cosine_accuracy	0.4046

Training Details

Training Dataset

Unnamed Dataset

• Size: 44,285 training samples
• Columns: question, context, and negative
• Approximate statistics based on the first 1000 samples:

  	question	context	negative
  type	string	string	string
  details	min: 7 tokens mean: 14.95 tokens max: 41 tokens	min: 28 tokens mean: 149.94 tokens max: 256 tokens	min: 30 tokens mean: 149.99 tokens max: 256 tokens

• Samples:

  question	context	negative
  In 2005, where did Rajan deliver a controversial paper that was critical of the financial paper?	Current Governor of the Reserve Bank of India Raghuram Rajan had predicted the crisis in 2005 when he became chief economist at the International Monetary Fund.In 2005, at a celebration honouring Alan Greenspan, who was about to retire as chairman of the US Federal Reserve, Rajan delivered a controversial paper that was critical of the financial sector. In that paper, "Has Financial Development Made the World Riskier?", Rajan "argued that disaster might loom." Rajan argued that financial sector managers were encouraged to "take risks that generate severe adverse consequences with small probability but, in return, offer generous compensation the rest of the time. These risks are known as tail risks. But perhaps the most important concern is whether banks will be able to provide liquidity to financial markets so that if the tail risk does materialise, financial positions can be unwound and losses allocated so that the consequences to the real economy are minimised."	Paper made from mechanical pulp contains significant amounts of lignin, a major component in wood. In the presence of light and oxygen, lignin reacts to give yellow materials, which is why newsprint and other mechanical paper yellows with age. Paper made from bleached kraft or sulfite pulps does not contain significant amounts of lignin and is therefore better suited for books, documents and other applications where whiteness of the paper is essential.
  Name a National Park in Florida	A peninsula between the Gulf of Mexico, the Atlantic Ocean, and the Straits of Florida, it has the longest coastline in the contiguous United States, approximately 1,350 miles (2,170 km), and is the only state that borders both the Gulf of Mexico and the Atlantic Ocean. Much of the state is at or near sea level and is characterized by sedimentary soil. The climate varies from subtropical in the north to tropical in the south. The American alligator, American crocodile, Florida panther, and manatee can be found in the Everglades National Park.	An example of the second view at the State level is found in the view of the Florida Supreme Court, that only the Florida Supreme Court may license and regulate attorneys appearing before the courts of Florida, and only the Florida Supreme Court may set rules for procedures in the Florida courts.[citation needed] The State of New Hampshire also follows this system.[citation needed]
  What keeps the cultivator from rebirth in the four realms of existance?	Śīla is the foundation of Samādhi/Bhāvana (Meditative cultivation) or mind cultivation. Keeping the precepts promotes not only the peace of mind of the cultivator, which is internal, but also peace in the community, which is external. According to the Law of Karma, keeping the precepts is meritorious and it acts as causes that would bring about peaceful and happy effects. Keeping these precepts keeps the cultivator from rebirth in the four woeful realms of existence.	The above are further subdivided into 31 planes of existence.[web 4] Rebirths in some of the higher heavens, known as the Śuddhāvāsa Worlds or Pure Abodes, can be attained only by skilled Buddhist practitioners known as anāgāmis (non-returners). Rebirths in the Ārūpyadhātu (formless realms) can be attained by only those who can meditate on the arūpajhānas, the highest object of meditation.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 5,000 evaluation samples
• Columns: question, context, and negative_1
• Approximate statistics based on the first 1000 samples:

  	question	context	negative_1
  type	string	string	string
  details	min: 7 tokens mean: 14.45 tokens max: 52 tokens	min: 33 tokens mean: 148.17 tokens max: 256 tokens	min: 28 tokens mean: 143.89 tokens max: 256 tokens

• Samples:

  question	context	negative_1
  How many commanly cited estimates did the body report on?	The U.S. GAO's 2010 findings regarding the great difficulty of accurately gauging the economic impact of copyright infringement was reinforced within the same report by the body's research into three commonly cited estimates that had previously been provided to U.S. agencies. The GAO report explained that the sources – a Federal Bureau of Investigation (FBI) estimate, a Customs and Border Protection (CBP) press release and a Motor and Equipment Manufacturers Association estimate – "cannot be substantiated or traced back to an underlying data source or methodology."	The U.S. GAO's 2010 findings regarding the great difficulty of accurately gauging the economic impact of copyright infringement was reinforced within the same report by the body's research into three commonly cited estimates that had previously been provided to U.S. agencies. The GAO report explained that the sources – a Federal Bureau of Investigation (FBI) estimate, a Customs and Border Protection (CBP) press release and a Motor and Equipment Manufacturers Association estimate – "cannot be substantiated or traced back to an underlying data source or methodology."
  Who introduced the term biological diversity to the scientific community?	The term biological diversity was used first by wildlife scientist and conservationist Raymond F. Dasmann in the year 1968 lay book A Different Kind of Country advocating conservation. The term was widely adopted only after more than a decade, when in the 1980s it came into common usage in science and environmental policy. Thomas Lovejoy, in the foreword to the book Conservation Biology, introduced the term to the scientific community. Until then the term "natural diversity" was common, introduced by The Science Division of The Nature Conservancy in an important 1975 study, "The Preservation of Natural Diversity." By the early 1980s TNC's Science program and its head, Robert E. Jenkins, Lovejoy and other leading conservation scientists at the time in America advocated the use of the term "biological diversity".	The term biological diversity was used first by wildlife scientist and conservationist Raymond F. Dasmann in the year 1968 lay book A Different Kind of Country advocating conservation. The term was widely adopted only after more than a decade, when in the 1980s it came into common usage in science and environmental policy. Thomas Lovejoy, in the foreword to the book Conservation Biology, introduced the term to the scientific community. Until then the term "natural diversity" was common, introduced by The Science Division of The Nature Conservancy in an important 1975 study, "The Preservation of Natural Diversity." By the early 1980s TNC's Science program and its head, Robert E. Jenkins, Lovejoy and other leading conservation scientists at the time in America advocated the use of the term "biological diversity".
  Which child of Jefferson identified as black?	Of numerous relationships between male slaveholders, overseers, or master's sons and women slaves, the most notable is likely that of President Thomas Jefferson with his slave Sally Hemings. As noted in the 2012 collaborative Smithsonian-Monticello exhibit, Slavery at Monticello: The Paradox of Liberty, Jefferson, then a widower, took Hemings as his concubine for nearly 40 years. They had six children of record; four Hemings children survived into adulthood, and he freed them all, among the very few slaves he freed. Two were allowed to "escape" to the North in 1822, and two were granted freedom by his will upon his death in 1826. Seven-eighths white by ancestry, all four of his Hemings children moved to northern states as adults; three of the four entered the white community, and all their descendants identified as white. Of the descendants of Madison Hemings, who continued to identify as black, some in future generations eventually identified as white and "married out", while others c...	Of numerous relationships between male slaveholders, overseers, or master's sons and women slaves, the most notable is likely that of President Thomas Jefferson with his slave Sally Hemings. As noted in the 2012 collaborative Smithsonian-Monticello exhibit, Slavery at Monticello: The Paradox of Liberty, Jefferson, then a widower, took Hemings as his concubine for nearly 40 years. They had six children of record; four Hemings children survived into adulthood, and he freed them all, among the very few slaves he freed. Two were allowed to "escape" to the North in 1822, and two were granted freedom by his will upon his death in 1826. Seven-eighths white by ancestry, all four of his Hemings children moved to northern states as adults; three of the four entered the white community, and all their descendants identified as white. Of the descendants of Madison Hemings, who continued to identify as black, some in future generations eventually identified as white and "married out", while others c...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 256
• num_train_epochs: 9
• warmup_ratio: 0.1
• fp16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 256
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 9
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss	gooqa-dev_cosine_accuracy
-1	-1	-	-	0.3286
0.5780	100	0.5627	0.8998	0.3924
1.1561	200	0.4739	0.8606	0.3996
1.7341	300	0.4257	0.8522	0.4082
2.3121	400	0.366	0.8413	0.4100
2.8902	500	0.3256	0.8489	0.4066
3.4682	600	0.2852	0.8421	0.4048
4.0462	700	0.2709	0.8474	0.4068
4.6243	800	0.2306	0.8498	0.4044
5.2023	900	0.2268	0.8433	0.4124
5.7803	1000	0.2032	0.8497	0.4152
6.3584	1100	0.1911	0.8434	0.4122
6.9364	1200	0.1878	0.8470	0.4116
7.5145	1300	0.1732	0.8523	0.4110
8.0925	1400	0.1708	0.8548	0.4148
8.6705	1500	0.1657	0.8517	0.4078
-1	-1	-	-	0.4046

Framework Versions

• Python: 3.11.0
• Sentence Transformers: 4.0.1
• Transformers: 4.50.3
• PyTorch: 2.6.0+cu124
• Accelerate: 1.5.2
• Datasets: 3.5.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}