Omartificial-Intelligence-Space commited on Jun 6

Commit

31eb4bc

verified ·

1 Parent(s): a23d2b3

upload mdoel files

Browse files

Files changed (24) hide show

.gitattributes +1 -0
1_Pooling/config.json +10 -0
README.md +727 -3
added_tokens.json +28 -0
config.json +30 -0
config_sentence_transformers.json +10 -0
merges.txt +0 -0
model.safetensors +3 -0
modules.json +14 -0
optimizer.pt +3 -0
rng_state_0.pth +3 -0
rng_state_1.pth +3 -0
rng_state_2.pth +3 -0
rng_state_3.pth +3 -0
rng_state_4.pth +3 -0
rng_state_5.pth +3 -0
scheduler.pt +3 -0
sentence_bert_config.json +4 -0
special_tokens_map.json +31 -0
tokenizer.json +3 -0
tokenizer_config.json +240 -0
trainer_state.json +2274 -0
training_args.bin +3 -0
vocab.json +0 -0

.gitattributes CHANGED Viewed

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text

 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

1_Pooling/config.json ADDED Viewed

	@@ -0,0 +1,10 @@

+{
+  "word_embedding_dimension": 1024,
+  "pooling_mode_cls_token": false,
+  "pooling_mode_mean_tokens": true,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false,
+  "pooling_mode_weightedmean_tokens": false,
+  "pooling_mode_lasttoken": false,
+  "include_prompt": true
+}

README.md CHANGED Viewed

@@ -1,3 +1,727 @@
----
-license: apache-2.0
----

+---
+language:
+- en
+tags:
+- sentence-transformers
+- sentence-similarity
+- feature-extraction
+- generated_from_trainer
+- dataset_size:2280319
+- loss:MatryoshkaLoss
+- loss:MultipleNegativesRankingLoss
+base_model: Qwen/Qwen3-Embedding-0.6B
+widget:
+- source_sentence: I'd suggest you find a bank in your local country, and consider
+    opening a Euro denominated bank account with them.
+  sentences:
+  - You can mix those, but in my experience, it will be very difficult at first.
+  - The woman is pencilling on eye shadow.
+  - I am not sure if you can open a bank account in France if you are not a resident.
+- source_sentence: Black and white image of a wave crashing in the ocean.
+  sentences:
+  - a small black dog in the ocean with some rocks in the background
+  - A woman is riding an elephant.
+  - A yellow and orange bird hold on to the side of a cage.
+- source_sentence: If you can get over the "ick factor," you have an easily-applied
+    source of organic nitrogen fertilizer close at hand.
+  sentences:
+  - The NPK numbers on the fertilizer represents the percent, by weight, of Nitrogen,
+    P2O5 and K2O, respectively.
+  - Wikipedia's compilation of Time Travel Rules is a good resource to check about
+    this subject.
+  - A man is playing a flute.
+- source_sentence: A man is speaking.
+  sentences:
+  - A man is dancing.
+  - A cougar is chasing a bear.
+  - For completeness, Apple's Pages has quite a few nice poster layouts.
+- source_sentence: Left side of a silver train engine.
+  sentences:
+  - A monkey is riding a bus.
+  - One idea that's been going around at least since the 80s is that you can distinguish
+    between Holds and Moves.
+  - A close-up of a black train engine.
+datasets:
+- silma-ai/silma-arabic-triplets-dataset-v1.0
+- sentence-transformers/stsb
+pipeline_tag: sentence-similarity
+library_name: sentence-transformers
+---
+# SentenceTransformer based on Qwen/Qwen3-Embedding-0.6B
+This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [Qwen/Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) on the [silma-arabic-triplets-dataset-v1.0](https://huggingface.co/datasets/silma-ai/silma-arabic-triplets-dataset-v1.0) dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
+## Model Details
+### Model Description
+- **Model Type:** Sentence Transformer
+- **Base model:** [Qwen/Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) <!-- at revision a579a21d7aff542145eebef8d60ed73ec281a0b4 -->
+- **Maximum Sequence Length:** 32768 tokens
+- **Output Dimensionality:** 1024 dimensions
+- **Similarity Function:** Cosine Similarity
+- **Training Dataset:**
+    - [silma-arabic-triplets-dataset-v1.0](https://huggingface.co/datasets/silma-ai/silma-arabic-triplets-dataset-v1.0)
+- **Language:** en
+<!-- - **License:** Unknown -->
+### Model Sources
+- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
+- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
+- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
+### Full Model Architecture
+```
+SentenceTransformer(
+  (0): Transformer({'max_seq_length': 32768, 'do_lower_case': False}) with Transformer model: Qwen3Model
+  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
+)
+```
+## Usage
+### Direct Usage (Sentence Transformers)
+First install the Sentence Transformers library:
+```bash
+pip install -U sentence-transformers
+```
+Then you can load this model and run inference.
+```python
+from sentence_transformers import SentenceTransformer
+# Download from the 🤗 Hub
+model = SentenceTransformer("sentence_transformers_model_id")
+# Run inference
+sentences = [
+    'Left side of a silver train engine.',
+    'A close-up of a black train engine.',
+    "One idea that's been going around at least since the 80s is that you can distinguish between Holds and Moves.",
+]
+embeddings = model.encode(sentences)
+print(embeddings.shape)
+# [3, 1024]
+# Get the similarity scores for the embeddings
+similarities = model.similarity(embeddings, embeddings)
+print(similarities.shape)
+# [3, 3]
+```
+<!--
+### Direct Usage (Transformers)
+<details><summary>Click to see the direct usage in Transformers</summary>
+</details>
+-->
+<!--
+### Downstream Usage (Sentence Transformers)
+You can finetune this model on your own dataset.
+<details><summary>Click to expand</summary>
+</details>
+-->
+<!--
+### Out-of-Scope Use
+*List how the model may foreseeably be misused and address what users ought not to do with the model.*
+-->
+<!--
+## Bias, Risks and Limitations
+*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
+-->
+<!--
+### Recommendations
+*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
+-->
+## Training Details
+### Training Dataset
+#### silma-arabic-triplets-dataset-v1.0
+* Dataset: [silma-arabic-triplets-dataset-v1.0](https://huggingface.co/datasets/silma-ai/silma-arabic-triplets-dataset-v1.0) at [77f8f6b](https://huggingface.co/datasets/silma-ai/silma-arabic-triplets-dataset-v1.0/tree/77f8f6b223049e7e5968929f5d6cdd2320d1a6dc)
+* Size: 2,280,319 training samples
+* Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
+* Approximate statistics based on the first 1000 samples:
+  |         | anchor                                                                            | positive                                                                          | negative                                                                          |
+  |:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|
+  | type    | string                                                                            | string                                                                            | string                                                                            |
+  | details | <ul><li>min: 2 tokens</li><li>mean: 17.04 tokens</li><li>max: 64 tokens</li></ul> | <ul><li>min: 2 tokens</li><li>mean: 15.17 tokens</li><li>max: 96 tokens</li></ul> | <ul><li>min: 3 tokens</li><li>mean: 14.81 tokens</li><li>max: 57 tokens</li></ul> |
+* Samples:
+  | anchor                                                                         | positive                                             | negative                                                                |
+  |:-------------------------------------------------------------------------------|:-----------------------------------------------------|:------------------------------------------------------------------------|
+  | <code>صبي صغير وفتاة صغيرة يركبان دراجتيهما على الرصيف مع عجلات مساعدة.</code> | <code>فتى وفتاة يتعلمون ركوب الدراجات</code>         | <code>الصبي الصغير يصل إلى العصا من الفتاة وهو يدير سباق التتابع</code> |
+  | <code>كيف أتجنب التفكير في نفسي أكثر من اللازم؟</code>                         | <code>كيف يمكنني تجنب التفكير أكثر من اللازم؟</code> | <code>كيف أتطوّر قدرة التفكير؟</code>                                   |
+  | <code>ما هو أفضل كتاب يقرأه مراهق؟</code>                                      | <code>ما هو أفضل كتاب للمراهقين؟</code>              | <code>ما هي الكتب التي يمكن للطلاب قراءتها؟</code>                      |
+* Loss: [<code>MatryoshkaLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters:
+  ```json
+  {
+      "loss": "MultipleNegativesRankingLoss",
+      "matryoshka_dims": [
+          1024,
+          768,
+          512,
+          256
+      ],
+      "matryoshka_weights": [
+          1,
+          1,
+          1,
+          1
+      ],
+      "n_dims_per_step": -1
+  }
+  ```
+### Evaluation Dataset
+#### stsb
+* Dataset: [stsb](https://huggingface.co/datasets/sentence-transformers/stsb) at [ab7a5ac](https://huggingface.co/datasets/sentence-transformers/stsb/tree/ab7a5ac0e35aa22088bdcf23e7fd99b220e53308)
+* Size: 1,500 evaluation samples
+* Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>score</code>
+* Approximate statistics based on the first 1000 samples:
+  |         | sentence1                                                                         | sentence2                                                                         | score                                                          |
+  |:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:---------------------------------------------------------------|
+  | type    | string                                                                            | string                                                                            | float                                                          |
+  | details | <ul><li>min: 3 tokens</li><li>mean: 12.98 tokens</li><li>max: 46 tokens</li></ul> | <ul><li>min: 4 tokens</li><li>mean: 12.96 tokens</li><li>max: 51 tokens</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.42</li><li>max: 1.0</li></ul> |
+* Samples:
+  | sentence1                                         | sentence2                                             | score             |
+  |:--------------------------------------------------|:------------------------------------------------------|:------------------|
+  | <code>A man with a hard hat is dancing.</code>    | <code>A man wearing a hard hat is dancing.</code>     | <code>1.0</code>  |
+  | <code>A young child is riding a horse.</code>     | <code>A child is riding a horse.</code>               | <code>0.95</code> |
+  | <code>A man is feeding a mouse to a snake.</code> | <code>The man is feeding a mouse to the snake.</code> | <code>1.0</code>  |
+* Loss: [<code>MatryoshkaLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters:
+  ```json
+  {
+      "loss": "MultipleNegativesRankingLoss",
+      "matryoshka_dims": [
+          1024,
+          768,
+          512,
+          256
+      ],
+      "matryoshka_weights": [
+          1,
+          1,
+          1,
+          1
+      ],
+      "n_dims_per_step": -1
+  }
+  ```
+### Training Hyperparameters
+#### Non-Default Hyperparameters
+- `per_device_train_batch_size`: 32
+- `per_device_eval_batch_size`: 32
+- `warmup_ratio`: 0.1
+- `bf16`: True
+- `batch_sampler`: no_duplicates
+#### All Hyperparameters
+<details><summary>Click to expand</summary>
+- `overwrite_output_dir`: False
+- `do_predict`: False
+- `eval_strategy`: no
+- `prediction_loss_only`: True
+- `per_device_train_batch_size`: 32
+- `per_device_eval_batch_size`: 32
+- `per_gpu_train_batch_size`: None
+- `per_gpu_eval_batch_size`: None
+- `gradient_accumulation_steps`: 1
+- `eval_accumulation_steps`: None
+- `torch_empty_cache_steps`: None
+- `learning_rate`: 5e-05
+- `weight_decay`: 0.0
+- `adam_beta1`: 0.9
+- `adam_beta2`: 0.999
+- `adam_epsilon`: 1e-08
+- `max_grad_norm`: 1.0
+- `num_train_epochs`: 3
+- `max_steps`: -1
+- `lr_scheduler_type`: linear
+- `lr_scheduler_kwargs`: {}
+- `warmup_ratio`: 0.1
+- `warmup_steps`: 0
+- `log_level`: passive
+- `log_level_replica`: warning
+- `log_on_each_node`: True
+- `logging_nan_inf_filter`: True
+- `save_safetensors`: True
+- `save_on_each_node`: False
+- `save_only_model`: False
+- `restore_callback_states_from_checkpoint`: False
+- `no_cuda`: False
+- `use_cpu`: False
+- `use_mps_device`: False
+- `seed`: 42
+- `data_seed`: None
+- `jit_mode_eval`: False
+- `use_ipex`: False
+- `bf16`: True
+- `fp16`: False
+- `fp16_opt_level`: O1
+- `half_precision_backend`: auto
+- `bf16_full_eval`: False
+- `fp16_full_eval`: False
+- `tf32`: None
+- `local_rank`: 0
+- `ddp_backend`: None
+- `tpu_num_cores`: None
+- `tpu_metrics_debug`: False
+- `debug`: []
+- `dataloader_drop_last`: True
+- `dataloader_num_workers`: 0
+- `dataloader_prefetch_factor`: None
+- `past_index`: -1
+- `disable_tqdm`: False
+- `remove_unused_columns`: True
+- `label_names`: None
+- `load_best_model_at_end`: False
+- `ignore_data_skip`: False
+- `fsdp`: []
+- `fsdp_min_num_params`: 0
+- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
+- `tp_size`: 0
+- `fsdp_transformer_layer_cls_to_wrap`: None
+- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
+- `deepspeed`: None
+- `label_smoothing_factor`: 0.0
+- `optim`: adamw_torch
+- `optim_args`: None
+- `adafactor`: False
+- `group_by_length`: False
+- `length_column_name`: length
+- `ddp_find_unused_parameters`: None
+- `ddp_bucket_cap_mb`: None
+- `ddp_broadcast_buffers`: False
+- `dataloader_pin_memory`: True
+- `dataloader_persistent_workers`: False
+- `skip_memory_metrics`: True
+- `use_legacy_prediction_loop`: False
+- `push_to_hub`: False
+- `resume_from_checkpoint`: None
+- `hub_model_id`: None
+- `hub_strategy`: every_save
+- `hub_private_repo`: None
+- `hub_always_push`: False
+- `gradient_checkpointing`: False
+- `gradient_checkpointing_kwargs`: None
+- `include_inputs_for_metrics`: False
+- `include_for_metrics`: []
+- `eval_do_concat_batches`: True
+- `fp16_backend`: auto
+- `push_to_hub_model_id`: None
+- `push_to_hub_organization`: None
+- `mp_parameters`:
+- `auto_find_batch_size`: False
+- `full_determinism`: False
+- `torchdynamo`: None
+- `ray_scope`: last
+- `ddp_timeout`: 1800
+- `torch_compile`: False
+- `torch_compile_backend`: None
+- `torch_compile_mode`: None
+- `include_tokens_per_second`: False
+- `include_num_input_tokens_seen`: False
+- `neftune_noise_alpha`: None
+- `optim_target_modules`: None
+- `batch_eval_metrics`: False
+- `eval_on_start`: False
+- `use_liger_kernel`: False
+- `eval_use_gather_object`: False
+- `average_tokens_across_devices`: False
+- `prompts`: None
+- `batch_sampler`: no_duplicates
+- `multi_dataset_batch_sampler`: proportional
+</details>
+### Training Logs
+<details><summary>Click to expand</summary>
+| Epoch  | Step  | Training Loss |
+|:------:|:-----:|:-------------:|
+| 0.2189 | 2600  | 0.2615        |
+| 0.2273 | 2700  | 0.2518        |
+| 0.2358 | 2800  | 0.2785        |
+| 0.2442 | 2900  | 0.2541        |
+| 0.2526 | 3000  | 0.266         |
+| 0.2610 | 3100  | 0.2671        |
+| 0.2695 | 3200  | 0.2895        |
+| 0.2779 | 3300  | 0.2658        |
+| 0.2863 | 3400  | 0.2622        |
+| 0.2947 | 3500  | 0.2607        |
+| 0.3031 | 3600  | 0.2883        |
+| 0.3116 | 3700  | 0.2747        |
+| 0.3200 | 3800  | 0.2525        |
+| 0.3284 | 3900  | 0.2471        |
+| 0.3368 | 4000  | 0.2564        |
+| 0.3452 | 4100  | 0.2541        |
+| 0.3537 | 4200  | 0.2421        |
+| 0.3621 | 4300  | 0.2559        |
+| 0.3705 | 4400  | 0.2562        |
+| 0.3789 | 4500  | 0.2548        |
+| 0.3873 | 4600  | 0.2504        |
+| 0.3958 | 4700  | 0.2585        |
+| 0.4042 | 4800  | 0.2368        |
+| 0.4126 | 4900  | 0.2298        |
+| 0.4210 | 5000  | 0.2277        |
+| 0.4294 | 5100  | 0.2809        |
+| 0.4379 | 5200  | 0.2945        |
+| 0.4463 | 5300  | 0.6972        |
+| 0.4547 | 5400  | 1.6206        |
+| 0.4631 | 5500  | 1.48          |
+| 0.4715 | 5600  | 1.3816        |
+| 0.4800 | 5700  | 1.3296        |
+| 0.4884 | 5800  | 1.2737        |
+| 0.4968 | 5900  | 1.188         |
+| 0.5052 | 6000  | 1.1852        |
+| 0.5136 | 6100  | 1.1271        |
+| 0.5221 | 6200  | 1.1654        |
+| 0.5305 | 6300  | 1.1028        |
+| 0.5389 | 6400  | 1.0713        |
+| 0.5473 | 6500  | 1.0654        |
+| 0.5557 | 6600  | 1.0779        |
+| 0.5642 | 6700  | 1.0448        |
+| 0.5726 | 6800  | 1.022         |
+| 0.5810 | 6900  | 1.0037        |
+| 0.5894 | 7000  | 1.026         |
+| 0.5978 | 7100  | 0.9852        |
+| 0.6063 | 7200  | 0.9583        |
+| 0.6147 | 7300  | 0.974         |
+| 0.6231 | 7400  | 0.9357        |
+| 0.6315 | 7500  | 0.9426        |
+| 0.6399 | 7600  | 0.9001        |
+| 0.6484 | 7700  | 0.8875        |
+| 0.6568 | 7800  | 0.8552        |
+| 0.6652 | 7900  | 0.9011        |
+| 0.6736 | 8000  | 0.8692        |
+| 0.6820 | 8100  | 0.8637        |
+| 0.6905 | 8200  | 0.8292        |
+| 0.6989 | 8300  | 0.8226        |
+| 0.7073 | 8400  | 0.8066        |
+| 0.7157 | 8500  | 0.8495        |
+| 0.7241 | 8600  | 0.794         |
+| 0.7326 | 8700  | 0.7676        |
+| 0.7410 | 8800  | 0.7671        |
+| 0.7494 | 8900  | 0.769         |
+| 0.7578 | 9000  | 0.7808        |
+| 0.7663 | 9100  | 0.7439        |
+| 0.7747 | 9200  | 0.7394        |
+| 0.7831 | 9300  | 0.7662        |
+| 0.7915 | 9400  | 0.7472        |
+| 0.7999 | 9500  | 0.7592        |
+| 0.8084 | 9600  | 0.7599        |
+| 0.8168 | 9700  | 0.7397        |
+| 0.8252 | 9800  | 0.1707        |
+| 0.8336 | 9900  | 0.036         |
+| 0.8420 | 10000 | 0.0272        |
+| 0.8505 | 10100 | 0.0189        |
+| 0.8589 | 10200 | 0.0139        |
+| 0.8673 | 10300 | 0.0155        |
+| 0.8757 | 10400 | 0.0145        |
+| 0.8841 | 10500 | 0.0124        |
+| 0.8926 | 10600 | 0.0098        |
+| 0.9010 | 10700 | 0.0091        |
+| 0.9094 | 10800 | 0.01          |
+| 0.9178 | 10900 | 0.1028        |
+| 0.9262 | 11000 | 0.1921        |
+| 0.9347 | 11100 | 0.1521        |
+| 0.9431 | 11200 | 0.1336        |
+| 0.9515 | 11300 | 0.1238        |
+| 0.9599 | 11400 | 0.1065        |
+| 0.9683 | 11500 | 0.0955        |
+| 0.9768 | 11600 | 0.0944        |
+| 0.9852 | 11700 | 0.0849        |
+| 0.9936 | 11800 | 0.0712        |
+| 1.0020 | 11900 | 0.3673        |
+| 1.0104 | 12000 | 0.4508        |
+| 1.0189 | 12100 | 0.3693        |
+| 1.0273 | 12200 | 0.3297        |
+| 1.0357 | 12300 | 0.3452        |
+| 1.0441 | 12400 | 0.3062        |
+| 1.0525 | 12500 | 0.2824        |
+| 1.0610 | 12600 | 0.2909        |
+| 1.0694 | 12700 | 0.2688        |
+| 1.0778 | 12800 | 0.261         |
+| 1.0862 | 12900 | 0.2616        |
+| 1.0946 | 13000 | 0.2496        |
+| 1.1031 | 13100 | 0.2415        |
+| 1.1115 | 13200 | 0.2469        |
+| 1.1199 | 13300 | 0.2444        |
+| 1.1283 | 13400 | 0.2489        |
+| 1.1367 | 13500 | 0.2338        |
+| 1.1452 | 13600 | 0.2185        |
+| 1.1536 | 13700 | 0.2288        |
+| 1.1620 | 13800 | 0.2032        |
+| 1.1704 | 13900 | 0.2005        |
+| 1.1788 | 14000 | 0.2053        |
+| 1.1873 | 14100 | 0.2042        |
+| 1.1957 | 14200 | 0.1877        |
+| 1.2041 | 14300 | 0.1711        |
+| 1.2125 | 14400 | 0.1973        |
+| 1.2209 | 14500 | 0.1624        |
+| 1.2294 | 14600 | 0.1646        |
+| 1.2378 | 14700 | 0.1702        |
+| 1.2462 | 14800 | 0.166         |
+| 1.2546 | 14900 | 0.1689        |
+| 1.2631 | 15000 | 0.1747        |
+| 1.2715 | 15100 | 0.1755        |
+| 1.2799 | 15200 | 0.1607        |
+| 1.2883 | 15300 | 0.1462        |
+| 1.2967 | 15400 | 0.1475        |
+| 1.3052 | 15500 | 0.1587        |
+| 1.3136 | 15600 | 0.1506        |
+| 1.3220 | 15700 | 0.1467        |
+| 1.3304 | 15800 | 0.1277        |
+| 1.3388 | 15900 | 0.1305        |
+| 1.3473 | 16000 | 0.1292        |
+| 1.3557 | 16100 | 0.1286        |
+| 1.3641 | 16200 | 0.1412        |
+| 1.3725 | 16300 | 0.1284        |
+| 1.3809 | 16400 | 0.1278        |
+| 1.3894 | 16500 | 0.1227        |
+| 1.3978 | 16600 | 0.1261        |
+| 1.4062 | 16700 | 0.1171        |
+| 1.4146 | 16800 | 0.1342        |
+| 1.4230 | 16900 | 0.1431        |
+| 1.4315 | 17000 | 0.1197        |
+| 1.4399 | 17100 | 0.13          |
+| 1.4483 | 17200 | 0.6417        |
+| 1.4567 | 17300 | 1.0493        |
+| 1.4651 | 17400 | 0.9416        |
+| 1.4736 | 17500 | 0.9058        |
+| 1.4820 | 17600 | 0.8439        |
+| 1.4904 | 17700 | 0.8125        |
+| 1.4988 | 17800 | 0.7498        |
+| 1.5072 | 17900 | 0.6975        |
+| 1.5157 | 18000 | 0.6554        |
+| 1.5241 | 18100 | 0.6563        |
+| 1.5325 | 18200 | 0.6458        |
+| 1.5409 | 18300 | 0.6086        |
+| 1.5493 | 18400 | 0.6098        |
+| 1.5578 | 18500 | 0.5725        |
+| 1.5662 | 18600 | 0.5794        |
+| 1.5746 | 18700 | 0.5525        |
+| 1.5830 | 18800 | 0.5345        |
+| 1.5914 | 18900 | 0.5765        |
+| 1.5999 | 19000 | 0.5097        |
+| 1.6083 | 19100 | 0.5681        |
+| 1.6167 | 19200 | 0.5473        |
+| 1.6251 | 19300 | 0.5206        |
+| 1.6335 | 19400 | 0.5175        |
+| 1.6420 | 19500 | 0.4796        |
+| 1.6504 | 19600 | 0.4866        |
+| 1.6588 | 19700 | 0.4706        |
+| 1.6672 | 19800 | 0.5092        |
+| 1.6756 | 19900 | 0.4799        |
+| 1.6841 | 20000 | 0.4548        |
+| 1.6925 | 20100 | 0.4639        |
+| 1.7009 | 20200 | 0.4297        |
+| 1.7093 | 20300 | 0.4382        |
+| 1.7178 | 20400 | 0.4428        |
+| 1.7262 | 20500 | 0.4308        |
+| 1.7346 | 20600 | 0.4038        |
+| 1.7430 | 20700 | 0.3842        |
+| 1.7514 | 20800 | 0.4227        |
+| 1.7599 | 20900 | 0.387         |
+| 1.7683 | 21000 | 0.3729        |
+| 1.7767 | 21100 | 0.3768        |
+| 1.7851 | 21200 | 0.3984        |
+| 1.7935 | 21300 | 0.4128        |
+| 1.8020 | 21400 | 0.3995        |
+| 1.8104 | 21500 | 0.416         |
+| 1.8188 | 21600 | 0.3488        |
+| 1.8272 | 21700 | 0.0116        |
+| 1.8356 | 21800 | 0.0066        |
+| 1.8441 | 21900 | 0.0051        |
+| 1.8525 | 22000 | 0.0064        |
+| 1.8609 | 22100 | 0.0038        |
+| 1.8693 | 22200 | 0.004         |
+| 1.8777 | 22300 | 0.003         |
+| 1.8862 | 22400 | 0.0023        |
+| 1.8946 | 22500 | 0.0026        |
+| 1.9030 | 22600 | 0.0021        |
+| 1.9114 | 22700 | 0.0021        |
+| 1.9198 | 22800 | 0.0317        |
+| 1.9283 | 22900 | 0.038         |
+| 1.9367 | 23000 | 0.0272        |
+| 1.9451 | 23100 | 0.0282        |
+| 1.9535 | 23200 | 0.0218        |
+| 1.9619 | 23300 | 0.0194        |
+| 1.9704 | 23400 | 0.0192        |
+| 1.9788 | 23500 | 0.0197        |
+| 1.9872 | 23600 | 0.0136        |
+| 1.9956 | 23700 | 0.0139        |
+| 2.0040 | 23800 | 0.1831        |
+| 2.0125 | 23900 | 0.1384        |
+| 2.0209 | 24000 | 0.1157        |
+| 2.0293 | 24100 | 0.104         |
+| 2.0377 | 24200 | 0.1123        |
+| 2.0461 | 24300 | 0.098         |
+| 2.0546 | 24400 | 0.0985        |
+| 2.0630 | 24500 | 0.099         |
+| 2.0714 | 24600 | 0.1033        |
+| 2.0798 | 24700 | 0.0897        |
+| 2.0882 | 24800 | 0.0939        |
+| 2.0967 | 24900 | 0.0846        |
+| 2.1051 | 25000 | 0.0913        |
+| 2.1135 | 25100 | 0.0892        |
+| 2.1219 | 25200 | 0.0914        |
+| 2.1303 | 25300 | 0.1019        |
+| 2.1388 | 25400 | 0.0858        |
+| 2.1472 | 25500 | 0.0886        |
+| 2.1556 | 25600 | 0.0875        |
+| 2.1640 | 25700 | 0.0828        |
+| 2.1724 | 25800 | 0.0769        |
+| 2.1809 | 25900 | 0.0768        |
+| 2.1893 | 26000 | 0.0769        |
+| 2.1977 | 26100 | 0.0744        |
+| 2.2061 | 26200 | 0.0675        |
+| 2.2146 | 26300 | 0.071         |
+| 2.2230 | 26400 | 0.0696        |
+| 2.2314 | 26500 | 0.0595        |
+| 2.2398 | 26600 | 0.0705        |
+| 2.2482 | 26700 | 0.0641        |
+| 2.2567 | 26800 | 0.0718        |
+| 2.2651 | 26900 | 0.0766        |
+| 2.2735 | 27000 | 0.0743        |
+| 2.2819 | 27100 | 0.0591        |
+| 2.2903 | 27200 | 0.055         |
+| 2.2988 | 27300 | 0.0642        |
+| 2.3072 | 27400 | 0.0624        |
+| 2.3156 | 27500 | 0.0568        |
+| 2.3240 | 27600 | 0.0582        |
+| 2.3324 | 27700 | 0.0477        |
+| 2.3409 | 27800 | 0.0513        |
+| 2.3493 | 27900 | 0.0486        |
+| 2.3577 | 28000 | 0.0485        |
+| 2.3661 | 28100 | 0.0571        |
+| 2.3745 | 28200 | 0.0575        |
+| 2.3830 | 28300 | 0.0503        |
+| 2.3914 | 28400 | 0.0482        |
+| 2.3998 | 28500 | 0.0578        |
+| 2.4082 | 28600 | 0.0496        |
+| 2.4166 | 28700 | 0.0545        |
+| 2.4251 | 28800 | 0.054         |
+| 2.4335 | 28900 | 0.0529        |
+| 2.4419 | 29000 | 0.0604        |
+| 2.4503 | 29100 | 0.462         |
+| 2.4587 | 29200 | 0.5095        |
+| 2.4672 | 29300 | 0.4603        |
+| 2.4756 | 29400 | 0.4584        |
+| 2.4840 | 29500 | 0.3933        |
+| 2.4924 | 29600 | 0.3533        |
+| 2.5008 | 29700 | 0.2919        |
+| 2.5093 | 29800 | 0.2907        |
+| 2.5177 | 29900 | 0.3201        |
+| 2.5261 | 30000 | 0.2896        |
+| 2.5345 | 30100 | 0.2852        |
+| 2.5429 | 30200 | 0.2182        |
+| 2.5514 | 30300 | 0.2423        |
+| 2.5598 | 30400 | 0.2496        |
+| 2.5682 | 30500 | 0.2303        |
+| 2.5766 | 30600 | 0.2065        |
+| 2.5850 | 30700 | 0.2031        |
+| 2.5935 | 30800 | 0.2384        |
+| 2.6019 | 30900 | 0.2441        |
+| 2.6103 | 31000 | 0.2914        |
+| 2.6187 | 31100 | 0.2695        |
+| 2.6271 | 31200 | 0.2366        |
+| 2.6356 | 31300 | 0.2007        |
+| 2.6440 | 31400 | 0.2088        |
+| 2.6524 | 31500 | 0.1975        |
+| 2.6608 | 31600 | 0.2162        |
+| 2.6692 | 31700 | 0.2201        |
+| 2.6777 | 31800 | 0.1907        |
+| 2.6861 | 31900 | 0.1675        |
+| 2.6945 | 32000 | 0.1866        |
+</details>
+### Framework Versions
+- Python: 3.10.12
+- Sentence Transformers: 3.3.1
+- Transformers: 4.51.3
+- PyTorch: 2.6.0+cu124
+- Accelerate: 1.2.1
+- Datasets: 3.5.0
+- Tokenizers: 0.21.1
+## Citation
+### BibTeX
+#### Sentence Transformers
+```bibtex
+@inproceedings{reimers-2019-sentence-bert,
+    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
+    author = "Reimers, Nils and Gurevych, Iryna",
+    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
+    month = "11",
+    year = "2019",
+    publisher = "Association for Computational Linguistics",
+    url = "https://arxiv.org/abs/1908.10084",
+}
+```
+#### MatryoshkaLoss
+```bibtex
+@misc{kusupati2024matryoshka,
+    title={Matryoshka Representation Learning},
+    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
+    year={2024},
+    eprint={2205.13147},
+    archivePrefix={arXiv},
+    primaryClass={cs.LG}
+}
+```
+#### MultipleNegativesRankingLoss
+```bibtex
+@misc{henderson2017efficient,
+    title={Efficient Natural Language Response Suggestion for Smart Reply},
+    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
+    year={2017},
+    eprint={1705.00652},
+    archivePrefix={arXiv},
+    primaryClass={cs.CL}
+}
+```
+<!--
+## Glossary
+*Clearly define terms in order to be accessible across audiences.*
+-->
+<!--
+## Model Card Authors
+*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
+-->
+<!--
+## Model Card Contact
+*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
+-->

added_tokens.json ADDED Viewed

	@@ -0,0 +1,28 @@

+{
+  "</think>": 151668,
+  "</tool_call>": 151658,
+  "</tool_response>": 151666,
+  "<think>": 151667,
+  "<tool_call>": 151657,
+  "<tool_response>": 151665,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

config.json ADDED Viewed

	@@ -0,0 +1,30 @@

+{
+  "architectures": [
+    "Qwen3Model"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 151643,
+  "eos_token_id": 151643,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "max_position_embeddings": 32768,
+  "max_window_layers": 28,
+  "model_type": "qwen3",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 8,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 1000000,
+  "sliding_window": null,
+  "tie_word_embeddings": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.51.3",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 151669
+}

config_sentence_transformers.json ADDED Viewed

	@@ -0,0 +1,10 @@

+{
+  "__version__": {
+    "sentence_transformers": "3.3.1",
+    "transformers": "4.51.3",
+    "pytorch": "2.6.0+cu124"
+  },
+  "prompts": {},
+  "default_prompt_name": null,
+  "similarity_fn_name": "cosine"
+}

merges.txt ADDED Viewed

The diff for this file is too large to render. See raw diff

model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b3732cd3499a3875e220a7179b5f25b452784d230b4df7dc3071189c099ea38b
+size 2383139480

modules.json ADDED Viewed

	@@ -0,0 +1,14 @@

+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  }
+]

optimizer.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:930a1cb040c853417c8b2ddffcf7c315a70f9f1a6f0956bdf7642e1ba7baa25a
+size 4766479998

rng_state_0.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:37da082521a1854fd9890918ac1e1cf29a15c5bd719056766a324764d00de274
+size 15408

rng_state_1.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a29b89cc0f5e10f02dddf0f01c67d3f1ac4bc6bbf614c36024bb2c607942af7b
+size 15472

rng_state_2.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bdd02267176cf89f336a32467643d9c67438cf9792e42d54729acf0c1cbec8b2
+size 15472

rng_state_3.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bdc51c626acedd92b06c746fe3672c0ce93a72abc1225229611acc757c810e38
+size 15472

rng_state_4.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8a547306496b237611bfa88571926633258d4868354d036a0c5b5fe72e82f4fc
+size 15472

rng_state_5.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a2773c2da36842cf52624593eaf2bfeef58662222f9fe3c334d1aed14f0a94d7
+size 15472

scheduler.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9ba0597be8406d3a44c603292e3feb3d74d1543531c3b0f91011c4d5ade1e249
+size 1064

sentence_bert_config.json ADDED Viewed

	@@ -0,0 +1,4 @@

+{
+  "max_seq_length": 32768,
+  "do_lower_case": false
+}

special_tokens_map.json ADDED Viewed

	@@ -0,0 +1,31 @@

+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:aed98cff73772352c260a48579f3b212cb2b06ce0553446799ebf0b5465b0673
+size 11422922

tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,240 @@

+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0].role == 'system' %}\n        {{- messages[0].content + '\\n\\n' }}\n    {%- endif %}\n    {{- \"# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0].role == 'system' %}\n        {{- '<|im_start|>system\\n' + messages[0].content + '<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}\n{%- for message in messages[::-1] %}\n    {%- set index = (messages|length - 1) - loop.index0 %}\n    {%- if ns.multi_step_tool and message.role == \"user\" and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}\n        {%- set ns.multi_step_tool = false %}\n        {%- set ns.last_query_index = index %}\n    {%- endif %}\n{%- endfor %}\n{%- for message in messages %}\n    {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) %}\n        {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {%- set content = message.content %}\n        {%- set reasoning_content = '' %}\n        {%- if message.reasoning_content is defined and message.reasoning_content is not none %}\n            {%- set reasoning_content = message.reasoning_content %}\n        {%- else %}\n            {%- if '</think>' in message.content %}\n                {%- set content = message.content.split('</think>')[-1].lstrip('\\n') %}\n                {%- set reasoning_content = message.content.split('</think>')[0].rstrip('\\n').split('<think>')[-1].lstrip('\\n') %}\n            {%- endif %}\n        {%- endif %}\n        {%- if loop.index0 > ns.last_query_index %}\n            {%- if loop.last or (not loop.last and reasoning_content) %}\n                {{- '<|im_start|>' + message.role + '\\n<think>\\n' + reasoning_content.strip('\\n') + '\\n</think>\\n\\n' + content.lstrip('\\n') }}\n            {%- else %}\n                {{- '<|im_start|>' + message.role + '\\n' + content }}\n            {%- endif %}\n        {%- else %}\n            {{- '<|im_start|>' + message.role + '\\n' + content }}\n        {%- endif %}\n        {%- if message.tool_calls %}\n            {%- for tool_call in message.tool_calls %}\n                {%- if (loop.first and content) or (not loop.first) %}\n                    {{- '\\n' }}\n                {%- endif %}\n                {%- if tool_call.function %}\n                    {%- set tool_call = tool_call.function %}\n                {%- endif %}\n                {{- '<tool_call>\\n{\"name\": \"' }}\n                {{- tool_call.name }}\n                {{- '\", \"arguments\": ' }}\n                {%- if tool_call.arguments is string %}\n                    {{- tool_call.arguments }}\n                {%- else %}\n                    {{- tool_call.arguments | tojson }}\n                {%- endif %}\n                {{- '}\\n</tool_call>' }}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if loop.first or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- message.content }}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n    {%- if enable_thinking is defined and enable_thinking is false %}\n        {{- '<think>\\n\\n</think>\\n\\n' }}\n    {%- endif %}\n{%- endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

trainer_state.json ADDED Viewed

	@@ -0,0 +1,2274 @@

+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 2.694509936005389,
+  "eval_steps": 500,
+  "global_step": 32000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.016840687100033683,
+      "grad_norm": 23.914043426513672,
+      "learning_rate": 2.777777777777778e-06,
+      "loss": 2.6825,
+      "step": 100
+    },
+    {
+      "epoch": 0.033681374200067365,
+      "grad_norm": 19.596776962280273,
+      "learning_rate": 5.583613916947251e-06,
+      "loss": 1.0725,
+      "step": 200
+    },
+    {
+      "epoch": 0.05052206130010104,
+      "grad_norm": 15.781028747558594,
+      "learning_rate": 8.389450056116724e-06,
+      "loss": 0.8306,
+      "step": 300
+    },
+    {
+      "epoch": 0.06736274840013473,
+      "grad_norm": 15.894832611083984,
+      "learning_rate": 1.1195286195286197e-05,
+      "loss": 0.7221,
+      "step": 400
+    },
+    {
+      "epoch": 0.08420343550016841,
+      "grad_norm": 14.838741302490234,
+      "learning_rate": 1.4001122334455668e-05,
+      "loss": 0.6888,
+      "step": 500
+    },
+    {
+      "epoch": 0.10104412260020208,
+      "grad_norm": 12.208050727844238,
+      "learning_rate": 1.6806958473625142e-05,
+      "loss": 0.6245,
+      "step": 600
+    },
+    {
+      "epoch": 0.11788480970023577,
+      "grad_norm": 11.734648704528809,
+      "learning_rate": 1.9612794612794612e-05,
+      "loss": 0.6413,
+      "step": 700
+    },
+    {
+      "epoch": 0.13472549680026946,
+      "grad_norm": 11.392744064331055,
+      "learning_rate": 2.2418630751964088e-05,
+      "loss": 0.6088,
+      "step": 800
+    },
+    {
+      "epoch": 0.15156618390030313,
+      "grad_norm": 11.298259735107422,
+      "learning_rate": 2.5224466891133558e-05,
+      "loss": 0.5892,
+      "step": 900
+    },
+    {
+      "epoch": 0.16840687100033683,
+      "grad_norm": 10.205204010009766,
+      "learning_rate": 2.803030303030303e-05,
+      "loss": 0.5764,
+      "step": 1000
+    },
+    {
+      "epoch": 0.1852475581003705,
+      "grad_norm": 9.070446968078613,
+      "learning_rate": 3.08361391694725e-05,
+      "loss": 0.5736,
+      "step": 1100
+    },
+    {
+      "epoch": 0.20208824520040417,
+      "grad_norm": 11.171720504760742,
+      "learning_rate": 3.364197530864198e-05,
+      "loss": 0.5493,
+      "step": 1200
+    },
+    {
+      "epoch": 0.21892893230043786,
+      "grad_norm": 8.007598876953125,
+      "learning_rate": 3.644781144781145e-05,
+      "loss": 0.5459,
+      "step": 1300
+    },
+    {
+      "epoch": 0.23576961940047153,
+      "grad_norm": 8.607964515686035,
+      "learning_rate": 3.9253647586980925e-05,
+      "loss": 0.543,
+      "step": 1400
+    },
+    {
+      "epoch": 0.2526103065005052,
+      "grad_norm": 8.324835777282715,
+      "learning_rate": 4.2059483726150395e-05,
+      "loss": 0.5274,
+      "step": 1500
+    },
+    {
+      "epoch": 0.2694509936005389,
+      "grad_norm": 7.262979507446289,
+      "learning_rate": 4.4865319865319864e-05,
+      "loss": 0.5424,
+      "step": 1600
+    },
+    {
+      "epoch": 0.2862916807005726,
+      "grad_norm": 7.70058012008667,
+      "learning_rate": 4.767115600448934e-05,
+      "loss": 0.5193,
+      "step": 1700
+    },
+    {
+      "epoch": 0.30313236780060626,
+      "grad_norm": 9.130758285522461,
+      "learning_rate": 4.9946981037924155e-05,
+      "loss": 0.5344,
+      "step": 1800
+    },
+    {
+      "epoch": 0.31997305490063993,
+      "grad_norm": 6.3734354972839355,
+      "learning_rate": 4.963510479041916e-05,
+      "loss": 0.5133,
+      "step": 1900
+    },
+    {
+      "epoch": 0.33681374200067365,
+      "grad_norm": 5.812978744506836,
+      "learning_rate": 4.9323228542914176e-05,
+      "loss": 0.4755,
+      "step": 2000
+    },
+    {
+      "epoch": 0.3536544291007073,
+      "grad_norm": 7.685583591461182,
+      "learning_rate": 4.901135229540918e-05,
+      "loss": 0.4735,
+      "step": 2100
+    },
+    {
+      "epoch": 0.370495116200741,
+      "grad_norm": 5.93942928314209,
+      "learning_rate": 4.86994760479042e-05,
+      "loss": 0.4876,
+      "step": 2200
+    },
+    {
+      "epoch": 0.38733580330077466,
+      "grad_norm": 4.325860500335693,
+      "learning_rate": 4.8387599800399204e-05,
+      "loss": 0.4483,
+      "step": 2300
+    },
+    {
+      "epoch": 0.40417649040080833,
+      "grad_norm": 5.2169928550720215,
+      "learning_rate": 4.807572355289421e-05,
+      "loss": 0.4459,
+      "step": 2400
+    },
+    {
+      "epoch": 0.42101717750084205,
+      "grad_norm": 4.603377342224121,
+      "learning_rate": 4.7763847305389225e-05,
+      "loss": 0.4133,
+      "step": 2500
+    },
+    {
+      "epoch": 0.21892893230043786,
+      "grad_norm": 4.210044860839844,
+      "learning_rate": 3.647207409486388e-05,
+      "loss": 0.2615,
+      "step": 2600
+    },
+    {
+      "epoch": 0.2273492758504547,
+      "grad_norm": 5.636209964752197,
+      "learning_rate": 3.787538591074937e-05,
+      "loss": 0.2518,
+      "step": 2700
+    },
+    {
+      "epoch": 0.23576961940047153,
+      "grad_norm": 4.7418389320373535,
+      "learning_rate": 3.927869772663486e-05,
+      "loss": 0.2785,
+      "step": 2800
+    },
+    {
+      "epoch": 0.2441899629504884,
+      "grad_norm": 6.645480632781982,
+      "learning_rate": 4.0682009542520346e-05,
+      "loss": 0.2541,
+      "step": 2900
+    },
+    {
+      "epoch": 0.2526103065005052,
+      "grad_norm": 4.639427185058594,
+      "learning_rate": 4.2085321358405836e-05,
+      "loss": 0.266,
+      "step": 3000
+    },
+    {
+      "epoch": 0.26103065005052206,
+      "grad_norm": 3.73759388923645,
+      "learning_rate": 4.348863317429133e-05,
+      "loss": 0.2671,
+      "step": 3100
+    },
+    {
+      "epoch": 0.2694509936005389,
+      "grad_norm": 7.948583126068115,
+      "learning_rate": 4.489194499017682e-05,
+      "loss": 0.2895,
+      "step": 3200
+    },
+    {
+      "epoch": 0.27787133715055573,
+      "grad_norm": 27.404747009277344,
+      "learning_rate": 4.629525680606231e-05,
+      "loss": 0.2658,
+      "step": 3300
+    },
+    {
+      "epoch": 0.2862916807005726,
+      "grad_norm": 3.8111979961395264,
+      "learning_rate": 4.7698568621947796e-05,
+      "loss": 0.2622,
+      "step": 3400
+    },
+    {
+      "epoch": 0.2947120242505894,
+      "grad_norm": 6.846595287322998,
+      "learning_rate": 4.9101880437833286e-05,
+      "loss": 0.2607,
+      "step": 3500
+    },
+    {
+      "epoch": 0.30313236780060626,
+      "grad_norm": 4.529717922210693,
+      "learning_rate": 4.994386402619679e-05,
+      "loss": 0.2883,
+      "step": 3600
+    },
+    {
+      "epoch": 0.3115527113506231,
+      "grad_norm": 6.5379838943481445,
+      "learning_rate": 4.9787930765632316e-05,
+      "loss": 0.2747,
+      "step": 3700
+    },
+    {
+      "epoch": 0.31997305490063993,
+      "grad_norm": 4.162957668304443,
+      "learning_rate": 4.963199750506784e-05,
+      "loss": 0.2525,
+      "step": 3800
+    },
+    {
+      "epoch": 0.3283933984506568,
+      "grad_norm": 5.669950485229492,
+      "learning_rate": 4.947606424450336e-05,
+      "loss": 0.2471,
+      "step": 3900
+    },
+    {
+      "epoch": 0.33681374200067365,
+      "grad_norm": 6.552733898162842,
+      "learning_rate": 4.932013098393888e-05,
+      "loss": 0.2564,
+      "step": 4000
+    },
+    {
+      "epoch": 0.34523408555069046,
+      "grad_norm": 5.492654800415039,
+      "learning_rate": 4.91641977233744e-05,
+      "loss": 0.2541,
+      "step": 4100
+    },
+    {
+      "epoch": 0.3536544291007073,
+      "grad_norm": 6.248386859893799,
+      "learning_rate": 4.900826446280992e-05,
+      "loss": 0.2421,
+      "step": 4200
+    },
+    {
+      "epoch": 0.36207477265072413,
+      "grad_norm": 7.152673721313477,
+      "learning_rate": 4.8852331202245445e-05,
+      "loss": 0.2559,
+      "step": 4300
+    },
+    {
+      "epoch": 0.370495116200741,
+      "grad_norm": 7.939176082611084,
+      "learning_rate": 4.869639794168096e-05,
+      "loss": 0.2562,
+      "step": 4400
+    },
+    {
+      "epoch": 0.37891545975075785,
+      "grad_norm": 6.023073673248291,
+      "learning_rate": 4.854046468111648e-05,
+      "loss": 0.2548,
+      "step": 4500
+    },
+    {
+      "epoch": 0.38733580330077466,
+      "grad_norm": 6.619602680206299,
+      "learning_rate": 4.8384531420552e-05,
+      "loss": 0.2504,
+      "step": 4600
+    },
+    {
+      "epoch": 0.3957561468507915,
+      "grad_norm": 4.903266906738281,
+      "learning_rate": 4.8228598159987524e-05,
+      "loss": 0.2585,
+      "step": 4700
+    },
+    {
+      "epoch": 0.40417649040080833,
+      "grad_norm": 4.036454200744629,
+      "learning_rate": 4.8072664899423046e-05,
+      "loss": 0.2368,
+      "step": 4800
+    },
+    {
+      "epoch": 0.4125968339508252,
+      "grad_norm": 3.432321310043335,
+      "learning_rate": 4.791673163885857e-05,
+      "loss": 0.2298,
+      "step": 4900
+    },
+    {
+      "epoch": 0.42101717750084205,
+      "grad_norm": 4.507967472076416,
+      "learning_rate": 4.776079837829409e-05,
+      "loss": 0.2277,
+      "step": 5000
+    },
+    {
+      "epoch": 0.42943752105085886,
+      "grad_norm": 4.154341697692871,
+      "learning_rate": 4.760486511772961e-05,
+      "loss": 0.2809,
+      "step": 5100
+    },
+    {
+      "epoch": 0.4378578646008757,
+      "grad_norm": 4.294288158416748,
+      "learning_rate": 4.744893185716513e-05,
+      "loss": 0.2945,
+      "step": 5200
+    },
+    {
+      "epoch": 0.44627820815089253,
+      "grad_norm": 13.444978713989258,
+      "learning_rate": 4.729299859660066e-05,
+      "loss": 0.6972,
+      "step": 5300
+    },
+    {
+      "epoch": 0.4546985517009094,
+      "grad_norm": 13.236030578613281,
+      "learning_rate": 4.713706533603618e-05,
+      "loss": 1.6206,
+      "step": 5400
+    },
+    {
+      "epoch": 0.46311889525092625,
+      "grad_norm": 9.054878234863281,
+      "learning_rate": 4.6981132075471704e-05,
+      "loss": 1.48,
+      "step": 5500
+    },
+    {
+      "epoch": 0.47153923880094306,
+      "grad_norm": 9.235391616821289,
+      "learning_rate": 4.6825198814907225e-05,
+      "loss": 1.3816,
+      "step": 5600
+    },
+    {
+      "epoch": 0.4799595823509599,
+      "grad_norm": 9.566539764404297,
+      "learning_rate": 4.666926555434275e-05,
+      "loss": 1.3296,
+      "step": 5700
+    },
+    {
+      "epoch": 0.4883799259009768,
+      "grad_norm": 10.539976119995117,
+      "learning_rate": 4.651333229377827e-05,
+      "loss": 1.2737,
+      "step": 5800
+    },
+    {
+      "epoch": 0.4968002694509936,
+      "grad_norm": 10.859559059143066,
+      "learning_rate": 4.635739903321379e-05,
+      "loss": 1.188,
+      "step": 5900
+    },
+    {
+      "epoch": 0.5052206130010104,
+      "grad_norm": 9.153464317321777,
+      "learning_rate": 4.620146577264931e-05,
+      "loss": 1.1852,
+      "step": 6000
+    },
+    {
+      "epoch": 0.5136409565510273,
+      "grad_norm": 11.076135635375977,
+      "learning_rate": 4.604553251208483e-05,
+      "loss": 1.1271,
+      "step": 6100
+    },
+    {
+      "epoch": 0.5220613001010441,
+      "grad_norm": 11.092999458312988,
+      "learning_rate": 4.588959925152035e-05,
+      "loss": 1.1654,
+      "step": 6200
+    },
+    {
+      "epoch": 0.530481643651061,
+      "grad_norm": 9.680800437927246,
+      "learning_rate": 4.573366599095587e-05,
+      "loss": 1.1028,
+      "step": 6300
+    },
+    {
+      "epoch": 0.5389019872010778,
+      "grad_norm": 7.664565086364746,
+      "learning_rate": 4.557773273039139e-05,
+      "loss": 1.0713,
+      "step": 6400
+    },
+    {
+      "epoch": 0.5473223307510946,
+      "grad_norm": 8.164676666259766,
+      "learning_rate": 4.542179946982691e-05,
+      "loss": 1.0654,
+      "step": 6500
+    },
+    {
+      "epoch": 0.5557426743011115,
+      "grad_norm": 7.794740676879883,
+      "learning_rate": 4.5265866209262434e-05,
+      "loss": 1.0779,
+      "step": 6600
+    },
+    {
+      "epoch": 0.5641630178511283,
+      "grad_norm": 9.8069486618042,
+      "learning_rate": 4.5109932948697956e-05,
+      "loss": 1.0448,
+      "step": 6700
+    },
+    {
+      "epoch": 0.5725833614011452,
+      "grad_norm": 8.124543190002441,
+      "learning_rate": 4.495399968813348e-05,
+      "loss": 1.022,
+      "step": 6800
+    },
+    {
+      "epoch": 0.581003704951162,
+      "grad_norm": 9.155059814453125,
+      "learning_rate": 4.4798066427569005e-05,
+      "loss": 1.0037,
+      "step": 6900
+    },
+    {
+      "epoch": 0.5894240485011788,
+      "grad_norm": 8.151762962341309,
+      "learning_rate": 4.464213316700453e-05,
+      "loss": 1.026,
+      "step": 7000
+    },
+    {
+      "epoch": 0.5978443920511957,
+      "grad_norm": 9.232538223266602,
+      "learning_rate": 4.448619990644005e-05,
+      "loss": 0.9852,
+      "step": 7100
+    },
+    {
+      "epoch": 0.6062647356012125,
+      "grad_norm": 10.167255401611328,
+      "learning_rate": 4.433026664587557e-05,
+      "loss": 0.9583,
+      "step": 7200
+    },
+    {
+      "epoch": 0.6146850791512294,
+      "grad_norm": 8.223834991455078,
+      "learning_rate": 4.417433338531109e-05,
+      "loss": 0.974,
+      "step": 7300
+    },
+    {
+      "epoch": 0.6231054227012462,
+      "grad_norm": 7.129863262176514,
+      "learning_rate": 4.401840012474661e-05,
+      "loss": 0.9357,
+      "step": 7400
+    },
+    {
+      "epoch": 0.631525766251263,
+      "grad_norm": 9.384492874145508,
+      "learning_rate": 4.3862466864182135e-05,
+      "loss": 0.9426,
+      "step": 7500
+    },
+    {
+      "epoch": 0.6399461098012799,
+      "grad_norm": 10.51504135131836,
+      "learning_rate": 4.3706533603617656e-05,
+      "loss": 0.9001,
+      "step": 7600
+    },
+    {
+      "epoch": 0.6483664533512967,
+      "grad_norm": 9.259176254272461,
+      "learning_rate": 4.355060034305318e-05,
+      "loss": 0.8875,
+      "step": 7700
+    },
+    {
+      "epoch": 0.6567867969013136,
+      "grad_norm": 10.058541297912598,
+      "learning_rate": 4.33946670824887e-05,
+      "loss": 0.8552,
+      "step": 7800
+    },
+    {
+      "epoch": 0.6652071404513304,
+      "grad_norm": 8.267607688903809,
+      "learning_rate": 4.323873382192422e-05,
+      "loss": 0.9011,
+      "step": 7900
+    },
+    {
+      "epoch": 0.6736274840013473,
+      "grad_norm": 9.002063751220703,
+      "learning_rate": 4.3082800561359736e-05,
+      "loss": 0.8692,
+      "step": 8000
+    },
+    {
+      "epoch": 0.6820478275513641,
+      "grad_norm": 11.397157669067383,
+      "learning_rate": 4.292686730079526e-05,
+      "loss": 0.8637,
+      "step": 8100
+    },
+    {
+      "epoch": 0.6904681711013809,
+      "grad_norm": 10.458459854125977,
+      "learning_rate": 4.277093404023078e-05,
+      "loss": 0.8292,
+      "step": 8200
+    },
+    {
+      "epoch": 0.6988885146513978,
+      "grad_norm": 8.840250968933105,
+      "learning_rate": 4.26150007796663e-05,
+      "loss": 0.8226,
+      "step": 8300
+    },
+    {
+      "epoch": 0.7073088582014146,
+      "grad_norm": 7.398085117340088,
+      "learning_rate": 4.245906751910182e-05,
+      "loss": 0.8066,
+      "step": 8400
+    },
+    {
+      "epoch": 0.7157292017514315,
+      "grad_norm": 8.141140937805176,
+      "learning_rate": 4.2303134258537344e-05,
+      "loss": 0.8495,
+      "step": 8500
+    },
+    {
+      "epoch": 0.7241495453014483,
+      "grad_norm": 8.317313194274902,
+      "learning_rate": 4.214720099797287e-05,
+      "loss": 0.794,
+      "step": 8600
+    },
+    {
+      "epoch": 0.7325698888514651,
+      "grad_norm": 8.556983947753906,
+      "learning_rate": 4.1991267737408393e-05,
+      "loss": 0.7676,
+      "step": 8700
+    },
+    {
+      "epoch": 0.740990232401482,
+      "grad_norm": 7.237626075744629,
+      "learning_rate": 4.1835334476843915e-05,
+      "loss": 0.7671,
+      "step": 8800
+    },
+    {
+      "epoch": 0.7494105759514988,
+      "grad_norm": 11.043583869934082,
+      "learning_rate": 4.167940121627944e-05,
+      "loss": 0.769,
+      "step": 8900
+    },
+    {
+      "epoch": 0.7578309195015157,
+      "grad_norm": 6.836021423339844,
+      "learning_rate": 4.152346795571496e-05,
+      "loss": 0.7808,
+      "step": 9000
+    },
+    {
+      "epoch": 0.7662512630515325,
+      "grad_norm": 7.930835723876953,
+      "learning_rate": 4.136753469515048e-05,
+      "loss": 0.7439,
+      "step": 9100
+    },
+    {
+      "epoch": 0.7746716066015493,
+      "grad_norm": 7.065630912780762,
+      "learning_rate": 4.1211601434586e-05,
+      "loss": 0.7394,
+      "step": 9200
+    },
+    {
+      "epoch": 0.7830919501515662,
+      "grad_norm": 12.510068893432617,
+      "learning_rate": 4.105566817402152e-05,
+      "loss": 0.7662,
+      "step": 9300
+    },
+    {
+      "epoch": 0.791512293701583,
+      "grad_norm": 5.61587381362915,
+      "learning_rate": 4.0899734913457044e-05,
+      "loss": 0.7472,
+      "step": 9400
+    },
+    {
+      "epoch": 0.7999326372515999,
+      "grad_norm": 21.683469772338867,
+      "learning_rate": 4.0743801652892566e-05,
+      "loss": 0.7592,
+      "step": 9500
+    },
+    {
+      "epoch": 0.8083529808016167,
+      "grad_norm": 9.07845401763916,
+      "learning_rate": 4.058786839232809e-05,
+      "loss": 0.7599,
+      "step": 9600
+    },
+    {
+      "epoch": 0.8167733243516335,
+      "grad_norm": 7.406639575958252,
+      "learning_rate": 4.043193513176361e-05,
+      "loss": 0.7397,
+      "step": 9700
+    },
+    {
+      "epoch": 0.8251936679016504,
+      "grad_norm": 0.6552740931510925,
+      "learning_rate": 4.0276001871199124e-05,
+      "loss": 0.1707,
+      "step": 9800
+    },
+    {
+      "epoch": 0.8336140114516672,
+      "grad_norm": 0.47074249386787415,
+      "learning_rate": 4.0120068610634645e-05,
+      "loss": 0.036,
+      "step": 9900
+    },
+    {
+      "epoch": 0.8420343550016841,
+      "grad_norm": 1.5905348062515259,
+      "learning_rate": 3.996413535007017e-05,
+      "loss": 0.0272,
+      "step": 10000
+    },
+    {
+      "epoch": 0.8504546985517009,
+      "grad_norm": 0.7267942428588867,
+      "learning_rate": 3.980820208950569e-05,
+      "loss": 0.0189,
+      "step": 10100
+    },
+    {
+      "epoch": 0.8588750421017177,
+      "grad_norm": 0.1869667023420334,
+      "learning_rate": 3.965226882894122e-05,
+      "loss": 0.0139,
+      "step": 10200
+    },
+    {
+      "epoch": 0.8672953856517346,
+      "grad_norm": 0.8771575689315796,
+      "learning_rate": 3.949633556837674e-05,
+      "loss": 0.0155,
+      "step": 10300
+    },
+    {
+      "epoch": 0.8757157292017514,
+      "grad_norm": 0.1621396690607071,
+      "learning_rate": 3.934040230781226e-05,
+      "loss": 0.0145,
+      "step": 10400
+    },
+    {
+      "epoch": 0.8841360727517683,
+      "grad_norm": 0.26454639434814453,
+      "learning_rate": 3.918446904724778e-05,
+      "loss": 0.0124,
+      "step": 10500
+    },
+    {
+      "epoch": 0.8925564163017851,
+      "grad_norm": 0.5660475492477417,
+      "learning_rate": 3.90285357866833e-05,
+      "loss": 0.0098,
+      "step": 10600
+    },
+    {
+      "epoch": 0.9009767598518019,
+      "grad_norm": 0.12594054639339447,
+      "learning_rate": 3.8872602526118825e-05,
+      "loss": 0.0091,
+      "step": 10700
+    },
+    {
+      "epoch": 0.9093971034018188,
+      "grad_norm": 0.10871898382902145,
+      "learning_rate": 3.8716669265554346e-05,
+      "loss": 0.01,
+      "step": 10800
+    },
+    {
+      "epoch": 0.9178174469518356,
+      "grad_norm": 5.482397556304932,
+      "learning_rate": 3.856073600498987e-05,
+      "loss": 0.1028,
+      "step": 10900
+    },
+    {
+      "epoch": 0.9262377905018525,
+      "grad_norm": 5.7357659339904785,
+      "learning_rate": 3.840480274442539e-05,
+      "loss": 0.1921,
+      "step": 11000
+    },
+    {
+      "epoch": 0.9346581340518694,
+      "grad_norm": 1.8957055807113647,
+      "learning_rate": 3.824886948386091e-05,
+      "loss": 0.1521,
+      "step": 11100
+    },
+    {
+      "epoch": 0.9430784776018861,
+      "grad_norm": 7.204999923706055,
+      "learning_rate": 3.809293622329643e-05,
+      "loss": 0.1336,
+      "step": 11200
+    },
+    {
+      "epoch": 0.951498821151903,
+      "grad_norm": 11.601072311401367,
+      "learning_rate": 3.7937002962731954e-05,
+      "loss": 0.1238,
+      "step": 11300
+    },
+    {
+      "epoch": 0.9599191647019198,
+      "grad_norm": 1.993941068649292,
+      "learning_rate": 3.7781069702167476e-05,
+      "loss": 0.1065,
+      "step": 11400
+    },
+    {
+      "epoch": 0.9683395082519367,
+      "grad_norm": 3.342390298843384,
+      "learning_rate": 3.7625136441603e-05,
+      "loss": 0.0955,
+      "step": 11500
+    },
+    {
+      "epoch": 0.9767598518019536,
+      "grad_norm": 7.570684432983398,
+      "learning_rate": 3.746920318103851e-05,
+      "loss": 0.0944,
+      "step": 11600
+    },
+    {
+      "epoch": 0.9851801953519703,
+      "grad_norm": 3.947699546813965,
+      "learning_rate": 3.7313269920474034e-05,
+      "loss": 0.0849,
+      "step": 11700
+    },
+    {
+      "epoch": 0.9936005389019872,
+      "grad_norm": 5.432759761810303,
+      "learning_rate": 3.715733665990956e-05,
+      "loss": 0.0712,
+      "step": 11800
+    },
+    {
+      "epoch": 1.002020882452004,
+      "grad_norm": 15.626507759094238,
+      "learning_rate": 3.7001403399345083e-05,
+      "loss": 0.3673,
+      "step": 11900
+    },
+    {
+      "epoch": 1.0104412260020208,
+      "grad_norm": 6.4510111808776855,
+      "learning_rate": 3.6845470138780605e-05,
+      "loss": 0.4508,
+      "step": 12000
+    },
+    {
+      "epoch": 1.0188615695520378,
+      "grad_norm": 6.159328460693359,
+      "learning_rate": 3.6689536878216127e-05,
+      "loss": 0.3693,
+      "step": 12100
+    },
+    {
+      "epoch": 1.0272819131020545,
+      "grad_norm": 4.550877571105957,
+      "learning_rate": 3.653360361765165e-05,
+      "loss": 0.3297,
+      "step": 12200
+    },
+    {
+      "epoch": 1.0357022566520715,
+      "grad_norm": 3.226945638656616,
+      "learning_rate": 3.637767035708717e-05,
+      "loss": 0.3452,
+      "step": 12300
+    },
+    {
+      "epoch": 1.0441226002020882,
+      "grad_norm": 3.6009678840637207,
+      "learning_rate": 3.622173709652269e-05,
+      "loss": 0.3062,
+      "step": 12400
+    },
+    {
+      "epoch": 1.052542943752105,
+      "grad_norm": 2.7150795459747314,
+      "learning_rate": 3.606580383595821e-05,
+      "loss": 0.2824,
+      "step": 12500
+    },
+    {
+      "epoch": 1.060963287302122,
+      "grad_norm": 4.014440059661865,
+      "learning_rate": 3.5909870575393734e-05,
+      "loss": 0.2909,
+      "step": 12600
+    },
+    {
+      "epoch": 1.0693836308521387,
+      "grad_norm": 2.9422249794006348,
+      "learning_rate": 3.5753937314829256e-05,
+      "loss": 0.2688,
+      "step": 12700
+    },
+    {
+      "epoch": 1.0778039744021557,
+      "grad_norm": 3.798882484436035,
+      "learning_rate": 3.559800405426478e-05,
+      "loss": 0.261,
+      "step": 12800
+    },
+    {
+      "epoch": 1.0862243179521724,
+      "grad_norm": 2.825617551803589,
+      "learning_rate": 3.54420707937003e-05,
+      "loss": 0.2616,
+      "step": 12900
+    },
+    {
+      "epoch": 1.0946446615021892,
+      "grad_norm": 4.132945537567139,
+      "learning_rate": 3.528613753313582e-05,
+      "loss": 0.2496,
+      "step": 13000
+    },
+    {
+      "epoch": 1.1030650050522062,
+      "grad_norm": 5.848606109619141,
+      "learning_rate": 3.513020427257134e-05,
+      "loss": 0.2415,
+      "step": 13100
+    },
+    {
+      "epoch": 1.111485348602223,
+      "grad_norm": 3.019437789916992,
+      "learning_rate": 3.4974271012006864e-05,
+      "loss": 0.2469,
+      "step": 13200
+    },
+    {
+      "epoch": 1.11990569215224,
+      "grad_norm": 5.2769455909729,
+      "learning_rate": 3.4818337751442385e-05,
+      "loss": 0.2444,
+      "step": 13300
+    },
+    {
+      "epoch": 1.1283260357022566,
+      "grad_norm": 5.284823894500732,
+      "learning_rate": 3.46624044908779e-05,
+      "loss": 0.2489,
+      "step": 13400
+    },
+    {
+      "epoch": 1.1367463792522734,
+      "grad_norm": 1.0584665536880493,
+      "learning_rate": 3.450647123031343e-05,
+      "loss": 0.2338,
+      "step": 13500
+    },
+    {
+      "epoch": 1.1451667228022904,
+      "grad_norm": 37.629119873046875,
+      "learning_rate": 3.435053796974895e-05,
+      "loss": 0.2185,
+      "step": 13600
+    },
+    {
+      "epoch": 1.1535870663523071,
+      "grad_norm": 16.267772674560547,
+      "learning_rate": 3.419460470918447e-05,
+      "loss": 0.2288,
+      "step": 13700
+    },
+    {
+      "epoch": 1.162007409902324,
+      "grad_norm": 4.195704936981201,
+      "learning_rate": 3.403867144861999e-05,
+      "loss": 0.2032,
+      "step": 13800
+    },
+    {
+      "epoch": 1.1704277534523408,
+      "grad_norm": 3.9956605434417725,
+      "learning_rate": 3.3882738188055515e-05,
+      "loss": 0.2005,
+      "step": 13900
+    },
+    {
+      "epoch": 1.1788480970023576,
+      "grad_norm": 2.037095069885254,
+      "learning_rate": 3.3726804927491036e-05,
+      "loss": 0.2053,
+      "step": 14000
+    },
+    {
+      "epoch": 1.1872684405523746,
+      "grad_norm": 10.881503105163574,
+      "learning_rate": 3.357087166692656e-05,
+      "loss": 0.2042,
+      "step": 14100
+    },
+    {
+      "epoch": 1.1956887841023913,
+      "grad_norm": 5.309749126434326,
+      "learning_rate": 3.341493840636208e-05,
+      "loss": 0.1877,
+      "step": 14200
+    },
+    {
+      "epoch": 1.2041091276524083,
+      "grad_norm": 3.2984306812286377,
+      "learning_rate": 3.32590051457976e-05,
+      "loss": 0.1711,
+      "step": 14300
+    },
+    {
+      "epoch": 1.212529471202425,
+      "grad_norm": 3.6049394607543945,
+      "learning_rate": 3.310307188523312e-05,
+      "loss": 0.1973,
+      "step": 14400
+    },
+    {
+      "epoch": 1.220949814752442,
+      "grad_norm": 6.268308162689209,
+      "learning_rate": 3.2947138624668644e-05,
+      "loss": 0.1624,
+      "step": 14500
+    },
+    {
+      "epoch": 1.2293701583024588,
+      "grad_norm": 2.217378616333008,
+      "learning_rate": 3.2791205364104166e-05,
+      "loss": 0.1646,
+      "step": 14600
+    },
+    {
+      "epoch": 1.2377905018524755,
+      "grad_norm": 4.168875694274902,
+      "learning_rate": 3.263527210353969e-05,
+      "loss": 0.1702,
+      "step": 14700
+    },
+    {
+      "epoch": 1.2462108454024925,
+      "grad_norm": 2.362311601638794,
+      "learning_rate": 3.247933884297521e-05,
+      "loss": 0.166,
+      "step": 14800
+    },
+    {
+      "epoch": 1.2546311889525092,
+      "grad_norm": 3.168797016143799,
+      "learning_rate": 3.232340558241073e-05,
+      "loss": 0.1689,
+      "step": 14900
+    },
+    {
+      "epoch": 1.263051532502526,
+      "grad_norm": 3.6625378131866455,
+      "learning_rate": 3.216747232184625e-05,
+      "loss": 0.1747,
+      "step": 15000
+    },
+    {
+      "epoch": 1.271471876052543,
+      "grad_norm": 5.1908793449401855,
+      "learning_rate": 3.201153906128177e-05,
+      "loss": 0.1755,
+      "step": 15100
+    },
+    {
+      "epoch": 1.2798922196025597,
+      "grad_norm": 4.5366387367248535,
+      "learning_rate": 3.1855605800717295e-05,
+      "loss": 0.1607,
+      "step": 15200
+    },
+    {
+      "epoch": 1.2883125631525767,
+      "grad_norm": 3.7079453468322754,
+      "learning_rate": 3.1699672540152816e-05,
+      "loss": 0.1462,
+      "step": 15300
+    },
+    {
+      "epoch": 1.2967329067025934,
+      "grad_norm": 3.3219869136810303,
+      "learning_rate": 3.154373927958834e-05,
+      "loss": 0.1475,
+      "step": 15400
+    },
+    {
+      "epoch": 1.3051532502526104,
+      "grad_norm": 2.976282835006714,
+      "learning_rate": 3.138780601902386e-05,
+      "loss": 0.1587,
+      "step": 15500
+    },
+    {
+      "epoch": 1.3135735938026272,
+      "grad_norm": 2.854203224182129,
+      "learning_rate": 3.123187275845938e-05,
+      "loss": 0.1506,
+      "step": 15600
+    },
+    {
+      "epoch": 1.321993937352644,
+      "grad_norm": 2.5714399814605713,
+      "learning_rate": 3.10759394978949e-05,
+      "loss": 0.1467,
+      "step": 15700
+    },
+    {
+      "epoch": 1.330414280902661,
+      "grad_norm": 1.7303158044815063,
+      "learning_rate": 3.0920006237330424e-05,
+      "loss": 0.1277,
+      "step": 15800
+    },
+    {
+      "epoch": 1.3388346244526776,
+      "grad_norm": 2.199556350708008,
+      "learning_rate": 3.0764072976765946e-05,
+      "loss": 0.1305,
+      "step": 15900
+    },
+    {
+      "epoch": 1.3472549680026944,
+      "grad_norm": 1.3136367797851562,
+      "learning_rate": 3.060813971620147e-05,
+      "loss": 0.1292,
+      "step": 16000
+    },
+    {
+      "epoch": 1.3556753115527114,
+      "grad_norm": 1.4371681213378906,
+      "learning_rate": 3.045220645563699e-05,
+      "loss": 0.1286,
+      "step": 16100
+    },
+    {
+      "epoch": 1.3640956551027281,
+      "grad_norm": 2.634970188140869,
+      "learning_rate": 3.029627319507251e-05,
+      "loss": 0.1412,
+      "step": 16200
+    },
+    {
+      "epoch": 1.372515998652745,
+      "grad_norm": 3.108686923980713,
+      "learning_rate": 3.0140339934508032e-05,
+      "loss": 0.1284,
+      "step": 16300
+    },
+    {
+      "epoch": 1.3809363422027618,
+      "grad_norm": 4.35302734375,
+      "learning_rate": 2.9984406673943554e-05,
+      "loss": 0.1278,
+      "step": 16400
+    },
+    {
+      "epoch": 1.3893566857527788,
+      "grad_norm": 2.4115021228790283,
+      "learning_rate": 2.9828473413379075e-05,
+      "loss": 0.1227,
+      "step": 16500
+    },
+    {
+      "epoch": 1.3977770293027956,
+      "grad_norm": 0.8723605275154114,
+      "learning_rate": 2.96725401528146e-05,
+      "loss": 0.1261,
+      "step": 16600
+    },
+    {
+      "epoch": 1.4061973728528123,
+      "grad_norm": 3.208569288253784,
+      "learning_rate": 2.951660689225012e-05,
+      "loss": 0.1171,
+      "step": 16700
+    },
+    {
+      "epoch": 1.4146177164028293,
+      "grad_norm": 3.3627731800079346,
+      "learning_rate": 2.9360673631685643e-05,
+      "loss": 0.1342,
+      "step": 16800
+    },
+    {
+      "epoch": 1.423038059952846,
+      "grad_norm": 2.9769978523254395,
+      "learning_rate": 2.9204740371121165e-05,
+      "loss": 0.1431,
+      "step": 16900
+    },
+    {
+      "epoch": 1.4314584035028628,
+      "grad_norm": 2.390516996383667,
+      "learning_rate": 2.904880711055668e-05,
+      "loss": 0.1197,
+      "step": 17000
+    },
+    {
+      "epoch": 1.4398787470528798,
+      "grad_norm": 2.3254621028900146,
+      "learning_rate": 2.88928738499922e-05,
+      "loss": 0.13,
+      "step": 17100
+    },
+    {
+      "epoch": 1.4482990906028965,
+      "grad_norm": 9.375923156738281,
+      "learning_rate": 2.8736940589427726e-05,
+      "loss": 0.6417,
+      "step": 17200
+    },
+    {
+      "epoch": 1.4567194341529135,
+      "grad_norm": 24.961429595947266,
+      "learning_rate": 2.8581007328863248e-05,
+      "loss": 1.0493,
+      "step": 17300
+    },
+    {
+      "epoch": 1.4651397777029302,
+      "grad_norm": 22.577869415283203,
+      "learning_rate": 2.842507406829877e-05,
+      "loss": 0.9416,
+      "step": 17400
+    },
+    {
+      "epoch": 1.4735601212529472,
+      "grad_norm": 18.104610443115234,
+      "learning_rate": 2.826914080773429e-05,
+      "loss": 0.9058,
+      "step": 17500
+    },
+    {
+      "epoch": 1.481980464802964,
+      "grad_norm": 11.02493667602539,
+      "learning_rate": 2.8113207547169812e-05,
+      "loss": 0.8439,
+      "step": 17600
+    },
+    {
+      "epoch": 1.4904008083529807,
+      "grad_norm": 7.483858585357666,
+      "learning_rate": 2.7957274286605334e-05,
+      "loss": 0.8125,
+      "step": 17700
+    },
+    {
+      "epoch": 1.4988211519029977,
+      "grad_norm": 7.024225234985352,
+      "learning_rate": 2.7801341026040855e-05,
+      "loss": 0.7498,
+      "step": 17800
+    },
+    {
+      "epoch": 1.5072414954530144,
+      "grad_norm": 39.53349304199219,
+      "learning_rate": 2.7645407765476377e-05,
+      "loss": 0.6975,
+      "step": 17900
+    },
+    {
+      "epoch": 1.5156618390030312,
+      "grad_norm": 9.007433891296387,
+      "learning_rate": 2.74894745049119e-05,
+      "loss": 0.6554,
+      "step": 18000
+    },
+    {
+      "epoch": 1.5240821825530482,
+      "grad_norm": 7.764451503753662,
+      "learning_rate": 2.733354124434742e-05,
+      "loss": 0.6563,
+      "step": 18100
+    },
+    {
+      "epoch": 1.5325025261030651,
+      "grad_norm": 8.649127960205078,
+      "learning_rate": 2.7177607983782945e-05,
+      "loss": 0.6458,
+      "step": 18200
+    },
+    {
+      "epoch": 1.540922869653082,
+      "grad_norm": 14.572016716003418,
+      "learning_rate": 2.7021674723218467e-05,
+      "loss": 0.6086,
+      "step": 18300
+    },
+    {
+      "epoch": 1.5493432132030986,
+      "grad_norm": 8.887728691101074,
+      "learning_rate": 2.6865741462653988e-05,
+      "loss": 0.6098,
+      "step": 18400
+    },
+    {
+      "epoch": 1.5577635567531156,
+      "grad_norm": 9.536375999450684,
+      "learning_rate": 2.670980820208951e-05,
+      "loss": 0.5725,
+      "step": 18500
+    },
+    {
+      "epoch": 1.5661839003031324,
+      "grad_norm": 69.17536926269531,
+      "learning_rate": 2.655387494152503e-05,
+      "loss": 0.5794,
+      "step": 18600
+    },
+    {
+      "epoch": 1.5746042438531491,
+      "grad_norm": 26.702865600585938,
+      "learning_rate": 2.6397941680960553e-05,
+      "loss": 0.5525,
+      "step": 18700
+    },
+    {
+      "epoch": 1.583024587403166,
+      "grad_norm": 6.380317211151123,
+      "learning_rate": 2.6242008420396068e-05,
+      "loss": 0.5345,
+      "step": 18800
+    },
+    {
+      "epoch": 1.5914449309531828,
+      "grad_norm": 8.336255073547363,
+      "learning_rate": 2.6086075159831593e-05,
+      "loss": 0.5765,
+      "step": 18900
+    },
+    {
+      "epoch": 1.5998652745031996,
+      "grad_norm": 14.956998825073242,
+      "learning_rate": 2.5930141899267114e-05,
+      "loss": 0.5097,
+      "step": 19000
+    },
+    {
+      "epoch": 1.6082856180532166,
+      "grad_norm": 16.499773025512695,
+      "learning_rate": 2.5774208638702636e-05,
+      "loss": 0.5681,
+      "step": 19100
+    },
+    {
+      "epoch": 1.6167059616032335,
+      "grad_norm": 99.75704956054688,
+      "learning_rate": 2.5618275378138157e-05,
+      "loss": 0.5473,
+      "step": 19200
+    },
+    {
+      "epoch": 1.6251263051532503,
+      "grad_norm": 8.305459976196289,
+      "learning_rate": 2.546234211757368e-05,
+      "loss": 0.5206,
+      "step": 19300
+    },
+    {
+      "epoch": 1.633546648703267,
+      "grad_norm": 9.534538269042969,
+      "learning_rate": 2.53064088570092e-05,
+      "loss": 0.5175,
+      "step": 19400
+    },
+    {
+      "epoch": 1.641966992253284,
+      "grad_norm": 10.031558990478516,
+      "learning_rate": 2.5150475596444722e-05,
+      "loss": 0.4796,
+      "step": 19500
+    },
+    {
+      "epoch": 1.6503873358033008,
+      "grad_norm": 10.884944915771484,
+      "learning_rate": 2.4994542335880244e-05,
+      "loss": 0.4866,
+      "step": 19600
+    },
+    {
+      "epoch": 1.6588076793533175,
+      "grad_norm": 38.82561111450195,
+      "learning_rate": 2.4838609075315765e-05,
+      "loss": 0.4706,
+      "step": 19700
+    },
+    {
+      "epoch": 1.6672280229033345,
+      "grad_norm": 30.905010223388672,
+      "learning_rate": 2.4682675814751287e-05,
+      "loss": 0.5092,
+      "step": 19800
+    },
+    {
+      "epoch": 1.6756483664533512,
+      "grad_norm": 6.405722141265869,
+      "learning_rate": 2.452674255418681e-05,
+      "loss": 0.4799,
+      "step": 19900
+    },
+    {
+      "epoch": 1.684068710003368,
+      "grad_norm": 19.763652801513672,
+      "learning_rate": 2.4370809293622333e-05,
+      "loss": 0.4548,
+      "step": 20000
+    },
+    {
+      "epoch": 1.692489053553385,
+      "grad_norm": 10.142723083496094,
+      "learning_rate": 2.421487603305785e-05,
+      "loss": 0.4639,
+      "step": 20100
+    },
+    {
+      "epoch": 1.700909397103402,
+      "grad_norm": 11.770618438720703,
+      "learning_rate": 2.4058942772493373e-05,
+      "loss": 0.4297,
+      "step": 20200
+    },
+    {
+      "epoch": 1.7093297406534187,
+      "grad_norm": 69.44645690917969,
+      "learning_rate": 2.3903009511928894e-05,
+      "loss": 0.4382,
+      "step": 20300
+    },
+    {
+      "epoch": 1.7177500842034354,
+      "grad_norm": 6.592321872711182,
+      "learning_rate": 2.3747076251364416e-05,
+      "loss": 0.4428,
+      "step": 20400
+    },
+    {
+      "epoch": 1.7261704277534524,
+      "grad_norm": 12.039219856262207,
+      "learning_rate": 2.3591142990799938e-05,
+      "loss": 0.4308,
+      "step": 20500
+    },
+    {
+      "epoch": 1.7345907713034692,
+      "grad_norm": 7.2263994216918945,
+      "learning_rate": 2.343520973023546e-05,
+      "loss": 0.4038,
+      "step": 20600
+    },
+    {
+      "epoch": 1.743011114853486,
+      "grad_norm": 6.700140476226807,
+      "learning_rate": 2.3279276469670984e-05,
+      "loss": 0.3842,
+      "step": 20700
+    },
+    {
+      "epoch": 1.751431458403503,
+      "grad_norm": 5.211968898773193,
+      "learning_rate": 2.3123343209106506e-05,
+      "loss": 0.4227,
+      "step": 20800
+    },
+    {
+      "epoch": 1.7598518019535199,
+      "grad_norm": 4.104666233062744,
+      "learning_rate": 2.2967409948542027e-05,
+      "loss": 0.387,
+      "step": 20900
+    },
+    {
+      "epoch": 1.7682721455035364,
+      "grad_norm": 8.997115135192871,
+      "learning_rate": 2.2811476687977545e-05,
+      "loss": 0.3729,
+      "step": 21000
+    },
+    {
+      "epoch": 1.7766924890535534,
+      "grad_norm": 10.214786529541016,
+      "learning_rate": 2.2655543427413067e-05,
+      "loss": 0.3768,
+      "step": 21100
+    },
+    {
+      "epoch": 1.7851128326035703,
+      "grad_norm": 8.062252044677734,
+      "learning_rate": 2.249961016684859e-05,
+      "loss": 0.3984,
+      "step": 21200
+    },
+    {
+      "epoch": 1.793533176153587,
+      "grad_norm": 8.471816062927246,
+      "learning_rate": 2.234367690628411e-05,
+      "loss": 0.4128,
+      "step": 21300
+    },
+    {
+      "epoch": 1.8019535197036038,
+      "grad_norm": 6.631267547607422,
+      "learning_rate": 2.218774364571963e-05,
+      "loss": 0.3995,
+      "step": 21400
+    },
+    {
+      "epoch": 1.8103738632536208,
+      "grad_norm": 7.958058834075928,
+      "learning_rate": 2.2031810385155157e-05,
+      "loss": 0.416,
+      "step": 21500
+    },
+    {
+      "epoch": 1.8187942068036376,
+      "grad_norm": 0.48709893226623535,
+      "learning_rate": 2.1875877124590678e-05,
+      "loss": 0.3488,
+      "step": 21600
+    },
+    {
+      "epoch": 1.8272145503536543,
+      "grad_norm": 0.13080443441867828,
+      "learning_rate": 2.17199438640262e-05,
+      "loss": 0.0116,
+      "step": 21700
+    },
+    {
+      "epoch": 1.8356348939036713,
+      "grad_norm": 0.12103134393692017,
+      "learning_rate": 2.156401060346172e-05,
+      "loss": 0.0066,
+      "step": 21800
+    },
+    {
+      "epoch": 1.8440552374536883,
+      "grad_norm": 0.128848135471344,
+      "learning_rate": 2.140807734289724e-05,
+      "loss": 0.0051,
+      "step": 21900
+    },
+    {
+      "epoch": 1.8524755810037048,
+      "grad_norm": 1.0999774932861328,
+      "learning_rate": 2.125214408233276e-05,
+      "loss": 0.0064,
+      "step": 22000
+    },
+    {
+      "epoch": 1.8608959245537218,
+      "grad_norm": 0.19511470198631287,
+      "learning_rate": 2.1096210821768282e-05,
+      "loss": 0.0038,
+      "step": 22100
+    },
+    {
+      "epoch": 1.8693162681037387,
+      "grad_norm": 0.2998058795928955,
+      "learning_rate": 2.0940277561203804e-05,
+      "loss": 0.004,
+      "step": 22200
+    },
+    {
+      "epoch": 1.8777366116537555,
+      "grad_norm": 0.3210146725177765,
+      "learning_rate": 2.078434430063933e-05,
+      "loss": 0.003,
+      "step": 22300
+    },
+    {
+      "epoch": 1.8861569552037722,
+      "grad_norm": 0.08681979775428772,
+      "learning_rate": 2.062841104007485e-05,
+      "loss": 0.0023,
+      "step": 22400
+    },
+    {
+      "epoch": 1.8945772987537892,
+      "grad_norm": 0.21799153089523315,
+      "learning_rate": 2.0472477779510372e-05,
+      "loss": 0.0026,
+      "step": 22500
+    },
+    {
+      "epoch": 1.902997642303806,
+      "grad_norm": 0.062381282448768616,
+      "learning_rate": 2.0316544518945894e-05,
+      "loss": 0.0021,
+      "step": 22600
+    },
+    {
+      "epoch": 1.9114179858538227,
+      "grad_norm": 0.13877958059310913,
+      "learning_rate": 2.0160611258381415e-05,
+      "loss": 0.0021,
+      "step": 22700
+    },
+    {
+      "epoch": 1.9198383294038397,
+      "grad_norm": 0.6393685936927795,
+      "learning_rate": 2.0004677997816933e-05,
+      "loss": 0.0317,
+      "step": 22800
+    },
+    {
+      "epoch": 1.9282586729538567,
+      "grad_norm": 1.2400928735733032,
+      "learning_rate": 1.9848744737252455e-05,
+      "loss": 0.038,
+      "step": 22900
+    },
+    {
+      "epoch": 1.9366790165038732,
+      "grad_norm": 0.9584667086601257,
+      "learning_rate": 1.9692811476687977e-05,
+      "loss": 0.0272,
+      "step": 23000
+    },
+    {
+      "epoch": 1.9450993600538902,
+      "grad_norm": 1.0835716724395752,
+      "learning_rate": 1.9536878216123498e-05,
+      "loss": 0.0282,
+      "step": 23100
+    },
+    {
+      "epoch": 1.9535197036039071,
+      "grad_norm": 2.3751420974731445,
+      "learning_rate": 1.9380944955559023e-05,
+      "loss": 0.0218,
+      "step": 23200
+    },
+    {
+      "epoch": 1.961940047153924,
+      "grad_norm": 0.33170026540756226,
+      "learning_rate": 1.9225011694994545e-05,
+      "loss": 0.0194,
+      "step": 23300
+    },
+    {
+      "epoch": 1.9703603907039406,
+      "grad_norm": 1.3425530195236206,
+      "learning_rate": 1.9069078434430066e-05,
+      "loss": 0.0192,
+      "step": 23400
+    },
+    {
+      "epoch": 1.9787807342539576,
+      "grad_norm": 0.2928735911846161,
+      "learning_rate": 1.8913145173865588e-05,
+      "loss": 0.0197,
+      "step": 23500
+    },
+    {
+      "epoch": 1.9872010778039744,
+      "grad_norm": 0.1353292614221573,
+      "learning_rate": 1.875721191330111e-05,
+      "loss": 0.0136,
+      "step": 23600
+    },
+    {
+      "epoch": 1.9956214213539911,
+      "grad_norm": 0.6597471833229065,
+      "learning_rate": 1.8601278652736627e-05,
+      "loss": 0.0139,
+      "step": 23700
+    },
+    {
+      "epoch": 2.004041764904008,
+      "grad_norm": 2.835427761077881,
+      "learning_rate": 1.844534539217215e-05,
+      "loss": 0.1831,
+      "step": 23800
+    },
+    {
+      "epoch": 2.012462108454025,
+      "grad_norm": 2.29437255859375,
+      "learning_rate": 1.828941213160767e-05,
+      "loss": 0.1384,
+      "step": 23900
+    },
+    {
+      "epoch": 2.0208824520040416,
+      "grad_norm": 2.420936346054077,
+      "learning_rate": 1.8133478871043196e-05,
+      "loss": 0.1157,
+      "step": 24000
+    },
+    {
+      "epoch": 2.0293027955540586,
+      "grad_norm": 2.158114194869995,
+      "learning_rate": 1.7977545610478717e-05,
+      "loss": 0.104,
+      "step": 24100
+    },
+    {
+      "epoch": 2.0377231391040755,
+      "grad_norm": 2.9640002250671387,
+      "learning_rate": 1.782161234991424e-05,
+      "loss": 0.1123,
+      "step": 24200
+    },
+    {
+      "epoch": 2.046143482654092,
+      "grad_norm": 3.13875675201416,
+      "learning_rate": 1.766567908934976e-05,
+      "loss": 0.098,
+      "step": 24300
+    },
+    {
+      "epoch": 2.054563826204109,
+      "grad_norm": 1.8173339366912842,
+      "learning_rate": 1.7509745828785282e-05,
+      "loss": 0.0985,
+      "step": 24400
+    },
+    {
+      "epoch": 2.062984169754126,
+      "grad_norm": 1.9323346614837646,
+      "learning_rate": 1.7353812568220803e-05,
+      "loss": 0.099,
+      "step": 24500
+    },
+    {
+      "epoch": 2.071404513304143,
+      "grad_norm": 1.5606794357299805,
+      "learning_rate": 1.719787930765632e-05,
+      "loss": 0.1033,
+      "step": 24600
+    },
+    {
+      "epoch": 2.0798248568541595,
+      "grad_norm": 2.589740753173828,
+      "learning_rate": 1.7041946047091843e-05,
+      "loss": 0.0897,
+      "step": 24700
+    },
+    {
+      "epoch": 2.0882452004041765,
+      "grad_norm": 0.7438304424285889,
+      "learning_rate": 1.6886012786527368e-05,
+      "loss": 0.0939,
+      "step": 24800
+    },
+    {
+      "epoch": 2.0966655439541935,
+      "grad_norm": 0.7309479713439941,
+      "learning_rate": 1.673007952596289e-05,
+      "loss": 0.0846,
+      "step": 24900
+    },
+    {
+      "epoch": 2.10508588750421,
+      "grad_norm": 0.8864309787750244,
+      "learning_rate": 1.657414626539841e-05,
+      "loss": 0.0913,
+      "step": 25000
+    },
+    {
+      "epoch": 2.113506231054227,
+      "grad_norm": 3.734173059463501,
+      "learning_rate": 1.6418213004833933e-05,
+      "loss": 0.0892,
+      "step": 25100
+    },
+    {
+      "epoch": 2.121926574604244,
+      "grad_norm": 1.031955599784851,
+      "learning_rate": 1.6262279744269454e-05,
+      "loss": 0.0914,
+      "step": 25200
+    },
+    {
+      "epoch": 2.1303469181542605,
+      "grad_norm": 1.262018084526062,
+      "learning_rate": 1.6106346483704976e-05,
+      "loss": 0.1019,
+      "step": 25300
+    },
+    {
+      "epoch": 2.1387672617042774,
+      "grad_norm": 0.6608179807662964,
+      "learning_rate": 1.5950413223140497e-05,
+      "loss": 0.0858,
+      "step": 25400
+    },
+    {
+      "epoch": 2.1471876052542944,
+      "grad_norm": 0.5380622744560242,
+      "learning_rate": 1.5794479962576016e-05,
+      "loss": 0.0886,
+      "step": 25500
+    },
+    {
+      "epoch": 2.1556079488043114,
+      "grad_norm": 2.167166233062744,
+      "learning_rate": 1.563854670201154e-05,
+      "loss": 0.0875,
+      "step": 25600
+    },
+    {
+      "epoch": 2.164028292354328,
+      "grad_norm": 3.2814202308654785,
+      "learning_rate": 1.5482613441447062e-05,
+      "loss": 0.0828,
+      "step": 25700
+    },
+    {
+      "epoch": 2.172448635904345,
+      "grad_norm": 1.1189013719558716,
+      "learning_rate": 1.5326680180882584e-05,
+      "loss": 0.0769,
+      "step": 25800
+    },
+    {
+      "epoch": 2.180868979454362,
+      "grad_norm": 0.8286951184272766,
+      "learning_rate": 1.5170746920318105e-05,
+      "loss": 0.0768,
+      "step": 25900
+    },
+    {
+      "epoch": 2.1892893230043784,
+      "grad_norm": 0.5198948383331299,
+      "learning_rate": 1.5014813659753627e-05,
+      "loss": 0.0769,
+      "step": 26000
+    },
+    {
+      "epoch": 2.1977096665543954,
+      "grad_norm": 1.3561586141586304,
+      "learning_rate": 1.4858880399189148e-05,
+      "loss": 0.0744,
+      "step": 26100
+    },
+    {
+      "epoch": 2.2061300101044123,
+      "grad_norm": 2.7382802963256836,
+      "learning_rate": 1.470294713862467e-05,
+      "loss": 0.0675,
+      "step": 26200
+    },
+    {
+      "epoch": 2.2145503536544293,
+      "grad_norm": 1.4717612266540527,
+      "learning_rate": 1.4547013878060191e-05,
+      "loss": 0.071,
+      "step": 26300
+    },
+    {
+      "epoch": 2.222970697204446,
+      "grad_norm": 2.3162336349487305,
+      "learning_rate": 1.4391080617495711e-05,
+      "loss": 0.0696,
+      "step": 26400
+    },
+    {
+      "epoch": 2.231391040754463,
+      "grad_norm": 0.35251349210739136,
+      "learning_rate": 1.4235147356931233e-05,
+      "loss": 0.0595,
+      "step": 26500
+    },
+    {
+      "epoch": 2.23981138430448,
+      "grad_norm": 2.207399368286133,
+      "learning_rate": 1.4079214096366754e-05,
+      "loss": 0.0705,
+      "step": 26600
+    },
+    {
+      "epoch": 2.2482317278544963,
+      "grad_norm": 2.2342419624328613,
+      "learning_rate": 1.3923280835802278e-05,
+      "loss": 0.0641,
+      "step": 26700
+    },
+    {
+      "epoch": 2.2566520714045133,
+      "grad_norm": 1.604312539100647,
+      "learning_rate": 1.37673475752378e-05,
+      "loss": 0.0718,
+      "step": 26800
+    },
+    {
+      "epoch": 2.2650724149545303,
+      "grad_norm": 2.674197196960449,
+      "learning_rate": 1.361141431467332e-05,
+      "loss": 0.0766,
+      "step": 26900
+    },
+    {
+      "epoch": 2.273492758504547,
+      "grad_norm": 2.6131813526153564,
+      "learning_rate": 1.3455481054108842e-05,
+      "loss": 0.0743,
+      "step": 27000
+    },
+    {
+      "epoch": 2.2819131020545638,
+      "grad_norm": 0.4795171618461609,
+      "learning_rate": 1.3299547793544364e-05,
+      "loss": 0.0591,
+      "step": 27100
+    },
+    {
+      "epoch": 2.2903334456045807,
+      "grad_norm": 0.6082819700241089,
+      "learning_rate": 1.3143614532979887e-05,
+      "loss": 0.055,
+      "step": 27200
+    },
+    {
+      "epoch": 2.2987537891545973,
+      "grad_norm": 1.9809715747833252,
+      "learning_rate": 1.2987681272415405e-05,
+      "loss": 0.0642,
+      "step": 27300
+    },
+    {
+      "epoch": 2.3071741327046142,
+      "grad_norm": 2.095233678817749,
+      "learning_rate": 1.2831748011850927e-05,
+      "loss": 0.0624,
+      "step": 27400
+    },
+    {
+      "epoch": 2.315594476254631,
+      "grad_norm": 0.3463571071624756,
+      "learning_rate": 1.267581475128645e-05,
+      "loss": 0.0568,
+      "step": 27500
+    },
+    {
+      "epoch": 2.324014819804648,
+      "grad_norm": 0.37240880727767944,
+      "learning_rate": 1.2519881490721972e-05,
+      "loss": 0.0582,
+      "step": 27600
+    },
+    {
+      "epoch": 2.3324351633546647,
+      "grad_norm": 0.3742882311344147,
+      "learning_rate": 1.2363948230157493e-05,
+      "loss": 0.0477,
+      "step": 27700
+    },
+    {
+      "epoch": 2.3408555069046817,
+      "grad_norm": 0.3263934850692749,
+      "learning_rate": 1.2208014969593015e-05,
+      "loss": 0.0513,
+      "step": 27800
+    },
+    {
+      "epoch": 2.3492758504546987,
+      "grad_norm": 0.7098066806793213,
+      "learning_rate": 1.2052081709028536e-05,
+      "loss": 0.0486,
+      "step": 27900
+    },
+    {
+      "epoch": 2.357696194004715,
+      "grad_norm": 0.8197891712188721,
+      "learning_rate": 1.1896148448464058e-05,
+      "loss": 0.0485,
+      "step": 28000
+    },
+    {
+      "epoch": 2.366116537554732,
+      "grad_norm": 0.7206839919090271,
+      "learning_rate": 1.174021518789958e-05,
+      "loss": 0.0571,
+      "step": 28100
+    },
+    {
+      "epoch": 2.374536881104749,
+      "grad_norm": 0.4567437767982483,
+      "learning_rate": 1.1584281927335101e-05,
+      "loss": 0.0575,
+      "step": 28200
+    },
+    {
+      "epoch": 2.382957224654766,
+      "grad_norm": 1.4102520942687988,
+      "learning_rate": 1.1428348666770623e-05,
+      "loss": 0.0503,
+      "step": 28300
+    },
+    {
+      "epoch": 2.3913775682047826,
+      "grad_norm": 0.22976179420948029,
+      "learning_rate": 1.1272415406206144e-05,
+      "loss": 0.0482,
+      "step": 28400
+    },
+    {
+      "epoch": 2.3997979117547996,
+      "grad_norm": 0.8997607231140137,
+      "learning_rate": 1.1116482145641666e-05,
+      "loss": 0.0578,
+      "step": 28500
+    },
+    {
+      "epoch": 2.4082182553048166,
+      "grad_norm": 0.5438327789306641,
+      "learning_rate": 1.0960548885077187e-05,
+      "loss": 0.0496,
+      "step": 28600
+    },
+    {
+      "epoch": 2.416638598854833,
+      "grad_norm": 0.38543766736984253,
+      "learning_rate": 1.0804615624512709e-05,
+      "loss": 0.0545,
+      "step": 28700
+    },
+    {
+      "epoch": 2.42505894240485,
+      "grad_norm": 0.45761021971702576,
+      "learning_rate": 1.064868236394823e-05,
+      "loss": 0.054,
+      "step": 28800
+    },
+    {
+      "epoch": 2.433479285954867,
+      "grad_norm": 1.072931170463562,
+      "learning_rate": 1.0492749103383752e-05,
+      "loss": 0.0529,
+      "step": 28900
+    },
+    {
+      "epoch": 2.441899629504884,
+      "grad_norm": 0.8028556704521179,
+      "learning_rate": 1.0336815842819274e-05,
+      "loss": 0.0604,
+      "step": 29000
+    },
+    {
+      "epoch": 2.4503199730549006,
+      "grad_norm": 5.610771656036377,
+      "learning_rate": 1.0180882582254795e-05,
+      "loss": 0.462,
+      "step": 29100
+    },
+    {
+      "epoch": 2.4587403166049175,
+      "grad_norm": 8.661577224731445,
+      "learning_rate": 1.0024949321690318e-05,
+      "loss": 0.5095,
+      "step": 29200
+    },
+    {
+      "epoch": 2.4671606601549345,
+      "grad_norm": 5.638516902923584,
+      "learning_rate": 9.869016061125838e-06,
+      "loss": 0.4603,
+      "step": 29300
+    },
+    {
+      "epoch": 2.475581003704951,
+      "grad_norm": 6.656158447265625,
+      "learning_rate": 9.71308280056136e-06,
+      "loss": 0.4584,
+      "step": 29400
+    },
+    {
+      "epoch": 2.484001347254968,
+      "grad_norm": 5.3289923667907715,
+      "learning_rate": 9.557149539996881e-06,
+      "loss": 0.3933,
+      "step": 29500
+    },
+    {
+      "epoch": 2.492421690804985,
+      "grad_norm": 4.8232831954956055,
+      "learning_rate": 9.401216279432405e-06,
+      "loss": 0.3533,
+      "step": 29600
+    },
+    {
+      "epoch": 2.5008420343550015,
+      "grad_norm": 5.679427623748779,
+      "learning_rate": 9.245283018867924e-06,
+      "loss": 0.2919,
+      "step": 29700
+    },
+    {
+      "epoch": 2.5092623779050185,
+      "grad_norm": 3.5916316509246826,
+      "learning_rate": 9.089349758303446e-06,
+      "loss": 0.2907,
+      "step": 29800
+    },
+    {
+      "epoch": 2.5176827214550355,
+      "grad_norm": 4.971811294555664,
+      "learning_rate": 8.933416497738968e-06,
+      "loss": 0.3201,
+      "step": 29900
+    },
+    {
+      "epoch": 2.526103065005052,
+      "grad_norm": 7.323381423950195,
+      "learning_rate": 8.77748323717449e-06,
+      "loss": 0.2896,
+      "step": 30000
+    },
+    {
+      "epoch": 2.534523408555069,
+      "grad_norm": 3.5097551345825195,
+      "learning_rate": 8.621549976610012e-06,
+      "loss": 0.2852,
+      "step": 30100
+    },
+    {
+      "epoch": 2.542943752105086,
+      "grad_norm": 3.7720704078674316,
+      "learning_rate": 8.465616716045532e-06,
+      "loss": 0.2182,
+      "step": 30200
+    },
+    {
+      "epoch": 2.551364095655103,
+      "grad_norm": 6.767735481262207,
+      "learning_rate": 8.309683455481054e-06,
+      "loss": 0.2423,
+      "step": 30300
+    },
+    {
+      "epoch": 2.5597844392051194,
+      "grad_norm": 3.3460731506347656,
+      "learning_rate": 8.153750194916577e-06,
+      "loss": 0.2496,
+      "step": 30400
+    },
+    {
+      "epoch": 2.5682047827551364,
+      "grad_norm": 17.407913208007812,
+      "learning_rate": 7.997816934352099e-06,
+      "loss": 0.2303,
+      "step": 30500
+    },
+    {
+      "epoch": 2.5766251263051534,
+      "grad_norm": 3.996459722518921,
+      "learning_rate": 7.841883673787618e-06,
+      "loss": 0.2065,
+      "step": 30600
+    },
+    {
+      "epoch": 2.58504546985517,
+      "grad_norm": 2.772590398788452,
+      "learning_rate": 7.68595041322314e-06,
+      "loss": 0.2031,
+      "step": 30700
+    },
+    {
+      "epoch": 2.593465813405187,
+      "grad_norm": 4.046528339385986,
+      "learning_rate": 7.5300171526586624e-06,
+      "loss": 0.2384,
+      "step": 30800
+    },
+    {
+      "epoch": 2.601886156955204,
+      "grad_norm": 2.2654547691345215,
+      "learning_rate": 7.374083892094184e-06,
+      "loss": 0.2441,
+      "step": 30900
+    },
+    {
+      "epoch": 2.610306500505221,
+      "grad_norm": 4.873164176940918,
+      "learning_rate": 7.218150631529706e-06,
+      "loss": 0.2914,
+      "step": 31000
+    },
+    {
+      "epoch": 2.6187268440552374,
+      "grad_norm": 2.461027145385742,
+      "learning_rate": 7.062217370965227e-06,
+      "loss": 0.2695,
+      "step": 31100
+    },
+    {
+      "epoch": 2.6271471876052543,
+      "grad_norm": 7.631922245025635,
+      "learning_rate": 6.906284110400749e-06,
+      "loss": 0.2366,
+      "step": 31200
+    },
+    {
+      "epoch": 2.635567531155271,
+      "grad_norm": 2.1125121116638184,
+      "learning_rate": 6.75035084983627e-06,
+      "loss": 0.2007,
+      "step": 31300
+    },
+    {
+      "epoch": 2.643987874705288,
+      "grad_norm": 3.188915967941284,
+      "learning_rate": 6.594417589271793e-06,
+      "loss": 0.2088,
+      "step": 31400
+    },
+    {
+      "epoch": 2.652408218255305,
+      "grad_norm": 3.0127580165863037,
+      "learning_rate": 6.438484328707313e-06,
+      "loss": 0.1975,
+      "step": 31500
+    },
+    {
+      "epoch": 2.660828561805322,
+      "grad_norm": 1.8730789422988892,
+      "learning_rate": 6.282551068142835e-06,
+      "loss": 0.2162,
+      "step": 31600
+    },
+    {
+      "epoch": 2.6692489053553388,
+      "grad_norm": 6.462364673614502,
+      "learning_rate": 6.1266178075783565e-06,
+      "loss": 0.2201,
+      "step": 31700
+    },
+    {
+      "epoch": 2.6776692489053553,
+      "grad_norm": 5.668213844299316,
+      "learning_rate": 5.970684547013879e-06,
+      "loss": 0.1907,
+      "step": 31800
+    },
+    {
+      "epoch": 2.6860895924553723,
+      "grad_norm": 2.8429136276245117,
+      "learning_rate": 5.8147512864494e-06,
+      "loss": 0.1675,
+      "step": 31900
+    },
+    {
+      "epoch": 2.694509936005389,
+      "grad_norm": 3.140132188796997,
+      "learning_rate": 5.658818025884922e-06,
+      "loss": 0.1866,
+      "step": 32000
+    }
+  ],
+  "logging_steps": 100,
+  "max_steps": 35628,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0.0,
+  "train_batch_size": 64,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f5f5e79079adecb3d2fe96a0f8a6b0c56742f44e7d29a6c4fcf98bcc4967a119
+size 5688

vocab.json ADDED Viewed

The diff for this file is too large to render. See raw diff