--- license: mit base_model: HuggingFaceH4/zephyr-7b-beta tags: - generated_from_trainer model-index: - name: non-qa-sft-zephyr-7b-beta-v1 results: [] --- # non-qa-sft-zephyr-7b-beta-v1 This model is a fine-tuned version of [HuggingFaceH4/zephyr-7b-beta](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta) on the None dataset. It achieves the following results on the evaluation set: - Loss: 0.5002 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 3e-05 - train_batch_size: 2 - eval_batch_size: 2 - seed: 42 - distributed_type: multi-GPU - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: cosine - lr_scheduler_warmup_ratio: 0.05 - num_epochs: 2 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:-----:|:----:|:---------------:| | 1.805 | 0.02 | 50 | 1.3674 | | 1.352 | 0.03 | 100 | 1.1858 | | 1.0793 | 0.05 | 150 | 0.8566 | | 0.828 | 0.07 | 200 | 0.7728 | | 0.7571 | 0.09 | 250 | 0.7385 | | 0.5984 | 0.1 | 300 | 0.7140 | | 0.7967 | 0.12 | 350 | 0.7132 | | 0.6366 | 0.14 | 400 | 0.7042 | | 0.6902 | 0.15 | 450 | 0.6834 | | 0.7354 | 0.17 | 500 | 0.6794 | | 0.7073 | 0.19 | 550 | 0.6877 | | 0.7558 | 0.21 | 600 | 0.6408 | | 0.6622 | 0.22 | 650 | 0.6255 | | 0.7071 | 0.24 | 700 | 0.6267 | | 0.6281 | 0.26 | 750 | 0.6259 | | 0.6623 | 0.27 | 800 | 0.6202 | | 0.6423 | 0.29 | 850 | 0.6144 | | 0.6506 | 0.31 | 900 | 0.6196 | | 0.6141 | 0.33 | 950 | 0.6135 | | 0.5386 | 0.34 | 1000 | 0.6097 | | 0.661 | 0.36 | 1050 | 0.6026 | | 0.628 | 0.38 | 1100 | 0.5952 | | 0.6502 | 0.39 | 1150 | 0.5898 | | 0.5601 | 0.41 | 1200 | 0.5858 | | 0.6326 | 0.43 | 1250 | 0.5749 | | 0.6376 | 0.44 | 1300 | 0.5655 | | 0.6145 | 0.46 | 1350 | 0.5619 | | 0.5654 | 0.48 | 1400 | 0.5553 | | 0.5373 | 0.5 | 1450 | 0.5542 | | 0.5696 | 0.51 | 1500 | 0.5436 | | 0.5909 | 0.53 | 1550 | 0.5480 | | 0.5193 | 0.55 | 1600 | 0.5511 | | 0.6237 | 0.56 | 1650 | 0.5496 | | 0.6133 | 0.58 | 1700 | 0.5484 | | 0.5312 | 0.6 | 1750 | 0.5489 | | 0.5502 | 0.62 | 1800 | 0.5475 | | 0.5906 | 0.63 | 1850 | 0.5363 | | 0.5848 | 0.65 | 1900 | 0.5400 | | 0.6144 | 0.67 | 1950 | 0.5446 | | 0.5733 | 0.68 | 2000 | 0.5437 | | 0.5974 | 0.7 | 2050 | 0.5422 | | 0.5307 | 0.72 | 2100 | 0.5451 | | 0.5956 | 0.74 | 2150 | 0.5411 | | 0.4696 | 0.75 | 2200 | 0.5418 | | 0.5438 | 0.77 | 2250 | 0.5392 | | 0.5265 | 0.79 | 2300 | 0.5309 | | 0.5366 | 0.8 | 2350 | 0.5366 | | 0.5636 | 0.82 | 2400 | 0.5327 | | 0.5827 | 0.84 | 2450 | 0.5313 | | 0.6114 | 0.86 | 2500 | 0.5311 | | 0.5059 | 0.87 | 2550 | 0.5290 | | 0.49 | 0.89 | 2600 | 0.5324 | | 0.5073 | 0.91 | 2650 | 0.5306 | | 0.4695 | 0.92 | 2700 | 0.5292 | | 0.5365 | 0.94 | 2750 | 0.5237 | | 0.5738 | 0.96 | 2800 | 0.5252 | | 0.596 | 0.98 | 2850 | 0.5189 | | 0.5542 | 0.99 | 2900 | 0.5165 | | 0.4898 | 1.01 | 2950 | 0.5182 | | 0.5882 | 1.03 | 3000 | 0.5180 | | 0.5004 | 1.04 | 3050 | 0.5175 | | 0.4946 | 1.06 | 3100 | 0.5153 | | 0.5238 | 1.08 | 3150 | 0.5161 | | 0.5233 | 1.09 | 3200 | 0.5134 | | 0.5175 | 1.11 | 3250 | 0.5130 | | 0.5392 | 1.13 | 3300 | 0.5123 | | 0.5032 | 1.15 | 3350 | 0.5090 | | 0.5184 | 1.16 | 3400 | 0.5122 | | 0.5088 | 1.18 | 3450 | 0.5136 | | 0.47 | 1.2 | 3500 | 0.5115 | | 0.5357 | 1.21 | 3550 | 0.5098 | | 0.4996 | 1.23 | 3600 | 0.5093 | | 0.5524 | 1.25 | 3650 | 0.5104 | | 0.575 | 1.27 | 3700 | 0.5110 | | 0.4943 | 1.28 | 3750 | 0.5091 | | 0.4933 | 1.3 | 3800 | 0.5073 | | 0.4441 | 1.32 | 3850 | 0.5086 | | 0.5472 | 1.33 | 3900 | 0.5065 | | 0.5409 | 1.35 | 3950 | 0.5080 | | 0.495 | 1.37 | 4000 | 0.5081 | | 0.4447 | 1.39 | 4050 | 0.5089 | | 0.4664 | 1.4 | 4100 | 0.5096 | | 0.4979 | 1.42 | 4150 | 0.5052 | | 0.5215 | 1.44 | 4200 | 0.5042 | | 0.4436 | 1.45 | 4250 | 0.5045 | | 0.5381 | 1.47 | 4300 | 0.5044 | | 0.4965 | 1.49 | 4350 | 0.5032 | | 0.4185 | 1.51 | 4400 | 0.5041 | | 0.4481 | 1.52 | 4450 | 0.5021 | | 0.5035 | 1.54 | 4500 | 0.5022 | | 0.4038 | 1.56 | 4550 | 0.5036 | | 0.5086 | 1.57 | 4600 | 0.5035 | | 0.512 | 1.59 | 4650 | 0.5024 | | 0.4631 | 1.61 | 4700 | 0.5034 | | 0.4992 | 1.63 | 4750 | 0.5019 | | 0.5503 | 1.64 | 4800 | 0.5018 | | 0.4813 | 1.66 | 4850 | 0.5016 | | 0.554 | 1.68 | 4900 | 0.5019 | | 0.4033 | 1.69 | 4950 | 0.5018 | | 0.507 | 1.71 | 5000 | 0.5011 | | 0.5168 | 1.73 | 5050 | 0.5011 | | 0.498 | 1.74 | 5100 | 0.5012 | | 0.4781 | 1.76 | 5150 | 0.5012 | | 0.4656 | 1.78 | 5200 | 0.5005 | | 0.5745 | 1.8 | 5250 | 0.5006 | | 0.5095 | 1.81 | 5300 | 0.5006 | | 0.4743 | 1.83 | 5350 | 0.5005 | | 0.5316 | 1.85 | 5400 | 0.5004 | | 0.5014 | 1.86 | 5450 | 0.5003 | | 0.4273 | 1.88 | 5500 | 0.5002 | | 0.4237 | 1.9 | 5550 | 0.5003 | | 0.47 | 1.92 | 5600 | 0.5003 | | 0.4668 | 1.93 | 5650 | 0.5002 | | 0.523 | 1.95 | 5700 | 0.5002 | | 0.479 | 1.97 | 5750 | 0.5002 | | 0.4931 | 1.98 | 5800 | 0.5002 | ### Framework versions - Transformers 4.35.2 - Pytorch 2.1.0 - Datasets 2.15.0 - Tokenizers 0.15.0