README.md

metadata

library_name: transformers
tags:
  - generated_from_trainer
metrics:
  - accuracy
  - bleu
model-index:
  - name: rotating-head-gp-gpt2-medium-wikitext
    results: []

rotating-head-gp-gpt2-medium-wikitext

This model is a fine-tuned version of on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 3.2064
• Accuracy: 0.4186
• Perplexity: 24.6904
• Bleu: 0.1335

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 64
• eval_batch_size: 64
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 5

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Perplexity	Bleu
5.9051	0.2806	500	5.7418	0.2238	311.6215	0.0481
4.8617	0.5612	1000	4.7405	0.2812	114.4922	0.0721
4.2992	0.8418	1500	4.2277	0.3179	68.5611	0.0832
3.9585	1.1223	2000	3.9252	0.3466	50.6640	0.0905
3.7838	1.4029	2500	3.7564	0.3626	42.7959	0.0986
3.6863	1.6835	3000	3.6388	0.3739	38.0459	0.1069
3.5869	1.9641	3500	3.5518	0.3826	34.8757	0.1100
3.4733	2.2447	4000	3.4846	0.3886	32.6092	0.1159
3.4122	2.5253	4500	3.4307	0.3941	30.8979	0.1212
3.3791	2.8058	5000	3.3804	0.3991	29.3811	0.1223
3.2616	3.0864	5500	3.3447	0.4026	28.3518	0.1222
3.2499	3.3670	6000	3.3096	0.4067	27.3740	0.1261
3.2277	3.6476	6500	3.2812	0.4100	26.6073	0.1299
3.1992	3.9282	7000	3.2523	0.4128	25.8505	0.1305
3.13	4.2088	7500	3.2332	0.4154	25.3608	0.1326
3.0915	4.4893	8000	3.2200	0.4168	25.0291	0.1317
3.1011	4.7699	8500	3.2064	0.4186	24.6904	0.1335

Framework versions

• Transformers 4.49.0
• Pytorch 2.6.0+cu124
• Datasets 3.3.2
• Tokenizers 0.21.0