End of training

README.md

@@ -16,13 +16,13 @@ This student model is distilled from the teacher model [gpt2](https://huggingfac
 The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
 
 It achieves the following results on the evaluation set:
-- eval_enwikippl: 1103.8743
-- eval_frwikippl: 4891.6226
-- eval_zhwikippl: 181095.5469
-- eval_loss: 1.4648
-- eval_runtime: 34.6071
-- eval_samples_per_second: 57.792
-- eval_steps_per_second: 7.224
+- eval_enwikippl: 321.9943
+- eval_frwikippl: 1758.7812
+- eval_zhwikippl: 13365.0029
+- eval_loss: 1.4163
+- eval_runtime: 38.2164
+- eval_samples_per_second: 52.334
+- eval_steps_per_second: 6.542
 
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment.
@@ -45,7 +45,7 @@ More information needed
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- distillation_objective: MultiObjective(logits_weight=1, logits_loss_fn=(fn:reverse_kl_divergence_loss()), activations_weight=0.2, activations_loss_fn=(fn:soft_mse_loss()), attentions_weight=0, attentions_loss_fn=(fn:soft_mse_loss()))
+- distillation_objective: MultiObjective(logits_weight=1, logits_loss_fn=(fn:cakld_loss()), activations_weight=0.2, activations_loss_fn=(fn:soft_mse_loss()), attentions_weight=0, attentions_loss_fn=(fn:soft_mse_loss()))
 - train_embeddings: True
 - learning_rate: 4e-05
 - train_batch_size: 8
@@ -56,38 +56,38 @@ The following hyperparameters were used during training:
 - num_epochs: 1.0
 
 ### Resource Usage
-Peak GPU Memory: 8.0873 GB
+Peak GPU Memory: 10.3900 GB
 
 ### Eval-Phase Metrics
 | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- |
 | **teacher eval** |  | 30.2086 | 57.2728 |  |  |  |  | 18.1784 |
-| 0 | 0 | 54069.2930 | 57285.3438 | 6.7941 | 34.4743 | 58.014 | 7.252 | 54227.1016 |
-| 1000 | 0.0404 | 2910.0552 | 15665.1387 | 2.0155 | 34.5048 | 57.963 | 7.245 | 728718.3125 |
-| 2000 | 0.0808 | 1922.8605 | 7541.5757 | 1.8376 | 34.6742 | 57.68 | 7.21 | 419463.0 |
-| 3000 | 0.1212 | 1727.4604 | 7344.2734 | 1.7664 | 34.5071 | 57.959 | 7.245 | 278908.7812 |
-| 4000 | 0.1616 | 1543.8636 | 6019.1860 | 1.6985 | 34.5865 | 57.826 | 7.228 | 194636.7344 |
-| 5000 | 0.2020 | 1539.0757 | 6114.1436 | 1.6518 | 34.6392 | 57.738 | 7.217 | 303388.4688 |
-| 6000 | 0.2424 | 1380.8512 | 5712.3887 | 1.5936 | 34.6683 | 57.69 | 7.211 | 583053.5625 |
-| 7000 | 0.2828 | 1236.5865 | 4909.5874 | 1.5434 | 34.5908 | 57.819 | 7.227 | 340582.5625 |
-| 8000 | 0.3232 | 1102.9314 | 4869.5972 | 1.5011 | 34.597 | 57.809 | 7.226 | 258743.5 |
-| 9000 | 0.3636 | 1103.8743 | 4891.6226 | 1.4648 | 34.6071 | 57.792 | 7.224 | 181095.5469 |
-| 10000 | 0.4040 | 984.7923 | 4532.3286 | 1.4278 | 34.6 | 57.803 | 7.225 | 185947.6875 |
-| 11000 | 0.4444 | 940.9836 | 3987.6274 | 1.3942 | 34.5943 | 57.813 | 7.227 | 199958.4531 |
-| 12000 | 0.4848 | 863.8093 | 3622.5100 | 1.3673 | 35.1415 | 56.913 | 7.114 | 195261.6094 |
-| 13000 | 0.5253 | 842.8707 | 3792.3918 | 1.3369 | 34.5948 | 57.812 | 7.227 | 225674.7188 |
-| 14000 | 0.5657 | 793.2106 | 3861.4631 | 1.3125 | 34.625 | 57.762 | 7.22 | 351017.6875 |
-| 15000 | 0.6061 | 740.2994 | 3135.8882 | 1.2954 | 34.6952 | 57.645 | 7.206 | 198680.8906 |
-| 16000 | 0.6465 | 722.9160 | 3156.2937 | 1.2711 | 34.846 | 57.395 | 7.174 | 240999.5781 |
-| 17000 | 0.6869 | 692.4757 | 3139.8684 | 1.2576 | 34.8258 | 57.429 | 7.179 | 279878.9062 |
-| 18000 | 0.7273 | 664.2447 | 2808.4719 | 1.2425 | 34.6399 | 57.737 | 7.217 | 299363.875 |
-| 19000 | 0.7677 | 659.3628 | 2955.1406 | 1.2301 | 34.7687 | 57.523 | 7.19 | 252868.3438 |
-| 20000 | 0.8081 | 641.1354 | 2671.7197 | 1.2159 | 34.6512 | 57.718 | 7.215 | 333471.875 |
-| 21000 | 0.8485 | 638.5021 | 2546.6489 | 1.2096 | 35.0645 | 57.038 | 7.13 | 200600.2188 |
-| 22000 | 0.8889 | 611.7115 | 2502.1531 | 1.2012 | 34.9815 | 57.173 | 7.147 | 176134.6875 |
-| 23000 | 0.9293 | 623.5575 | 2596.3247 | 1.1943 | 35.0514 | 57.059 | 7.132 | 147867.2031 |
-| 24000 | 0.9697 | 598.1364 | 2496.8660 | 1.1913 | 34.6448 | 57.729 | 7.216 | 413457.1562 |
-| 24750 | 1.0 | 608.4899 | 2650.3318 | 1.1902 | 35.1043 | 56.973 | 7.122 | 152194.375 |
+| 0 | 0 | 54069.2930 | 57285.3438 | 6.3621 | 38.3646 | 52.131 | 6.516 | 54227.1016 |
+| 1000 | 0.0404 | 1067.6234 | 5608.6260 | 2.0789 | 38.0715 | 52.533 | 6.567 | 96472.8984 |
+| 2000 | 0.0808 | 757.3948 | 3725.6042 | 1.9128 | 38.2045 | 52.35 | 6.544 | 17909.9004 |
+| 3000 | 0.1212 | 658.3904 | 3416.6064 | 1.8048 | 38.242 | 52.299 | 6.537 | 7773.6802 |
+| 4000 | 0.1616 | 560.8882 | 3119.1289 | 1.7206 | 38.3142 | 52.2 | 6.525 | 2095.8083 |
+| 5000 | 0.2020 | 497.5148 | 2912.9402 | 1.6453 | 38.3234 | 52.187 | 6.523 | 1419.0811 |
+| 6000 | 0.2424 | 431.3407 | 2770.3191 | 1.5753 | 38.2865 | 52.238 | 6.53 | 1246.3398 |
+| 7000 | 0.2828 | 380.4417 | 2245.9678 | 1.5120 | 38.2419 | 52.299 | 6.537 | 2103.0974 |
+| 8000 | 0.3232 | 348.0757 | 1937.6904 | 1.4605 | 38.1726 | 52.394 | 6.549 | 1825.5139 |
+| 9000 | 0.3636 | 321.9943 | 1758.7812 | 1.4163 | 38.2164 | 52.334 | 6.542 | 13365.0029 |
+| 10000 | 0.4040 | 302.1302 | 1743.9637 | 1.3804 | 38.2909 | 52.232 | 6.529 | 1458.2706 |
+| 11000 | 0.4444 | 283.6677 | 1578.4840 | 1.3479 | 38.0626 | 52.545 | 6.568 | 2750.6077 |
+| 12000 | 0.4848 | 270.2290 | 1480.8057 | 1.3235 | 38.2617 | 52.272 | 6.534 | 1864.9377 |
+| 13000 | 0.5253 | 253.0666 | 1502.0465 | 1.2928 | 38.0543 | 52.556 | 6.57 | 5540.5454 |
+| 14000 | 0.5657 | 241.2089 | 1442.4714 | 1.2666 | 38.1426 | 52.435 | 6.554 | 1532.5410 |
+| 15000 | 0.6061 | 225.8018 | 1385.6504 | 1.2418 | 38.089 | 52.509 | 6.564 | 984.5054 |
+| 16000 | 0.6465 | 219.4576 | 1220.6696 | 1.2177 | 38.2186 | 52.33 | 6.541 | 876.7557 |
+| 17000 | 0.6869 | 208.0084 | 1206.6365 | 1.1978 | 38.223 | 52.324 | 6.541 | 1094.0415 |
+| 18000 | 0.7273 | 204.6121 | 1193.9426 | 1.1802 | 38.2118 | 52.34 | 6.542 | 1641.8646 |
+| 19000 | 0.7677 | 197.2790 | 1129.5767 | 1.1681 | 38.2359 | 52.307 | 6.538 | 693.8603 |
+| 20000 | 0.8081 | 191.9000 | 1025.8606 | 1.1505 | 38.008 | 52.621 | 6.578 | 992.1599 |
+| 21000 | 0.8485 | 190.8004 | 1014.3529 | 1.1373 | 38.0057 | 52.624 | 6.578 | 1086.7607 |
+| 22000 | 0.8889 | 185.3244 | 978.5265 | 1.1308 | 38.0856 | 52.513 | 6.564 | 728.6151 |
+| 23000 | 0.9293 | 185.4108 | 969.4617 | 1.1227 | 38.0654 | 52.541 | 6.568 | 623.9739 |
+| 24000 | 0.9697 | 178.5300 | 955.3481 | 1.1107 | 38.1566 | 52.416 | 6.552 | 568.7457 |
+| 24750 | 1.0 | 179.6843 | 993.8232 | 1.1067 | 38.1733 | 52.393 | 6.549 | 685.6628 |
 
 ### Framework versions
 - Distily 0.2.0

logs/distillation_objective=MultiObjective(logits_weight_1__logits_loss_fn_(fn_cakld_loss())__activations_weight_0.2__activations_loss_fn_(fn_soft_mse_loss())__attentions_weight_0__attentions_loss_fn_(fn_s/events.out.tfevents.1723528003.93d6cbb3ad53

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5464dff160ade15d2520075da40a1ea6d40dbcc6617635026708f5e390e50abc
+size 253