|
|
--- |
|
|
library_name: transformers |
|
|
datasets: |
|
|
- cardiffnlp/tweet_eval |
|
|
language: |
|
|
- en |
|
|
metrics: |
|
|
- f1 |
|
|
base_model: |
|
|
- TinyLlama/TinyLlama-1.1B-Chat-v1.0 |
|
|
pipeline_tag: text-classification |
|
|
--- |
|
|
|
|
|
Модель `TinyLlama/TinyLlama-1.1B-Chat-v1.0` дообученная на датасете `cardiffnlp/tweet_eval`, задача классификации сентимента твита, вывести одно из трех слов - |
|
|
`negative`, `neutral`, `positive`. |
|
|
|
|
|
## Дообучение |
|
|
|
|
|
Модель дообучалась при помощи QLoRA. |
|
|
|
|
|
- Ранг LoRA = `8` |
|
|
- QLoRA применялась ко всем весам attention, Q, K, V, O |
|
|
- `load_in_4bit=True` |
|
|
- `bnb_4bit_quant_type="nf4"` |
|
|
- `bnb_4bit_compute_dtype=torch.float16` |
|
|
- `lora_alpha=4` |
|
|
- `lora_dropout=0.0` |
|
|
- `bias="none"` |
|
|
- `task_type="CAUSAL_LM"` |
|
|
- `num_train_epochs=1` |
|
|
- `per_device_train_batch_size=32` |
|
|
- `gradient_accumulation_steps=1` |
|
|
- `warmup_steps=100` |
|
|
- `group_by_length=True` |
|
|
- `lr_scheduler_type="cosine"` |
|
|
- `fp16=True` |
|
|
- `learning_rate=2e-4` |
|
|
- `weight_decay=1e-3` |
|
|
|
|
|
## Метрика на валидации |
|
|
|
|
|
F1=0.18 |
|
|
|
|
|
|
|
|
 |
|
|
|
