File size: 3,585 Bytes
e5c924e 6886580 a585f92 6886580 e5c924e 6886580 04207f5 ac01a3e 6886580 ac01a3e 6886580 ac01a3e 70c8f6f ac01a3e 70c8f6f 6886580 14be679 6886580 70c8f6f 6886580 70c8f6f 6886580 70c8f6f ac01a3e 6886580 70c8f6f 6886580 ac01a3e 6886580 ac01a3e 6886580 ac01a3e 6886580 a585f92 70c8f6f ac01a3e 6886580 a585f92 6886580 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 |
---
license: unknown
library_name: peft
tags:
- mistral
datasets:
- ehartford/dolphin
- garage-bAInd/Open-Platypus
inference: false
pipeline_tag: text-generation
base_model: mistralai/Mistral-7B-v0.1
---
# mistral-7b-instruct-v0.1
General instruction-following llm finetuned from [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1).
## Model Details
### Model Description
This instruction-following llm was built via parameter-efficient QLoRA finetuning of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on the first 5k rows of [ehartford/dolphin](https://huggingface.co/datasets/ehartford/dolphin). Finetuning was executed on 1x A100 (40 GB SXM) for roughly 1 hour on Google Colab. **Only** the `peft` adapter weights are included in this model repo, alonside the tokenizer.
- **Developed by:** Daniel Furman
- **Model type:** Decoder-only
- **Language(s) (NLP):** English
- **License:** Yi model license
- **Finetuned from model:** [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)
### Model Sources
- **Repository:** [github.com/daniel-furman/sft-demos](https://github.com/daniel-furman/sft-demos/blob/main/src/sft/one_gpu/mistral/sft-mistral-7b-instruct-peft.ipynb)
### Evaluation
| Metric | Value |
|-----------------------|-------|
| MMLU (5-shot) | Coming |
| ARC (25-shot) | Coming |
| HellaSwag (10-shot) | Coming |
| TruthfulQA (0-shot) | Coming |
| Avg. | Coming |
We use Eleuther.AI's [Language Model Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) to run the benchmark tests above, the same version as Hugging Face's [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).
## Training
It took ~1 hour to train 1 epoch on 1x A100.
Prompt format:
This model (and all my future releases) use [ChatML](https://huggingface.co/docs/transformers/chat_templating#what-template-should-i-use) prompt format.
```
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
### Training Hyperparameters
We use the [`SFTTrainer`] (https://huggingface.co/docs/trl/main/en/sft_trainer) from 🤗's TRL package to easily fine-tune llms on instruction-following datasets.
The following `TrainingArguments` config was used:
- num_train_epochs = 1
- auto_find_batch_size = True
- gradient_accumulation_steps = 1
- optim = "paged_adamw_32bit"
- save_strategy = "epoch"
- learning_rate = 3e-4
- lr_scheduler_type = "cosine"
- warmup_ratio = 0.03
- logging_strategy = "steps"
- logging_steps = 25
- bf16 = True
The following `bitsandbytes` quantization config was used:
- quant_method: bitsandbytes
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: nf4
- bnb_4bit_use_double_quant: False
- bnb_4bit_compute_dtype: bfloat16
## How to Get Started with the Model
Use the code below to get started with the model.
[More Information Needed]
### Speeds, Sizes, Times
| runtime / 50 tokens (sec) | GPU | attn | torch dtype | VRAM (GB) |
|:-----------------------------:|:----------------------:|:---------------------:|:-------------:|:-----------------------:|
| 3.1 | 1x A100 (40 GB SXM) | torch | fp16 | 13 |
## Model Card Contact
dryanfurman at gmail
## Framework versions
- PEFT 0.6.0.dev0
|