Upload README file

README.md

@@ -0,0 +1,135 @@
+---
+tags:
+- axolot
+- code
+- coding
+- Tinyllama
+- axolot
+model-index:
+- name: TinyLlama-1431k-python-coder
+  results: []
+license: apache-2.0
+language:
+- code
+datasets:
+- iamtarun/python_code_instructions_18k_alpaca
+pipeline_tag: text-generation
+---
+
+
+# TinyLlaMa 1.1B 1431k 4-bit Python Coder 👩‍💻 
+
+**TinyLlaMa 1.1B** fine-tuned on the **python_code_instructions_18k_alpaca Code instructions dataset** by using the **Axolot** library in 4-bit with [PEFT](https://github.com/huggingface/peft) library.
+
+## Pretrained description
+
+[TinyLlama-1.1B](https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T)
+
+The [TinyLlama project](https://github.com/jzhang38/TinyLlama) aims to pretrain a 1.1B Llama model on 3 trillion tokens. With some proper optimization, they can achieve this within a span of "just" 90 days using 16 A100-40G GPUs 🚀🚀.
+
+They adopted exactly the same architecture and tokenizer as Llama 2. This means TinyLlama can be plugged and played in many open-source projects built upon Llama. Besides, TinyLlama is compact with only 1.1B parameters. This compactness allows it to cater to a multitude of applications demanding a restricted computation and memory footprint.
+
+## Training data
+
+[python_code_instructions_18k_alpaca](https://huggingface.co/datasets/iamtarun/python_code_instructions_18k_alpaca)
+
+The dataset contains problem descriptions and code in python language. This dataset is taken from sahil2801/code_instructions_120k, which adds a prompt column in alpaca style.
+
+### Training hyperparameters
+
+The following `axolot` configuration was used during training:
+
+- load_in_8bit: false
+- load_in_4bit: true
+- strict: false
+
+- datasets:
+    - path: iamtarun/python_code_instructions_18k_alpaca
+      type: alpaca
+- dataset_prepared_path:
+- val_set_size: 0.05
+- output_dir: ./qlora-out
+
+- adapter: qlora
+- sequence_len: 1096
+- sample_packing: true
+- pad_to_sequence_len: true
+- lora_r: 32
+- lora_alpha: 16
+- lora_dropout: 0.05
+- lora_target_modules:
+- lora_target_linear: true
+- lora_fan_in_fan_out:
+- gradient_accumulation_steps: 1
+- micro_batch_size: 1
+- num_epochs: 2
+- max_steps:
+- optimizer: paged_adamw_32bit
+- lr_scheduler: cosine
+- learning_rate: 0.0002
+- train_on_inputs: false
+- group_by_length: false
+- bf16: false
+- fp16: true
+- tf32: false
+- gradient_checkpointing: true
+- logging_steps: 10
+- flash_attention: false
+- warmup_steps: 10
+- weight_decay: 0.0
+
+### Framework versions
+- torch=="2.1.2"
+- flash-attn=="2.5.0"
+- deepspeed=="0.13.1"
+- axolotl=="0.4.0"
+
+
+### Example of usage
+
+```py
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "edumunozsala/TinyLlama-1431k-python-coder"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+
+model = AutoModelForCausalLM.from_pretrained(model_id, load_in_4bit=True, torch_dtype=torch.float16, 
+                                             device_map="auto")
+
+instruction="Write a Python function to display the first and last elements of a list."
+input=""
+
+prompt = f"""### Instruction:
+Use the Task below and the Input given to write the Response, which is a programming code that can solve the Task.
+
+### Task:
+{instruction}
+
+### Input:
+{input}
+
+### Response:
+"""
+
+input_ids = tokenizer(prompt, return_tensors="pt", truncation=True).input_ids.cuda()
+# with torch.inference_mode():
+outputs = model.generate(input_ids=input_ids, max_new_tokens=100, do_sample=True, top_p=0.9,temperature=0.3)
+
+print(f"Prompt:\n{prompt}\n")
+print(f"Generated instruction:\n{tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True)[0][len(prompt):]}")
+
+```
+
+### Citation
+
+```
+@misc {edumunozsala_2023,
+	author       = { {Eduardo Muñoz} },
+	title        = { TinyLlama-1431k-python-coder },
+	year         = 2024,
+	url          = { https://huggingface.co/edumunozsala/TinyLlama-1431k-python-coder },
+	publisher    = { Hugging Face }
+}
+```