Update README.md

README.md

@@ -41,7 +41,11 @@ Addressing the efficay of Quantization and PEFT. Implemented as a personal Proje
 
 ### How to use
 
-The quantized model is finetuned as PEFT. We have the trained Adapter. <br>The trained adpated needs to be merged with Base Model on which it was trained.
+```
+The quantized model is finetuned as PEFT. We have the trained Adapter.
+Merging LoRA adapated with GPTQ quantized model is not yet supported.
+So instead of loading a single finetuned model, we need to load the mase model and merge the finetuned adapter on top.
+```
 
 ```python
 instruction = """model_input = "Help me set up my daily to-do list!""""
@@ -73,20 +77,15 @@ print(code)
 
 HuggingFace Accelerate with Training Loop.
 
-#### Preprocessing 
-
-- ***Encoder Input:*** "sql_prompt: " + data['sql_prompt']+" sql_context: "+data['sql_context']
-- ***Decoder Input:*** data['sql']
-
 
 #### Training Hyperparameters
 
 - **Optimizer:** AdamW
 - **lr:** 2e-5
 - **decay:** linear
-- **num_warmup_steps:** 0
-- **batch_size:** 8
-- **num_training_steps:** 12500
+- **batch_size:** 4
+- **gradient_accumulation_steps:** 8
+- **global_step:** 625
 
 
 #### Hardware
@@ -94,51 +93,18 @@ HuggingFace Accelerate with Training Loop.
 - **GPU:** P100
 
 
-### Citing Dataset and BaseModel
-
-```
-  @software{gretel-synthetic-text-to-sql-2024,
-  author = {Meyer, Yev and Emadi, Marjan and Nathawani, Dhruv and Ramaswamy, Lipika and Boyd, Kendrick and Van Segbroeck, Maarten and Grossman, Matthew and Mlocek, Piotr and Newberry, Drew},
-  title = {{Synthetic-Text-To-SQL}: A synthetic dataset for training language models to generate SQL queries from natural language prompts},
-  month = {April},
-  year = {2024},
-  url = {https://huggingface.co/datasets/gretelai/synthetic-text-to-sql}
-}
-```
-
-```
-@article{DBLP:journals/corr/abs-1910-13461,
-  author    = {Mike Lewis and
-               Yinhan Liu and
-               Naman Goyal and
-               Marjan Ghazvininejad and
-               Abdelrahman Mohamed and
-               Omer Levy and
-               Veselin Stoyanov and
-               Luke Zettlemoyer},
-  title     = {{BART:} Denoising Sequence-to-Sequence Pre-training for Natural Language
-               Generation, Translation, and Comprehension},
-  journal   = {CoRR},
-  volume    = {abs/1910.13461},
-  year      = {2019},
-  url       = {http://arxiv.org/abs/1910.13461},
-  eprinttype = {arXiv},
-  eprint    = {1910.13461},
-  timestamp = {Thu, 31 Oct 2019 14:02:26 +0100},
-  biburl    = {https://dblp.org/rec/journals/corr/abs-1910-13461.bib},
-  bibsource = {dblp computer science bibliography, https://dblp.org}
-}
-
-```
-
 ## Additional Information
 
-- ***Github:*** [Repository](https://github.com/swastikmaiti/SwastikM-bart-large-nl2sql.git)
+- ***Github:*** [Repository]()
+- ***Intro to quantization:*** [Blog](https://huggingface.co/blog/merve/quantization)
+- ***Emergent Feature:*** [Academic](https://timdettmers.com/2022/08/17/llm-int8-and-emergent-features)
+- ***GPTQ Paper:*** [GPTQ](https://arxiv.org/pdf/2210.17323)
+- ***BITSANDBYTES and further*** [LLM.int8()](https://arxiv.org/pdf/2208.07339)
 
 ## Acknowledgment
 
-Thanks to [@AI at Meta](https://huggingface.co/facebook) for adding the Pre Trained Model.
-Thanks to [@Gretel.ai](https://huggingface.co/gretelai) for adding the datset.
+Thanks to [@AMerve Noyan](https://huggingface.co/blog/merve/quantization) for precise intro.
+Thanks to [@HuggungFace Team](https://colab.research.google.com/drive/1_TIrmuKOFhuRRiTWN94iLKUFu6ZX4ceb?usp=sharing#scrollTo=vT0XjNc2jYKy) for coding guide on gptq.
 
 
 ## Model Card Authors