Phi-3-mini-finetuned-qa

A QLoRA fine-tuned version of microsoft/Phi-3-mini-4k-instruct trained for context-grounded question answering — the model reads a provided passage and generates a grounded answer. Designed for use as the generator in a Retrieval-Augmented Generation (RAG) pipeline.

Model Details

Model Description

Field	Detail
Developed by	Mufaddal Kangsawala
Model type	Causal Language Model — QLoRA LoRA adapter
Language(s) (NLP)	English
License	MIT (adapter weights); base model under Phi-3 License
Finetuned from model	microsoft/Phi-3-mini-4k-instruct (3.8B parameters)

Direct Use

Ask questions against a provided context paragraph. Best suited for:

• RAG pipelines — retrieve relevant document chunks, feed as context, get a grounded answer
• Document Q&A — PDF, DOCX, Markdown over a local vector store (ChromaDB)
• Personal knowledge base assistants

Input Format

The model expects Phi-3's native chat template:

<|system|>
You answer questions using the given context.<|end|>
<|user|>
Context:
{retrieved_passage}

Question: {user_question}<|end|>
<|assistant|>

Downstream Use

Plug directly into a RAG stack: retrieve top-k chunks from a vector store (e.g. ChromaDB with sentence-transformers/all-MiniLM-L6-v2 embeddings), pass as context, and stream the generated answer.

Out-of-Scope Use

• Open-ended generation without a context (the model was trained to be grounded — it may refuse or hallucinate without a passage)
• Languages other than English
• Tasks requiring factual world knowledge beyond the provided context

Bias, Risks, and Limitations

• The base Phi-3-mini-4k-instruct carries inherited biases from its pretraining data
• Fine-tuned on narrativeqa (fictional stories) — may perform less well on highly technical or scientific documents
• Hallucination risk: if the context does not contain the answer, the model may still generate plausible-sounding text
• CPU inference is slow (~5–15 sec per answer on a modern laptop)
• Context window limited to 4 096 tokens; very long documents must be chunked

Recommendations

• Always retrieve context before querying — do not use without a passage
• Validate answers against source documents for high-stakes use
• Use a relevance score threshold on retrieval to avoid low-quality context being passed in

How to Get Started with the Model

Use the code below to get started with the model.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "microsoft/Phi-3-mini-4k-instruct",
    torch_dtype=torch.float16,
    device_map="auto",
    attn_implementation="eager",
)
model = PeftModel.from_pretrained(base, "Mufaddalk/Phi-3-mini-finetuned-qa")
model = model.merge_and_unload()   # merge for faster inference
model.eval()

tokenizer = AutoTokenizer.from_pretrained("Mufaddalk/Phi-3-mini-finetuned-qa")

context  = "Python was created by Guido van Rossum and first released in 1991."
question = "Who created Python?"

prompt = (
    f"<|system|>\nYou answer questions using the given context.<|end|>\n"
    f"<|user|>\nContext:\n{context}\n\nQuestion: {question}<|end|>\n"
    f"<|assistant|>\n"
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=128, do_sample=False,
                         pad_token_id=tokenizer.pad_token_id)
answer = tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(answer)
# → "Python was created by Guido van Rossum."

Training Details

Training Data

Field	Value
Dataset	nvidia/ChatQA-Training-Data
Config	narrativeqa
Train split	4 500 examples (sampled + shuffled, seed=42)
Eval split	500 examples
Format	context + question → answer using Phi-3 chat template

Each example was formatted as:

<|system|>You answer questions using the given context.<|end|>
<|user|>Context:
{document}

Question: {question}<|end|>
<|assistant|>{answer}<|end|>

Training Procedure

Preprocessing

• Dataset shuffled with seed=42, then split 90/10 train/eval
• Each example serialised to the Phi-3 chat template with a leading <|system|> turn
• Sequences packed to max_length=1024 tokens (eliminates padding waste)

Training Hyperparameters

Hyperparameter	Value	Notes
Base model	Phi-3-mini-4k-instruct	3.8B params
Quantisation	4-bit NF4 + double quant	bitsandbytes
LoRA rank (r)	16
LoRA alpha	32	scaling = alpha/r = 2
LoRA dropout	0.05
Target modules	qkv_proj, o_proj	Phi-3 fused QKV
Trainable params	9 437 184 (0.25 % of 3.8B)
Learning rate	5e-5	cosine schedule
LR scheduler	cosine
Warmup steps	50
Epochs	1
Batch size	2 (per device)
Gradient accumulation	8	effective batch = 16
Sequence packing	True	eliminates padding waste
Max sequence length	1 024 tokens
Optimiser	paged AdamW 8-bit
Precision	fp32 (LoRA params) + fp16 (compute)	bf16 grads incompatible with T4
Gradient checkpointing	False	sufficient VRAM at bs=2

Speeds, Sizes, Times

Field	Value
Hardware	NVIDIA Tesla T4 (16 GB VRAM)
Training time	~90 minutes
Adapter size	~19 MB
Framework	PyTorch 2.4 · Transformers 4.46 · PEFT 0.13 · TRL 0.12

Evaluation

Testing Data, Factors & Metrics

Testing Data

200 held-out examples from the narrativeqa eval split of nvidia/ChatQA-Training-Data.

Factors

Single domain (fictional narratives); no disaggregation by document length or story genre.

Metrics

ROUGE scores (n-gram overlap between generated and reference answers). Standard for extractive/abstractive QA benchmarking.

Results

Metric	Score	Interpretation
ROUGE-1	0.5926	Unigram overlap
ROUGE-2	0.4019	Bigram overlap
ROUGE-L	0.5841	Longest common subsequence
ROUGE-Lsum	0.5830	Summary-level LCS

ROUGE-L > 0.35 is considered good for narrativeqa; 0.58 exceeds that benchmark significantly.

Summary

The fine-tuned adapter achieves strong ROUGE-L (0.584) on held-out narrativeqa examples, indicating the model reliably grounds its answers in the provided context rather than hallucinating.

Model Examination

The model uses LoRA adapters targeting the fused qkv_proj and o_proj projection layers only. All other weights remain frozen at 4-bit NF4 quantisation. After merge_and_unload(), the adapter is absorbed into the base weights for faster inference with no additional PEFT overhead.

Environmental Impact

Carbon emissions estimated using the ML CO2 Impact calculator.

Field	Value
Hardware	NVIDIA Tesla T4 (16 GB)
Hours used	~1.5 hours
Cloud provider	Google Cloud
Compute region	us-central1
Carbon emitted	~0.03 kg CO₂eq

Technical Specifications

Model Architecture and Objective

Phi-3-mini-4k-instruct is a 3.8B-parameter decoder-only transformer. The fine-tuned variant adds LoRA adapters (rank 16, alpha 32) to the fused QKV and output projection layers only, leaving all other weights frozen. Training objective is causal language modelling (next-token prediction) on the formatted chat template.

Compute Infrastructure

Hardware

NVIDIA Tesla T4 GPU — 16 GB VRAM.

Software

Library	Version
PyTorch	2.4
Transformers	4.46
PEFT	0.13
TRL	0.12
bitsandbytes	0.44
sentence-transformers	2.x

Citation

If you use this model, please cite the base model and training dataset:

@article{abdin2024phi3,
  title={Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone},
  author={Abdin, Marah and others},
  journal={arXiv preprint arXiv:2404.14219},
  year={2024}
}

@article{liu2024chatqa,
  title={ChatQA: Surpassing GPT-4 on Conversational QA and RAG},
  author={Liu, Zihan and others},
  journal={arXiv preprint arXiv:2401.10225},
  year={2024}
}

@article{dettmers2023qlora,
  title={QLoRA: Efficient Finetuning of Quantized LLMs},
  author={Dettmers, Tim and others},
  journal={arXiv preprint arXiv:2305.14314},
  year={2023}
}

Glossary

• QLoRA: Quantised Low-Rank Adaptation — fine-tunes a 4-bit quantised base model using small floating-point LoRA matrices
• NF4: NormalFloat 4-bit — a data type optimised for normally distributed weights
• RAG: Retrieval-Augmented Generation — retrieve relevant passages first, then generate a grounded answer
• ROUGE-L: Recall-Oriented Understudy for Gisting Evaluation (Longest Common Subsequence variant)

More Information

• Author LinkedIn: Mufaddal Kangsawala
• Base model: microsoft/Phi-3-mini-4k-instruct
• Training dataset: nvidia/ChatQA-Training-Data

Model Card Authors

Mufaddal Kangsawala

Model Card Contact

[More Information Needed]