|
|
--- |
|
|
library_name: transformers |
|
|
tags: |
|
|
- llama-factory |
|
|
base_model: |
|
|
- mistralai/Mistral-7B-Instruct-v0.3 |
|
|
--- |
|
|
|
|
|
## Summary |
|
|
|
|
|
This model is a fine-tuned version of the Mistral-7B-Instruct-v0.3 optimised for answering questions in the domain of forensic investigations. The model has been trained using a specialised dataset titled **Advanced_Forensic_Investigations_Knowledge_Library_v1**, which consists of approximately 100 domain-specific question-answer pairs. The objective is to support advanced forensic investigative reasoning, rapid knowledge retrieval, and high-precision forensic domain assistance. |
|
|
|
|
|
--- |
|
|
|
|
|
## Model Details |
|
|
|
|
|
### Model Description |
|
|
|
|
|
- **Model type:** Instruction-following Language Model (LoRA-based fine-tuning) |
|
|
|
|
|
- **Language(s):** English |
|
|
|
|
|
- **Fine-tuned from model:** Mistral-7B-Instruct-v0.3 |
|
|
|
|
|
--- |
|
|
## Training Details |
|
|
|
|
|
### Training Data |
|
|
|
|
|
- **Dataset:** `Advanced_Forensic_Investigations_Knowledge_Library_v1` |
|
|
|
|
|
- **Data size:** ~100 high-quality, domain-specific QA pairs |
|
|
|
|
|
|
|
|
### Training Procedure |
|
|
|
|
|
#### Preprocessing |
|
|
|
|
|
- Template: `mistral` |
|
|
|
|
|
- Token truncation/cutoff: 2048 |
|
|
|
|
|
- No vocab resizing or prompt packing |
|
|
|
|
|
|
|
|
#### Hyperparameters |
|
|
|
|
|
- **Finetuning type:** LoRA |
|
|
|
|
|
- **Precision:** `bf16` |
|
|
|
|
|
- **LoRA rank:** 16 |
|
|
|
|
|
- **LoRA alpha:** 32 |
|
|
|
|
|
- **Batch size:** 4 |
|
|
|
|
|
- **Gradient accumulation:** 8 |
|
|
|
|
|
- **Learning rate:** `3e-4` |
|
|
|
|
|
- **Epochs:** 35 |
|
|
|
|
|
- **LR scheduler:** cosine |
|
|
|
|
|
- **Quantisation:** 4-bit (bitsandbytes) |
|
|
|
|
|
- **Cutoff length:** 2048 |
|
|
|
|
|
|
|
|
#### Compute |
|
|
|
|
|
- **Training time:** close to 30 minutes |
|
|
|
|
|
- **Framework:** LLaMA-Factory |
|
|
|
|
|
--- |
|
|
|
|
|
## Evaluation |
|
|
|
|
|
### Testing Data, Factors & Metrics |
|
|
|
|
|
- **Metrics Used:** BLEU-4, ROUGE-1, ROUGE-2 |
|
|
|
|
|
- **Results:** |
|
|
|
|
|
- BLEU-4: 100% |
|
|
|
|
|
- ROUGE-1: 100% |
|
|
|
|
|
- ROUGE-2: 100% |
|
|
|
|
|
|
|
|
These scores reflect perfect overlap with reference answers within the scope of the evaluation dataset. |
|
|
|
|
|
--- |
|
|
|
|
|
## Technical Specifications |
|
|
|
|
|
### Model Architecture and Objective |
|
|
|
|
|
- **Base:** Transformer (Mistral-7B architecture) |
|
|
|
|
|
- **Fine-tuning method:** LoRA |
|
|
|
|
|
- **Objective:** Instruction-following with forensic legal knowledge adaptation |
|
|
|
|
|
|
|
|
### Compute Infrastructure |
|
|
|
|
|
- **Hardware:** 2xL40s |
|
|
|
|
|
- **Software:** LLaMA-Factory, PyTorch, Transformers, bitsandbytes |
