license: apache-2.0
language:
- en
base_model:
- meta-llama/Llama-3.1-8B-Instruct
pipeline_tag: text-generation
tags:
- llama
- education
- transformers
- fine-tuning
- LoRA
- PEFT
- RSLoRA
- quantized
model_name: 169Pi/generic_slm
model_description: > The 169Pi/generic_slm is a fine-tuned version of the Meta-Llama-3.1-8B-bnb-4bit model, designed to deliver high-quality educational content. Leveraging techniques such as LoRA, PEFT, and RSLoRA, it aims to provide engaging, accurate, and contextually appropriate educational materials for students and educators.
tags:
- transformers
- llama
- education
- fine-tuning
- LoRA
- PEFT
- RSLoRA
- quantized
uses: direct_use: - Summarizing chapters or concepts - Answering curriculum-aligned questions - Generating practice questions and explanations - Recommending study materials downstream_use: - Interactive learning tools - Educational chatbots - Personalized study guides - Automated assessment materials out_of_scope: - Legal or financial decision-making - Generating non-educational content - Applications requiring high precision in non-educational contexts
training_details: dataset: Proprietary dataset by 169Pi preprocessing_steps: - Removed duplicates - Cleaned noisy and irrelevant data - Normalized text for consistency parameter_size: 4.65 billion (quantized to 4-bit) hyperparameters: - learning_rate: 5e-5 - lr_scheduler_type: cosine - batch_size_per_device: 32 - gradient_accumulation_steps: 4 - num_epochs: 3 - fp16: True - bf16: True - optimizer: adamw_8bit - weight_decay: 0.05 - warmup_steps: 1000 - logging_steps: 1000 - evaluation_strategy: steps - eval_steps: 1000 - save_strategy: steps - save_steps: 1000 - output_dir: output - random_seed: 3407
architecture: base_model: Meta-Llama-3.1-8B quantization: 4-bit techniques: - LoRA - PEFT - RSLoRA
bias_risks_and_limitations: known_biases: > Potential biases in educational content sources, including cultural or linguistic preferences. risks: > Model may generate incorrect or general responses for ambiguous queries. recommendations: > Use cautiously in critical contexts. Regularly evaluate outputs for accuracy and bias.
technical_specifications: model_architecture: > Transformer-based architecture with multi-head self-attention, enhanced using LoRA, PEFT, and RSLoRA. Optimized for educational tasks. objective: > Generate high-quality educational content, including summarization, question-answering, and study material generation.
evaluation: metrics: primary: Loss during training secondary: Accuracy and relevance through manual evaluation results: > Achieved low validation loss during training, demonstrating generalization capability.
environmental_impact: hardware: NVIDIA A100 training_duration: 26 hours compute_provider: Not specified region: Not specified
citation: > @misc{169Pi_generic_slm, title={169Pi/generic_slm: Fine-Tuned Educational Model}, author={169Pi}, year={2024}, publisher={Hugging Face}, url={https://huggingface.co/169Pi/generic_slm} } contact: developer: 169Pi AI email: [email protected]