symbolic-math-qwen2p5-1p5b-lora

LoRA adapter for Qwen2.5-Math-1.5B fine-tuned for step-by-step symbolic and numeric math problem solving (word problems, algebra, simple reasoning).

Model Details

• Developed by: Kevinmastascusa
• Model type: Causal LM (LoRA adapter)
• Language(s): English (math reasoning)
• License: Inherits base model’s license (see Qwen/Qwen2.5-Math-1.5B)
• Finetuned from: Qwen/Qwen2.5-Math-1.5B

Model Sources

• Base model: Qwen/Qwen2.5-Math-1.5B
• Repository: This page

Uses

• Direct Use: Attach this adapter to the base model to solve math word problems, simple algebra, and numeric reasoning.
• Downstream Use: Further fine-tune the adapter for domain-specific math tasks or curriculum data.

Out-of-Scope Use

• Safety-critical decision making
• Factual reasoning outside math
• Formal proof generation

How to Get Started

Basic usage (recommended: use tokenizer from this adapter repo):

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

adapter = "Kevinmastascusa/symbolic-math-qwen2p5-1p5b-lora"
base = "Qwen/Qwen2.5-Math-1.5B"

tok = AutoTokenizer.from_pretrained(adapter, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(base, trust_remote_code=True)

# Attach LoRA
model = PeftModel.from_pretrained(model, adapter)

prompt = "Solve for x: 2x + 5 = 13"
inputs = tok(prompt, return_tensors="pt")
gen = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tok.decode(gen[0], skip_special_tokens=True))

Robust loading if you see vocab-size mismatch errors:

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

adapter = "Kevinmastascusa/symbolic-math-qwen2p5-1p5b-lora"
base = "Qwen/Qwen2.5-Math-1.5B"

tok = AutoTokenizer.from_pretrained(adapter, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(
    base, trust_remote_code=True, ignore_mismatched_sizes=True
)
base_model.resize_token_embeddings(len(tok))
model = PeftModel.from_pretrained(base_model, adapter)

Optionally merge and save a standalone model:

merged = model.merge_and_unload()  # converts to a regular HF model
merged.save_pretrained("qwen2p5-math-1p5b-merged")
tok.save_pretrained("qwen2p5-math-1p5b-merged")

Training Details

• Method: PEFT LoRA on 4-bit quantized base (QLoRA style)
• LoRA config: r=8, alpha=16, dropout=0.05
• Batching: small per-device batch with gradient accumulation
• Other: gradient checkpointing enabled

Training Data

Mixture of common math datasets:

• GSM8K, SVAMP, MAWPS, MathQA, Math500 (subsets; cleaned/preprocessed)

Hyperparameters (typical)

• max_length: 256
• learning_rate: 2e-5
• epochs: 1–2
• eval/save every ~200 steps

Evaluation

This adapter targets step-by-step math reasoning. Evaluate with:

• token-level loss/perplexity on validation splits
• exact-match on final numeric answers (e.g., GSM8K-style EM)

Results will depend on your evaluation protocol and decoding settings.

Bias, Risks, and Limitations

• Math problems can be ambiguous; outputs may include reasoning lapses or formatting inconsistencies.
• Not robust to adversarial prompts or non-math tasks.
• Potential data biases from source datasets.

Recommendations

• Use deterministic decoding for answer extraction (no sampling).
• Post-process outputs to extract final numeric answers reliably.

Environmental Impact

• LoRA fine-tuning substantially reduces compute and memory compared to full fine-tunes.
• Actual emissions depend on your hardware and training duration.

Technical Specifications

• Architecture: Qwen2.5-Math-1.5B (Causal LM) + LoRA adapters on attention/MLP modules
• Objective: Causal LM (next-token prediction)

Citation

If you use this adapter, please cite the Qwen2.5 base model and PEFT.

Model Card Contact

For questions or issues, open a discussion on this model page.

Framework versions

• PEFT 0.17.0
• Transformers >= 4.41
• PyTorch >= 2.1