edithatogo/qwen3-4b-hermes-lora-peft-converted

Qwen3 4B Hermes Strict Tool-Call V4 Targeted PEFT Conversion

Release status: public experimental converted adapter release.

Summary

This is an experimental PEFT-format conversion of the public MLX LoRA adapter edithatogo/qwen3-4b-hermes-lora. It is intended to make the Qwen3 v4 strict Hermes tool-call adapter usable from CUDA/Hugging Face tooling such as transformers, peft, and lm-evaluation-harness.

The PEFT base model is:

Qwen/Qwen3-4B

Source MLX adapter repo:

https://huggingface.co/edithatogo/qwen3-4b-hermes-lora

Converted PEFT adapter repo:

https://huggingface.co/edithatogo/qwen3-4b-hermes-lora-peft-converted

The adapter is intended for local evaluation and agent-runtime packaging. It requires the recorded runtime prompt condition:

• first user turn prefixed with /no_think
• assistant prefill: <think>\n\n</think>\n\n

Without the assistant prefill, the model still emits an empty leading thinking wrapper and does not satisfy the strict raw-output gate.

Base Model

• PEFT base: Qwen/Qwen3-4B
• Source adapter base: Qwen/Qwen3-4B-MLX-4bit
• Base license: Apache-2.0, checked via Hugging Face API on 2026-05-25

Conversion

• Source adapter: gemma4/experiments/qwen3-4b-strict-toolcall-v4-targeted/lora_adapter
• Conversion script: scripts/convert_mlx_lora_to_peft.py
• Conversion report: reports/cloud/qwen3-v4-mlx-to-peft-conversion-20260613.md
• Source tensors: 112
• Converted PEFT tensors: 112
• LoRA rank: 8
• LoRA alpha: 16.0
• Layers: 28-35
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Training

• Training config: gemma4/scripts/train_config.qwen3-4b.strict-toolcall-v4-targeted.yaml
• Data: gemma4/data/strict_tool_call/expanded_splits_v4_targeted
• Adapter: gemma4/experiments/qwen3-4b-strict-toolcall-v4-targeted/lora_adapter
• Training tokens: 37,936
• Dataset token audit: reports/publication/qwen3-4b-strict-toolcall-v4-targeted/dataset-token-audit.json
• Dataset overlap audit: reports/publication/qwen3-4b-strict-toolcall-v4-targeted/dataset-overlap-audit.json
• Peak memory: 3.785 GB

Evaluation

PEFT conversion checks:

Check	Status
Static PEFT config load	pass
Colab T4 4-bit PEFT load smoke	pass
Colab T4 lm_eval[hf] selected task route, limit 5	pass
Full no-limit lm_eval scorecard	blocked by Colab session pruning

Bounded lm_eval route pilot on Colab T4:

Task	Metric	Value	Samples
arc_challenge	acc_norm	0.2000	5
hellaswag	acc_norm	0.6000	5
truthfulqa_mc2	acc	0.5166	5
gsm8k	exact_match,strict-match	0.8000	5
winogrande	acc	0.4000	5

These are route-pilot scores only and must not be used as no-limit benchmark claims.

Held-out strict local tool-call gate:

Suite	Pass	JSON valid	Arguments	Invalid tool	Multi-turn
benchmarks/tool_call_local/heldout_suite.json	1.000	1.000	1.000	1.000	1.000

Mirrored regression:

Suite	Pass
benchmarks/tool_call_local/suite.json	1.000

Repo-native pilot benchmarks:

Pilot	Pass	Notes
BFCL-style pilot	0.667	local pilot only, not official BFCL
IFEval-style pilot	0.667	local pilot only, not official IFEval
Coding sanity pilot	1.000	local pilot only, not HumanEval/MBPP

Exact held-out command:

source scripts/env.sh
PYTHONPATH=scripts ./.venv/bin/python scripts/run_tool_call_benchmark.py \
  --model Qwen/Qwen3-4B-MLX-4bit \
  --adapter gemma4/experiments/qwen3-4b-strict-toolcall-v4-targeted/lora_adapter \
  --suite benchmarks/tool_call_local/heldout_suite.json \
  --user-prefix /no_think \
  --assistant-prefill $'<think>\n\n</think>\n\n' \
  --run-id qwen3-4b-strict-toolcall-v4-targeted-heldout-prefill-20260525 \
  --max-tokens 256

Raw local artifact:

/Volumes/PortableSSD/hermes-evals/tool-call-benchmark/qwen3-4b-strict-toolcall-v4-targeted-heldout-prefill-20260525

The reusable runtime prompt contract is recorded in RUNTIME_PROMPT_PROFILES.yaml as qwen3-no-think-assistant-prefill.

Limitations

• This is an experimental conversion from MLX LoRA tensor orientation to PEFT tensor orientation. Use the original MLX adapter repo for the canonical MLX release.
• This is a small local strict-format benchmark, not broad BFCL or production tool-use evidence.
• The PEFT route has a successful Colab T4 load smoke and bounded lm_eval pilot, but no full no-limit lm_eval scorecard yet.
• The release does not include official BFCL, HumanEval, MBPP, EvalPlus, BigCodeBench, LiveCodeBench, safety/refusal, or RULER long-context scores.
• The selected lm_eval endpoint route was attempted separately, but the current local MLX endpoint is not loglikelihood-compatible for those tasks. A direct MLX adapter has scored bounded selected-task limit-10 and limit-25 runs; treat those as pilot evidence only, not as full official lm_eval or leaderboard scores.
• The adapter is sensitive to runtime prompt formatting.
• The V4 training data has no held-out user-prompt overlap in the recorded audit, but it shares one generic held-out tool name, notify_care_team.
• Dataset/source redistribution review is complete for adapter-release purposes with caveats. The separately approved cleaned synthetic-only dataset has been published at https://huggingface.co/datasets/edithatogo/qwen3-hermes-strict-toolcall-synthetic-v4.
• Public release approval is recorded in release-decision.md; the publication bundle is expected to pass scripts/validate_publication_bundle.py --require-ready.