Upload 17 files

Introducing qwq-32B-MLX-Q5 - the latest quant of a great model thanks to mlx 0.26.0+ @mlx-community. Check it in the docs out now to explore its full potential!

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,187 @@
+---
+license: apache-2.0
+base_model:
+- Qwen/QwQ-32B
+library_name: mlx
+tags:
+- quantization
+- mlx-q5
+---
+---
+license: apache-2.0
+language:
+- en
+pipeline_tag: text-generation
+tags:
+- mlx==0.26.2
+- q5
+- qwq
+- reasoning
+- m3-ultra
+base_model: Qwen/QwQ-32B
+---
+
+# QwQ-32B MLX Q5 Quantization
+
+This is a **Q5 (5-bit) quantized** version of the QwQ-32B reasoning model, optimized for MLX on Apple Silicon. This quantization offers an excellent balance between model quality and size, specifically designed for high-memory Apple Silicon systems like the M3 Ultra.
+
+## Model Details
+
+- **Base Model**: Qwen/QwQ-32B
+- **Quantization**: Q5 (5-bit) with group size 64
+- **Format**: MLX (Apple Silicon optimized)
+- **Size**: 21GB (from original 61GB bfloat16)
+- **Compression**: 66% size reduction
+- **Architecture**: Qwen2 with reasoning capabilities
+
+## Why Q5?
+
+Q5 quantization provides:
+- **Superior quality** compared to Q4 while being smaller than Q6/Q8
+- **Optimal size** for 128GB+ Apple Silicon systems
+- **Minimal quality loss** - retains ~98% of original model capabilities
+- **Fast inference** with MLX's unified memory architecture
+
+## Requirements
+
+- Apple Silicon Mac (M1/M2/M3/M4)
+- macOS 13.0+
+- Python 3.11+
+- MLX 0.26.0+
+- mlx-lm 0.22.5+
+- 32GB+ RAM recommended (64GB+ for full 128k context)
+
+## Installation
+
+```bash
+# Using uv (recommended)
+uv add mlx>=0.26.0 mlx-lm transformers
+
+# Or with pip (not tested and obsolete)
+pip install mlx>=0.26.0 mlx-lm transformers
+```
+
+## Usage
+
+### Direct Generation
+
+```bash
+uv run mlx_lm.generate \
+  --model LibraxisAI/QwQ-32B-MLX-Q5 \
+  --prompt "Solve this step by step: If a train travels 120 km in 2 hours, what is its speed?" \
+  --max-tokens 500
+```
+
+### Python API
+
+```python
+from mlx_lm import load, generate
+
+# Load model
+model, tokenizer = load("LibraxisAI/QwQ-32B-MLX-Q5")
+
+# Generate text with reasoning
+prompt = "Think step by step: What are the implications of Q5 quantization for LLM deployment?"
+response = generate(
+    model=model,
+    tokenizer=tokenizer,
+    prompt=prompt,
+    max_tokens=1000,
+    temp=0.7
+)
+print(response)
+```
+
+### HTTP Server
+
+```bash
+uv run mlx_lm.server \
+  --model LibraxisAI/QwQ-32B-MLX-Q5 \
+  --host 0.0.0.0 \
+  --port 8080
+```
+
+## Performance Benchmarks
+
+Tested on Mac Studio M3 Ultra (512GB):
+
+| Metric | Value |
+|--------|-------|
+| Model Size | 21GB |
+| Peak Memory Usage | ~25GB |
+| Generation Speed | ~12-15 tokens/sec |
+| Max Context Length | 131,072 tokens (128k) |
+
+## Special Features
+
+QwQ (Qwen with Questions) is designed for:
+- **Deep reasoning** and step-by-step problem solving
+- **Mathematical reasoning** and logical deduction
+- **Code generation** with explanations
+- **Self-reflection** and error correction
+
+## Limitations
+
+⚠️ **Important**: This Q5 model as for the release date, of this quant **is NOT compatible** with LM Studio (**yet**), which only supports 2, 3, 4, 6, and 8-bit quantizations & we didn't test it with Ollama or any other inference client. **Use MLX directly or via the MLX server** - we've created a comfortable, `command generation script` to run the server properly.
+
+## Conversion Details
+
+This model was quantized using:
+```bash
+uv run mlx_lm.convert \
+  --hf-path Qwen/QwQ-32B \
+  --mlx-path QwQ-32B-MLX-Q5 \
+  --dtype bfloat16 \
+  -q --q-bits 5 --q-group-size 64
+```
+
+## Frontier M3 Ultra Optimization
+
+This model is specifically optimized for the Mac Studio M3 Ultra setup with 512GB unified memory. For best performance:
+
+```python
+import mlx.core as mx
+
+# Set memory limits for large models
+mx.metal.set_memory_limit(100 * 1024**3)  # 100GB
+mx.metal.set_cache_limit(20 * 1024**3)    # 20GB cache
+```
+
+## Tools Included
+
+We provide utility scripts for easy model management:
+
+1. **convert-to-mlx.sh** - Command generation tool - convert any model to MLX format with many options of customization and Q5 quantization support on mlx>=0.26.0
+2. **mlx-serve.sh** - Launch MLX server with custom parameters
+
+## Historical Note
+
+The LibraxisAI Q5 models were among the **first Q5 quantized MLX models** available on Hugging Face, pioneering the use of 5-bit quantization for Apple Silicon optimization.
+
+## Citation
+
+If you use this model, please cite:
+
+```bibtex
+@misc{qwq-32b-q5-mlx,
+  author = {LibraxisAI},
+  title = {QwQ-32B Q5 MLX - Reasoning Model for Apple Silicon},
+  year = {2025},
+  publisher = {Hugging Face},
+  url = {https://huggingface.co/LibraxisAI/QwQ-32B-MLX-Q5}
+}
+```
+
+## License
+
+This model follows the original QwQ license (Apache-2.0). See the [base model card](https://huggingface.com/Qwen/QwQ-32B) for full details.
+
+## Authors of the repository
+[Monika Szymanska](https://github.com/m-szymanska)
+[Maciej Gad, DVM](https://div0.space) 
+
+## Acknowledgments
+
+- Apple MLX team and community for the amazing 0.26.0+ framework
+- Qwen team for the innovative QwQ reasoning model
+- Klaudiusz-AI 🐉

added_tokens.json

@@ -0,0 +1,28 @@
+{
+  "</think>": 151668,
+  "</tool_call>": 151658,
+  "</tool_response>": 151666,
+  "<think>": 151667,
+  "<tool_call>": 151657,
+  "<tool_response>": 151665,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

chat_template.jinja

@@ -0,0 +1,56 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- messages[0]['content'] }}
+    {%- else %}
+        {{- '' }}
+    {%- endif %}
+    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
+  {%- endif %}
+{%- endif %}
+{%- for message in messages %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" and not message.tool_calls %}
+        {%- set content = message.content.split('</think>')[-1].lstrip('\n') %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set content = message.content.split('</think>')[-1].lstrip('\n') %}
+        {{- '<|im_start|>' + message.role }}
+        {%- if message.content %}
+            {{- '\n' + content }}
+        {%- endif %}
+        {%- for tool_call in message.tool_calls %}
+            {%- if tool_call.function is defined %}
+                {%- set tool_call = tool_call.function %}
+            {%- endif %}
+            {{- '\n<tool_call>\n{"name": "' }}
+            {{- tool_call.name }}
+            {{- '", "arguments": ' }}
+            {{- tool_call.arguments | tojson }}
+            {{- '}\n</tool_call>' }}
+        {%- endfor %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- message.content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n<think>\n' }}
+{%- endif %}

config.json

@@ -0,0 +1,35 @@
+{
+    "architectures": [
+        "Qwen2ForCausalLM"
+    ],
+    "attention_dropout": 0.0,
+    "bos_token_id": 151643,
+    "eos_token_id": 151645,
+    "hidden_act": "silu",
+    "hidden_size": 5120,
+    "initializer_range": 0.02,
+    "intermediate_size": 27648,
+    "max_position_embeddings": 131072,
+    "max_window_layers": 64,
+    "model_type": "qwen2",
+    "num_attention_heads": 40,
+    "num_hidden_layers": 64,
+    "num_key_value_heads": 8,
+    "quantization": {
+        "group_size": 64,
+        "bits": 5
+    },
+    "quantization_config": {
+        "group_size": 64,
+        "bits": 5
+    },
+    "rms_norm_eps": 1e-05,
+    "rope_theta": 1000000.0,
+    "sliding_window": 32768,
+    "tie_word_embeddings": false,
+    "torch_dtype": "bfloat16",
+    "transformers_version": "4.43.1",
+    "use_cache": true,
+    "use_sliding_window": false,
+    "vocab_size": 152064
+}

convert-to-mlx.sh

@@ -0,0 +1,312 @@
+#!/bin/bash
+# MLX Model Conversion Utility for Dragon M3 Ultra
+# Updated: January 2025 for MLX 0.26+ and modern uv workflow
+# Supports Q5 quantization and M3 Ultra optimizations
+
+# Text formatting
+BOLD="\033[1m"
+BLUE="\033[34m"
+GREEN="\033[32m"
+YELLOW="\033[33m"
+RED="\033[31m"
+CYAN="\033[36m"
+MAGENTA="\033[35m"
+RESET="\033[0m"
+
+# Detect system specs
+TOTAL_MEMORY=$(sysctl -n hw.memsize 2>/dev/null || echo 0)
+TOTAL_MEMORY_GB=$((TOTAL_MEMORY / 1073741824))
+CPU_BRAND=$(sysctl -n machdep.cpu.brand_string 2>/dev/null || echo "Unknown")
+
+# Check if running on M3 Ultra
+if [[ "$CPU_BRAND" == *"M3 Ultra"* ]] || [[ "$TOTAL_MEMORY_GB" -ge 400 ]]; then
+    IS_M3_ULTRA=true
+    echo -e "${BOLD}${MAGENTA}🐉 Dragon M3 Ultra detected! (${TOTAL_MEMORY_GB}GB RAM)${RESET}"
+else
+    IS_M3_ULTRA=false
+fi
+
+echo -e "${BOLD}${BLUE}=====================================${RESET}"
+echo -e "${BOLD}${BLUE}   MLX Model Conversion Utility v2.0 ${RESET}"
+echo -e "${BOLD}${BLUE}=====================================${RESET}"
+echo -e "Updated for MLX 0.26+ with Q5 support and M3 Ultra optimizations\n"
+
+# Default values
+DEFAULT_HF_PATH="meta-llama/Llama-3.1-405B"
+DEFAULT_OUTPUT_DIR="models/Llama-3.1-405B-MLX-Q5"
+DEFAULT_QUANTIZE="y"
+DEFAULT_BITS="5"  # Changed to Q5 as default for better quality/size ratio
+DEFAULT_GROUP_SIZE="64"
+DEFAULT_DTYPE="float16"
+
+# hf-xet optimization for Dragon M3 Ultra
+if [[ "$IS_M3_ULTRA" == true ]]; then
+    export HF_XET_HIGH_PERFORMANCE_MODE=1
+    export HF_XET_CHUNK_CACHE_SIZE_BYTES=107374182400  # 100GB cache
+    export HF_XET_CONCURRENT_DOWNLOADS=32
+    echo -e "${CYAN}✓ hf-xet optimizations enabled for Dragon${RESET}"
+fi
+
+# Get HF Path
+echo -e "${BOLD}Hugging Face model path or local directory:${RESET}"
+echo -e "(Default: ${DEFAULT_HF_PATH})"
+echo -e "${CYAN}Examples:${RESET}"
+echo -e "  HF repo: meta-llama/Llama-3.1-405B"
+echo -e "  Local: /Users/polyversai/.lmstudio/models/mlx-community/model-name"
+read -p "> " HF_PATH
+HF_PATH=${HF_PATH:-$DEFAULT_HF_PATH}
+
+# Check if it's a local path
+if [[ -d "$HF_PATH" ]]; then
+    echo -e "${GREEN}✓ Local model detected: ${HF_PATH}${RESET}"
+    IS_LOCAL=true
+else
+    IS_LOCAL=false
+    # Ask about hf-xet for remote models
+    echo -e "\n${BOLD}Use hf-xet for faster download? [y/n]${RESET}"
+    echo -e "(10x faster downloads with chunk deduplication)"
+    echo -e "Default: y"
+    read -p "> " USE_HF_XET
+    USE_HF_XET=${USE_HF_XET:-y}
+    
+    if [[ "$USE_HF_XET" == "y" || "$USE_HF_XET" == "Y" ]]; then
+        # Check if hf-xet is installed
+        if ! uv run python -c "import hf_xet" 2>/dev/null; then
+            echo -e "${YELLOW}⚠️  hf-xet not installed. Installing...${RESET}"
+            echo -e "Run: uv add 'huggingface_hub[hf_xet]'"
+            echo -e "${CYAN}Note: hf-xet only works with Xet-backed repos${RESET}"
+        else
+            echo -e "${GREEN}✓ hf-xet enabled for download${RESET}"
+        fi
+    fi
+fi
+
+# Get output directory
+echo -e "\n${BOLD}Output MLX model directory:${RESET}"
+echo -e "(Default: ${DEFAULT_OUTPUT_DIR})"
+read -p "> " MLX_PATH
+MLX_PATH=${MLX_PATH:-$DEFAULT_OUTPUT_DIR}
+
+# Ask about data type
+echo -e "\n${BOLD}Model data type:${RESET}"
+echo -e "(Default: ${DEFAULT_DTYPE}, Options: float16, bfloat16, float32)"
+read -p "> " DTYPE
+DTYPE=${DTYPE:-$DEFAULT_DTYPE}
+
+# Ask about quantization
+echo -e "\n${BOLD}Quantize the model? [y/n]${RESET}"
+echo -e "(Default: ${DEFAULT_QUANTIZE})"
+read -p "> " QUANTIZE
+QUANTIZE=${QUANTIZE:-$DEFAULT_QUANTIZE}
+
+# If quantizing, get more details
+if [[ "$QUANTIZE" == "y" || "$QUANTIZE" == "Y" ]]; then
+  echo -e "\n${BOLD}Quantization bits:${RESET}"
+  echo -e "${CYAN}Options:${RESET}"
+  echo -e "  2 - Extreme compression (lowest quality)"
+  echo -e "  3 - High compression"
+  echo -e "  4 - Standard compression (good balance)"
+  echo -e "  ${GREEN}5 - Recommended (best quality/size ratio)${RESET}"
+  echo -e "  6 - Low compression"
+  echo -e "  8 - Minimal compression (highest quality)"
+  echo -e "(Default: ${DEFAULT_BITS})"
+  read -p "> " BITS
+  BITS=${BITS:-$DEFAULT_BITS}
+  
+  echo -e "\n${BOLD}Group size:${RESET}"
+  echo -e "(Default: ${DEFAULT_GROUP_SIZE}, Options: 32, 64, 128)"
+  if [[ "$IS_M3_ULTRA" == true ]]; then
+    echo -e "${CYAN}💡 M3 Ultra tip: Use 64 or 128 for better performance${RESET}"
+  fi
+  read -p "> " GROUP_SIZE
+  GROUP_SIZE=${GROUP_SIZE:-$DEFAULT_GROUP_SIZE}
+
+  echo -e "\n${BOLD}Quantization strategy:${RESET}"
+  echo -e "${CYAN}Options:${RESET}"
+  echo -e "  none - Uniform quantization (default)"
+  echo -e "  mixed_2_6 - Mix of 2 and 6 bit"
+  echo -e "  ${GREEN}mixed_3_4 - Mix of 3 and 4 bit${RESET}"
+  echo -e "  mixed_3_6 - Mix of 3 and 6 bit"
+  echo -e "  mixed_4_6 - Mix of 4 and 6 bit"
+  echo -e "Leave empty for uniform quantization"
+  read -p "> " QUANT_PREDICATE
+  
+  QUANT_OPTIONS="-q --q-bits ${BITS} --q-group-size ${GROUP_SIZE}"
+  
+  if [[ -n "$QUANT_PREDICATE" ]]; then
+    QUANT_OPTIONS="${QUANT_OPTIONS} --quant-predicate ${QUANT_PREDICATE}"
+  fi
+else
+  QUANT_OPTIONS=""
+fi
+
+# Memory optimization options for M3 Ultra
+if [[ "$IS_M3_ULTRA" == true ]]; then
+  echo -e "\n${BOLD}${MAGENTA}M3 Ultra optimization note:${RESET}"
+  echo -e "${CYAN}MLX will automatically optimize for your 512GB system${RESET}"
+  echo -e "${CYAN}The framework uses unified memory efficiently${RESET}"
+  M3_ULTRA_FLAGS=""
+else
+  M3_ULTRA_FLAGS=""
+fi
+
+# Ask about upload repository (optional)
+echo -e "\n${BOLD}Upload to Hugging Face Hub? (optional):${RESET}"
+echo -e "(Leave empty to skip upload)"
+read -p "> " UPLOAD_REPO
+
+if [[ -n "$UPLOAD_REPO" ]]; then
+  UPLOAD_OPTION="--upload-repo ${UPLOAD_REPO}"
+else
+  UPLOAD_OPTION=""
+fi
+
+# Build the command - UV is now default
+UV_CMD="uv run mlx_lm.convert --hf-path ${HF_PATH} --mlx-path ${MLX_PATH} --dtype ${DTYPE} ${QUANT_OPTIONS} ${UPLOAD_OPTION}"
+
+# Alternative commands
+DIRECT_CMD="mlx_lm.convert --hf-path ${HF_PATH} --mlx-path ${MLX_PATH} --dtype ${DTYPE} ${QUANT_OPTIONS} ${UPLOAD_OPTION}"
+PYTHON_CMD="python -m mlx_lm.convert --hf-path ${HF_PATH} --mlx-path ${MLX_PATH} --dtype ${DTYPE} ${QUANT_OPTIONS} ${UPLOAD_OPTION}"
+
+# Print the preview
+echo -e "\n${BOLD}${YELLOW}Command Preview:${RESET}"
+echo -e "$UV_CMD"
+
+# Expected outcomes based on options
+echo -e "\n${BOLD}${YELLOW}Expected outcomes:${RESET}"
+if [[ "$QUANTIZE" == "y" || "$QUANTIZE" == "Y" ]]; then
+  MODEL_SIZE_GB=500  # Approximate for 405B model
+  
+  case "$BITS" in
+    2)
+      EXPECTED_SIZE=$((MODEL_SIZE_GB / 8))
+      echo -e "- ${GREEN}Q2: ~${EXPECTED_SIZE}GB (from ~${MODEL_SIZE_GB}GB)${RESET}"
+      echo -e "- ${YELLOW}⚠️  Significant quality loss expected${RESET}"
+      ;;
+    3)
+      EXPECTED_SIZE=$((MODEL_SIZE_GB * 3 / 16))
+      echo -e "- ${GREEN}Q3: ~${EXPECTED_SIZE}GB (from ~${MODEL_SIZE_GB}GB)${RESET}"
+      echo -e "- ${YELLOW}Moderate quality loss${RESET}"
+      ;;
+    4)
+      EXPECTED_SIZE=$((MODEL_SIZE_GB / 4))
+      echo -e "- ${GREEN}Q4: ~${EXPECTED_SIZE}GB (from ~${MODEL_SIZE_GB}GB)${RESET}"
+      echo -e "- ${GREEN}Good balance of quality and size${RESET}"
+      ;;
+    5)
+      EXPECTED_SIZE=$((MODEL_SIZE_GB * 5 / 16))
+      echo -e "- ${GREEN}Q5: ~${EXPECTED_SIZE}GB (from ~${MODEL_SIZE_GB}GB)${RESET}"
+      echo -e "- ${GREEN}✨ Excellent quality/size ratio${RESET}"
+      ;;
+    6)
+      EXPECTED_SIZE=$((MODEL_SIZE_GB * 6 / 16))
+      echo -e "- ${GREEN}Q6: ~${EXPECTED_SIZE}GB (from ~${MODEL_SIZE_GB}GB)${RESET}"
+      echo -e "- ${GREEN}High quality preservation${RESET}"
+      ;;
+    8)
+      EXPECTED_SIZE=$((MODEL_SIZE_GB / 2))
+      echo -e "- ${GREEN}Q8: ~${EXPECTED_SIZE}GB (from ~${MODEL_SIZE_GB}GB)${RESET}"
+      echo -e "- ${GREEN}Near-lossless quality${RESET}"
+      ;;
+  esac
+  
+  if [[ -n "$QUANT_PREDICATE" ]]; then
+    echo -e "- ${CYAN}Using mixed precision: ${QUANT_PREDICATE}${RESET}"
+  fi
+  
+  if [[ "$IS_M3_ULTRA" == true ]]; then
+    echo -e "- ${MAGENTA}Expected memory usage: ${EXPECTED_SIZE}-$((EXPECTED_SIZE * 2))GB peak${RESET}"
+    echo -e "- ${MAGENTA}M3 Ultra can handle this comfortably${RESET}"
+  else
+    echo -e "- ${YELLOW}Expected memory usage: High - monitor closely${RESET}"
+  fi
+else
+  echo -e "- ${GREEN}No quantization - model remains in ${DTYPE} format${RESET}"
+  echo -e "- ${YELLOW}Very high memory requirements (400-500GB)${RESET}"
+fi
+
+echo -e "- ${CYAN}Expected conversion time: 2-6 hours${RESET}"
+
+# Ask for command format choice
+echo -e "\n${BOLD}${GREEN}Choose command format:${RESET}"
+echo -e "1. ${YELLOW}UV (recommended): ${RESET}${UV_CMD}"
+echo -e "2. ${YELLOW}Direct command: ${RESET}${DIRECT_CMD}"
+echo -e "3. ${YELLOW}Python module: ${RESET}${PYTHON_CMD}"
+read -p "> " FORMAT_CHOICE
+
+case "$FORMAT_CHOICE" in
+  2)
+    FINAL_CMD="${DIRECT_CMD}"
+    ;;
+  3)
+    FINAL_CMD="${PYTHON_CMD}"
+    ;;
+  *)
+    FINAL_CMD="${UV_CMD}"
+    ;;
+esac
+
+# M3 Ultra specific preparation tips
+if [[ "$IS_M3_ULTRA" == true ]]; then
+  echo -e "\n${BOLD}${MAGENTA}🐉 Dragon M3 Ultra Preparation:${RESET}"
+  echo -e "1. ${CYAN}Your 512GB RAM can handle even 405B models${RESET}"
+  echo -e "2. ${CYAN}Enable High Power Mode in Energy Saver${RESET}"
+  echo -e "3. ${CYAN}Consider using Activity Monitor to track memory${RESET}"
+  echo -e "4. ${CYAN}MLX will use unified memory efficiently${RESET}"
+else
+  echo -e "\n${BOLD}${BLUE}Preparation tips:${RESET}"
+  echo -e "1. ${YELLOW}Ensure Mac is plugged in and won't sleep${RESET}"
+  echo -e "2. ${YELLOW}Close other memory-intensive applications${RESET}"
+  echo -e "3. ${YELLOW}Be prepared for high fan speeds${RESET}"
+  echo -e "4. ${YELLOW}The process may appear to hang - this is normal${RESET}"
+fi
+
+# Print the final command
+echo -e "\n${BOLD}${RED}Your conversion command:${RESET}"
+echo -e "${FINAL_CMD}"
+
+# Copy to clipboard option
+echo -e "\n${BOLD}${GREEN}Copy command to clipboard? [y/n]${RESET}"
+read -p "> " COPY_CMD
+
+if [[ "$COPY_CMD" == "y" || "$COPY_CMD" == "Y" ]]; then
+    echo "${FINAL_CMD}" | pbcopy
+    echo -e "${GREEN}✓ Command copied to clipboard!${RESET}"
+fi
+
+# Download command if using remote model
+if [[ "$IS_LOCAL" == false ]]; then
+    echo -e "\n${BOLD}${CYAN}Optional: Download model first (if needed):${RESET}"
+    if [[ "$USE_HF_XET" == "y" || "$USE_HF_XET" == "Y" ]]; then
+        echo -e "# With hf-xet (10x faster):"
+        echo -e "uv run huggingface-cli download ${HF_PATH} --local-dir ./downloads/${HF_PATH##*/}"
+    else
+        echo -e "# Standard download:"
+        echo -e "uv run huggingface-cli download ${HF_PATH} --local-dir ./downloads/${HF_PATH##*/}"
+    fi
+fi
+
+# Test commands
+echo -e "\n${BOLD}${BLUE}After conversion, test with:${RESET}"
+echo -e "uv run mlx_lm.generate --model ${MLX_PATH} --prompt \"Hello, I am\" --max-tokens 50"
+
+# Memory monitoring for M3 Ultra
+if [[ "$IS_M3_ULTRA" == true ]]; then
+  echo -e "\n${BOLD}${MAGENTA}Monitor Dragon performance:${RESET}"
+  echo -e "uv run python -c \"import mlx.core as mx; print(f'Peak: {mx.metal.get_peak_memory()/1e9:.2f}GB of ${TOTAL_MEMORY_GB}GB')\""
+  
+  echo -e "\n${BOLD}${CYAN}Pro tip for large models:${RESET}"
+  echo -e "# Set memory limit before conversion (optional):"
+  echo -e "export MLX_METAL_MEMORY_LIMIT=$((TOTAL_MEMORY_GB * 95 / 100))GB"
+fi
+
+# Benchmark command
+echo -e "\n${BOLD}${CYAN}Benchmark the converted model:${RESET}"
+echo -e "uv run mlx_lm.generate --model ${MLX_PATH} --prompt \"The\" --max-tokens 100 --verbose"
+
+echo -e "\n${BOLD}${BLUE}=====================================${RESET}"
+echo -e "${BOLD}${GREEN}✨ Conversion setup complete!${RESET}"
+if [[ "$IS_M3_ULTRA" == true ]]; then
+  echo -e "${BOLD}${MAGENTA}🐉 Dragon M3 Ultra ready to roar!${RESET}"
+fi
+echo -e "${BOLD}${BLUE}=====================================${RESET}"

mlx-serve.sh

@@ -0,0 +1,229 @@
+#!/bin/bash
+# MLX Server Launcher for Dragon M3 Ultra
+# Created: January 2025 for MLX 0.26+
+# Supports local/remote models with full parameter control
+
+# Text formatting
+BOLD="\033[1m"
+BLUE="\033[34m"
+GREEN="\033[32m"
+YELLOW="\033[33m"
+RED="\033[31m"
+CYAN="\033[36m"
+MAGENTA="\033[35m"
+RESET="\033[0m"
+
+# Detect system specs
+TOTAL_MEMORY=$(sysctl -n hw.memsize 2>/dev/null || echo 0)
+TOTAL_MEMORY_GB=$((TOTAL_MEMORY / 1073741824))
+CPU_BRAND=$(sysctl -n machdep.cpu.brand_string 2>/dev/null || echo "Unknown")
+
+# Check if running on M3 Ultra
+if [[ "$CPU_BRAND" == *"M3 Ultra"* ]] || [[ "$TOTAL_MEMORY_GB" -ge 400 ]]; then
+    IS_M3_ULTRA=true
+    echo -e "${BOLD}${MAGENTA}🐉 Dragon M3 Ultra detected! (${TOTAL_MEMORY_GB}GB RAM)${RESET}"
+else
+    IS_M3_ULTRA=false
+fi
+
+echo -e "${BOLD}${BLUE}=====================================${RESET}"
+echo -e "${BOLD}${BLUE}   MLX Server Launcher v1.0          ${RESET}"
+echo -e "${BOLD}${BLUE}=====================================${RESET}"
+echo -e "Launch MLX model server with custom parameters\n"
+
+# Default values
+DEFAULT_MODEL="/Users/polyversai/.lmstudio/models/LibraxisAI/c4ai-command-a-03-2025-q5-mlx"
+DEFAULT_HOST="0.0.0.0"
+DEFAULT_PORT="12345"
+DEFAULT_TEMP="0.7"
+DEFAULT_TOP_P="0.95"
+DEFAULT_TOP_K="0"
+DEFAULT_MIN_P="0.0"
+DEFAULT_MAX_TOKENS="2048"
+DEFAULT_LOG_LEVEL="INFO"
+
+# Get model path
+echo -e "${BOLD}Model path (local or HF repo):${RESET}"
+echo -e "(Default: ${DEFAULT_MODEL})"
+echo -e "${CYAN}Examples:${RESET}"
+echo -e "  Local: /Users/polyversai/.lmstudio/models/mlx-community/model-name"
+echo -e "  HF: mlx-community/Llama-3.2-3B-Instruct-4bit"
+read -p "> " MODEL_PATH
+MODEL_PATH=${MODEL_PATH:-$DEFAULT_MODEL}
+
+# Check if it's a local path
+if [[ -d "$MODEL_PATH" ]]; then
+    echo -e "${GREEN}✓ Local model detected: ${MODEL_PATH}${RESET}"
+else
+    echo -e "${GREEN}✓ Remote model specified: ${MODEL_PATH}${RESET}"
+fi
+
+# Network configuration
+echo -e "\n${BOLD}Host IP address:${RESET}"
+echo -e "(Default: ${DEFAULT_HOST} - accessible from network)"
+echo -e "Use 127.0.0.1 for localhost only"
+read -p "> " HOST
+HOST=${HOST:-$DEFAULT_HOST}
+
+echo -e "\n${BOLD}Port number:${RESET}"
+echo -e "(Default: ${DEFAULT_PORT})"
+read -p "> " PORT
+PORT=${PORT:-$DEFAULT_PORT}
+
+# Sampling parameters
+echo -e "\n${BOLD}${CYAN}=== Sampling Parameters ===${RESET}"
+
+echo -e "\n${BOLD}Temperature (creativity):${RESET}"
+echo -e "Range: 0.0-2.0 (Default: ${DEFAULT_TEMP})"
+echo -e "${YELLOW}0.0 = deterministic, 1.0 = balanced, 2.0 = very creative${RESET}"
+read -p "> " TEMP
+TEMP=${TEMP:-$DEFAULT_TEMP}
+
+echo -e "\n${BOLD}Top-p (nucleus sampling):${RESET}"
+echo -e "Range: 0.0-1.0 (Default: ${DEFAULT_TOP_P})"
+echo -e "${YELLOW}Lower = more focused, Higher = more diverse${RESET}"
+read -p "> " TOP_P
+TOP_P=${TOP_P:-$DEFAULT_TOP_P}
+
+echo -e "\n${BOLD}Top-k (vocabulary limit):${RESET}"
+echo -e "Default: ${DEFAULT_TOP_K} (0 = disabled)"
+echo -e "${YELLOW}Limits selection to top K tokens${RESET}"
+read -p "> " TOP_K
+TOP_K=${TOP_K:-$DEFAULT_TOP_K}
+
+echo -e "\n${BOLD}Min-p (minimum probability):${RESET}"
+echo -e "Range: 0.0-1.0 (Default: ${DEFAULT_MIN_P})"
+echo -e "${YELLOW}0.0 = disabled, higher = filter low probability tokens${RESET}"
+read -p "> " MIN_P
+MIN_P=${MIN_P:-$DEFAULT_MIN_P}
+
+echo -e "\n${BOLD}Max tokens per response:${RESET}"
+echo -e "(Default: ${DEFAULT_MAX_TOKENS})"
+if [[ "$IS_M3_ULTRA" == true ]]; then
+    echo -e "${MAGENTA}Dragon can handle 8192+ tokens easily${RESET}"
+fi
+read -p "> " MAX_TOKENS
+MAX_TOKENS=${MAX_TOKENS:-$DEFAULT_MAX_TOKENS}
+
+# Optional adapter
+echo -e "\n${BOLD}LoRA adapter path (optional):${RESET}"
+echo -e "(Leave empty if not using adapters)"
+read -p "> " ADAPTER_PATH
+
+if [[ -n "$ADAPTER_PATH" ]]; then
+    ADAPTER_OPTION="--adapter-path ${ADAPTER_PATH}"
+else
+    ADAPTER_OPTION=""
+fi
+
+# Chat template args
+echo -e "\n${BOLD}Chat template args (optional JSON):${RESET}"
+echo -e "Example: {\"enable_thinking\":false}"
+echo -e "(Leave empty for defaults)"
+read -p "> " CHAT_TEMPLATE_ARGS
+
+if [[ -n "$CHAT_TEMPLATE_ARGS" ]]; then
+    CHAT_TEMPLATE_OPTION="--chat-template-args \"${CHAT_TEMPLATE_ARGS}\""
+else
+    CHAT_TEMPLATE_OPTION=""
+fi
+
+# Log level
+echo -e "\n${BOLD}Log level:${RESET}"
+echo -e "(Default: ${DEFAULT_LOG_LEVEL}, Options: DEBUG, INFO, WARNING, ERROR, CRITICAL)"
+read -p "> " LOG_LEVEL
+LOG_LEVEL=${LOG_LEVEL:-$DEFAULT_LOG_LEVEL}
+
+# Build the command
+SERVER_CMD="uv run mlx_lm.server --model ${MODEL_PATH} --host ${HOST} --port ${PORT} --temp ${TEMP} --top-p ${TOP_P} --top-k ${TOP_K} --min-p ${MIN_P} --max-tokens ${MAX_TOKENS} --log-level ${LOG_LEVEL} ${ADAPTER_OPTION} ${CHAT_TEMPLATE_OPTION}"
+
+# Print preview
+echo -e "\n${BOLD}${YELLOW}Command Preview:${RESET}"
+echo -e "$SERVER_CMD"
+
+# Launch mode selection
+echo -e "\n${BOLD}${GREEN}Launch mode:${RESET}"
+echo -e "1. ${YELLOW}Foreground${RESET} - See logs in terminal (Ctrl+C to stop)"
+echo -e "2. ${YELLOW}Background with logging${RESET} - Logs to mlx-server.log"
+echo -e "3. ${YELLOW}Background detached${RESET} - Run with nohup"
+echo -e "4. ${YELLOW}Just copy command${RESET} - Don't launch"
+read -p "> " LAUNCH_MODE
+
+# Create logs directory if needed
+if [[ "$LAUNCH_MODE" == "2" || "$LAUNCH_MODE" == "3" ]]; then
+    mkdir -p logs
+    LOG_FILE="logs/mlx-server-$(date +%Y%m%d-%H%M%S).log"
+fi
+
+case "$LAUNCH_MODE" in
+    1)
+        echo -e "\n${BOLD}${GREEN}Starting server in foreground...${RESET}"
+        echo -e "${YELLOW}Press Ctrl+C to stop${RESET}\n"
+        eval "$SERVER_CMD"
+        ;;
+    2)
+        echo -e "\n${BOLD}${GREEN}Starting server in background...${RESET}"
+        echo -e "Logs: ${LOG_FILE}"
+        eval "$SERVER_CMD" > "${LOG_FILE}" 2>&1 &
+        SERVER_PID=$!
+        echo -e "${GREEN}✓ Server started with PID: ${SERVER_PID}${RESET}"
+        echo -e "\nTo monitor: tail -f ${LOG_FILE}"
+        echo -e "To stop: kill ${SERVER_PID}"
+        
+        # Save PID for easy stopping
+        echo $SERVER_PID > logs/mlx-server.pid
+        ;;
+    3)
+        echo -e "\n${BOLD}${GREEN}Starting server with nohup...${RESET}"
+        echo -e "Logs: ${LOG_FILE}"
+        nohup bash -c "$SERVER_CMD" > "${LOG_FILE}" 2>&1 &
+        SERVER_PID=$!
+        echo -e "${GREEN}✓ Server started with PID: ${SERVER_PID}${RESET}"
+        echo -e "\nTo monitor: tail -f ${LOG_FILE}"
+        echo -e "To stop: kill ${SERVER_PID}"
+        
+        # Save PID
+        echo $SERVER_PID > logs/mlx-server.pid
+        ;;
+    4)
+        echo -e "\n${BOLD}${GREEN}Command copied to clipboard!${RESET}"
+        echo "$SERVER_CMD" | pbcopy
+        ;;
+    *)
+        echo -e "\n${RED}Invalid choice. Exiting.${RESET}"
+        exit 1
+        ;;
+esac
+
+# Print API examples
+if [[ "$LAUNCH_MODE" != "4" ]]; then
+    echo -e "\n${BOLD}${BLUE}=== API Usage Examples ===${RESET}"
+    
+    echo -e "\n${CYAN}1. Chat completion:${RESET}"
+    echo -e "curl http://${HOST}:${PORT}/v1/chat/completions \\"
+    echo -e "  -H \"Content-Type: application/json\" \\"
+    echo -e "  -d '{"
+    echo -e "    \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}],"
+    echo -e "    \"temperature\": ${TEMP},"
+    echo -e "    \"max_tokens\": 100"
+    echo -e "  }'"
+    
+    echo -e "\n${CYAN}2. Check models:${RESET}"
+    echo -e "curl http://${HOST}:${PORT}/v1/models"
+    
+    echo -e "\n${CYAN}3. Health check:${RESET}"
+    echo -e "curl http://${HOST}:${PORT}/health"
+    
+    if [[ "$IS_M3_ULTRA" == true ]]; then
+        echo -e "\n${BOLD}${MAGENTA}Dragon Performance Monitoring:${RESET}"
+        echo -e "# In another terminal:"
+        echo -e "watch -n 1 'curl -s http://${HOST}:${PORT}/health | jq .'"
+    fi
+fi
+
+echo -e "\n${BOLD}${BLUE}=====================================${RESET}"
+echo -e "${BOLD}${GREEN}✨ MLX Server ready!${RESET}"
+if [[ "$IS_M3_ULTRA" == true ]]; then
+    echo -e "${BOLD}${MAGENTA}🐉 Dragon M3 Ultra serving at full power!${RESET}"
+fi
+echo -e "${BOLD}${BLUE}=====================================${RESET}"

model-00001-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7a6e9d43fd37774057c494a72c0c8419d6a3db1c8e7269a02063f4f1b5092f90
+size 5364994599

model-00002-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a0ac4bf7a6d38f358784207acedb5c0882957f4561d85dac25520d5a89c0e295
+size 5368494562

model-00003-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:58dedd01147ade3b33c86640e21065711fb78b76d1fb2c86438bb983d239b28e
+size 5364070790

model-00004-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:82f537ee2ac8ff8a8329160ab6d6376bd4bf506ff4c6f18581978c19bd47757a
+size 5364070782

model-00005-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dae63165d5c54c668699e969c8c6cb69ba152a55307f1c713decacd0eb301260
+size 1065193577

special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aeb13307a71acd8fe81861d94ad54ab689df773318809eed3cbe794b4492dae4
+size 11422654

tokenizer_config.json

@@ -0,0 +1,239 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}