randmaru/zeta-2.1-mlx-mxfp4

This is an MXFP4 MLX quantization of zed-industries/zeta-2.1 for Apple Silicon inference.

MXFP4 vs 4Bit Quantization Comparison

Parameter	MXFP4	4Bit
Quantization format	4‑bit floating point with microscaling, group 32, shared exponent E8M0	4‑bit integer (INT4/NF4)
Tensor types	U8, U32, BF16	BF16, U32
Parameter size (safetensors)	~1.55 GB (1,547,177,984 bytes)	~1.29 GB (1,289,359,360 bytes)
Total storage (all files)	~4.40 GB (4,395,399,599 bytes)	~4.65 GB (4,653,244,288 bytes)
Hardware support	Most efficient on GPUs with microscaling / FP8 tensor core support	Broad support, but often requires specialized INT4 kernels
Apple Silicon compatibility	Designed with hardware microscaling support in Apple Neural Engine / GPU	Works, but without specialized Neural Engine optimization
Inference speed	Higher on compatible hardware: FP path, lower dequantization overhead, higher throughput	Kernel‑dependent; usually lower or comparable at similar quality
Quality	Better preserves dynamic range, less degradation on outliers	Higher risk of accuracy loss on outliers at the same bitrate

Key takeaways:

• Parameter size: The MXFP4 version has a slightly larger safetensors.total (~1.55 GB vs ~1.29 GB). This is because it stores weights in floating‑point format with additional metadata (U8/U32/BF16), while the integer version uses a more compact representation (BF16/U32).
• Total storage: Despite the larger parameter size, the MXFP4 version occupies less disk space (~4.40 GB vs ~4.65 GB). This suggests more efficient packing of auxiliary files and/or less duplication.
• Performance: MXFP4 typically delivers higher inference throughput (tokens/s) on hardware with microscaling/FP8 support, especially on Apple Silicon. The shared exponent per group of 32 elements reduces dequantization overhead and enables the use of floating‑point tensor cores, which can be faster than conventional INT4 kernels.
• Apple Silicon optimization: The MXFP4 format was developed with the hardware capabilities of Apple chips (M‑series) in mind, particularly the Neural Engine, which natively supports microscaling. Therefore, MXFP4 is the recommended choice for MacBook and offers the best balance of speed and quality on these devices.
• Quality: MXFP4 preserves the dynamic range of weights better, so generation quality can be higher at the same compression level.

Note: Actual speed depends on the backend, GPU, batch size, and quantization implementation.