Bartowski Shrinks Qwen2.5-32B Model Down to 9GB

AI developer Bartowski has published a complete ladder of 27 imatrix GGUF quantizations for the Qwen2.5-32B-Instruct artificial intelligence model, shrinking the original 32.5-billion-parameter release into files starting as small as 9GB for the IQ2_XXS build. The quantization suite extends up to roughly 65GB for full F16 sharded weights. For everyday deployment, the Q4_K_M build serves as the default recommendation, requiring 19.85GB of storage and easily fitting onto standard 24GB consumer GPUs when utilizing moderate context sizes.

Distributed under a flexible Apache 2.0 license, the 64-layer base transformer incorporates grouped-query attention, SwiGLU feed-forward networks, RMSNorm, QKV bias, and RoPE positional encodings. It demonstrates high accuracy across complex mathematical reasoning, software development, structured data handling, and JSON output generation across more than 29 languages. The system accepts inputs formatted with Qwen ChatML tokens <|im_start|> and <|im_end|>, supporting context windows up to 131,072 tokens and generating up to 8,192 tokens per response, though lengths beyond 32,768 tokens necessitate YaRN context scaling.

To construct the compressed files without severe quality loss, Bartowski applied the llama.cpp importance-matrix method to evaluate weight significance during quantization. Specific Q4_0_X_X builds offer targeted speedups for ARM CPUs leveraging SVE or i8mm vector instructions. Additionally, developers focused heavily on software engineering can elect to run the Qwen2.5-Coder-32B GGUF variant, which achieves performance on par with GPT-4o across code benchmarks.

These models are designed for fast one-line setup through various local execution tools including llama.cpp, Ollama, LM Studio, Jan, vLLM, and Docker Model Runner. Early community uptake has been significant, with the repository accumulating approximately 800,000 downloads and 3,600 likes on Hugging Face.