FreeToken Runs Massive MoE Models on Consumer Hardware

Researchers from UC Berkeley and MIT have released FreeToken, an open-source inference engine designed to run massive Mixture-of-Experts (MoE) models on consumer-grade hardware. Co-authored by prominent computer scientists including Matei Zaharia, Ion Stoica, Song Han, and Kurt Keutzer, the project shifts local AI execution away from static offloading. Instead of treating personal computers as limited datacenter nodes, FreeToken coordinates heterogeneous consumer hardware to handle massive weight-routing demands.

To overcome the bottlenecks of consumer PCIe lanes, which typically run at 16 to 64 GB/s, FreeToken introduces a dynamic co-scheduling system called the q* policy. Rather than stalling the GPU during cache misses, this policy splits token computation between CPU and GPU tensor cores in real time. The engine utilizes a fast weight format (FTW) and full-layer double buffering to overlap weight streaming with active computation. Additionally, an elastic memory manager dynamically reallocates VRAM between KV cache entries and resident experts, while semantic anchor checkpointing preserves intermediate attention states during agentic workflows.

In benchmark tests, FreeToken achieved impressive speeds on consumer setups. It ran Qwen3.6-35B at approximately 39 tokens per second on an 8GB RTX 4060 laptop, served the 284-billion-parameter DeepSeek-V4-Flash on an RTX 5090 desktop, and processed the 753-billion-parameter GLM-5.2 on a single workstation GPU. Compared to popular runtimes like Ollama and llama.cpp, FreeToken delivers three to four times faster decoding and six to 30 times faster prefill speeds on equivalent MoE models.

For developers and practitioners, this release represents a major step toward local hardware sovereignty. By pairing affordable consumer memory, such as used RTX 3090 or 4080 GPUs with standard DDR4 or DDR5 RAM, developers can self-host frontier-class reasoning agents. This setup eliminates recurring cloud API fees, cuts agent iteration costs to zero, and secures intellectual property. The CLI and desktop client are currently available via FlashML.ai and GitHub, supporting NVIDIA RTX 30, 40, and 50 series GPUs on Windows and Linux.