FreeToken Runs 753B GLM-5.2 on One Workstation GPU
UC Berkeley and UT Austin researchers have launched FreeToken, an open-source engine that runs massive AI models locally on consumer hardware to slash developer API costs.
The new edge-native Mixture-of-Experts (MoE) serving engine, FreeToken, allows developers to run frontier open-weight models on local workstations and laptops. Developed by researchers at UC Berkeley and UT Austin, the system can run the 753-billion-parameter GLM-5.2 model at 14.9 tokens per second on a single RTX PRO 6000 workstation GPU, compared to just 7.3 tokens per second using llama.cpp. On an RTX 5090, FreeToken sustains 77 to 83 tokens per second on Qwen3.6-35B-A3B in BF16 precision, and 22 to 25 tokens per second on the 284-billion-parameter DeepSeek-V4-Flash in MXFP4. Even on an 8 GB RTX 4060 laptop GPU, it serves a 35-billion-parameter model at 39.3 tokens per second using NVFP4.
To achieve these speeds, FreeToken addresses the hardware bottlenecks that typically choke local MoE inference, such as slow PCIe lanes and limited GPU memory. It introduces a bandwidth-adaptive execution policy, called the q-star policy, which profiles the system's PCIe and host memory bandwidths. The engine then splits expert cache misses, filling some into the GPU cache while computing the rest directly on the CPU. It also uses semantic-aware caching, which double-buffers layers during prefill and anchors recurrent-state checkpoints at special-token boundaries. An elastic memory management system rebuilds the GPU expert cache under revised VRAM budgets without needing to restart the engine.
These optimizations yield a 1.5-fold to 2.3-fold decode throughput speedup over existing engines like llama.cpp, Ollama, and KTransformers. FreeToken keeps worst-case time-to-first-token (TTFT) below 44 seconds, whereas llama.cpp, Ollama, and KTransformers can spike to 232, 179, and 946 seconds respectively. At an equal cache capacity of 37 percent of the Qwen3.6 pool, FreeToken's global Least Recently Used (LRU) cache misses only 16 percent of decode-time expert reads, compared to 41 percent for KTransformers and 62 percent for llama.cpp.
For practitioners, FreeToken offers an air-gapped, cost-effective path for handling sensitive data in legal, healthcare, and finance sectors. However, running these massive models still demands significant host memory; the 753-billion-parameter GLM-5.2 setup requires 512 GiB of DDR5 DRAM on a Xeon Platinum 8559C host, while the 284-billion-parameter desktop setup requires 192 GB. FreeToken is licensed under Apache-2.0 and is available on PyPI as freetoken v0.1.2, requiring an NVIDIA GPU on driver r580 or higher with CUDA 13. It exposes OpenAI- and Anthropic-compatible endpoints on port 1919.
This is our own summary of reporting by MarkTechPost
