HyperQwen Runs Qwen3.8-27B on a Single RTX 3090

The open-source HyperQwen project has introduced a method to run the 27-billion-parameter Qwen3.8-27B model on a single 24GB Nvidia RTX 3090 graphics card. Typically, serving a model of this size on consumer hardware is difficult because the model weights, temporary activations, and key-value cache compete for the limited onboard memory. HyperQwen overcomes this bottleneck by combining a patched vLLM stack with a custom requantization pipeline that compresses the model checkpoint into lower-precision formats.

In benchmark tests conducted on an RTX 3090 capped at a 250-watt power limit, the system demonstrated high throughput. Under a heavy workload of 64 concurrent requests, HyperQwen achieved an aggregate decode speed of approximately 1,035 tokens per second. For single-user scenarios, the setup clocked about 127 tokens per second, while standard open-ended chat reached 133 tokens per second. When reproducing prompt text with a 56k context, the speed rose to 381 tokens per second.

The system utilizes a specialized KVarN 4/2-bit KV cache backend, allowing it to support massive context lengths ranging from 150k to 262k tokens. Performance scales with the context size, yielding 95 to 100 tokens per second at a 150k context and 67 tokens per second for mixed generation at a 240k context. Additionally, HyperQwen features a prefix cache that dramatically reduces the time-to-first-token from 22 seconds down to 0.56 seconds when users ask repeated questions about the same document.

Developers can deploy the system using two pre-configured Docker Compose profiles, which cater to either a single-user chat mode or a high-throughput batch mode. Released under the Apache 2.0 license, the project provides an OpenAI-compatible API and includes a fully documented 38-file vLLM patch series. This release offers practitioners a highly efficient way to serve large models locally without relying on expensive enterprise-grade hardware.