Yandex Releases AliceAI 80B Open-Source Model
Russian tech giant Yandex has open-sourced AliceAI-Foundation-80B-A3B-Base, a highly efficient mixture-of-experts model that outperforms rivals on complex mathematics benchmarks.
Yandex has released the weights for AliceAI-Foundation-80B-A3B-Base, a bilingual Russian-English mixture-of-experts model built entirely from scratch. Distributed under the permissive Apache 2.0 license on Hugging Face, the model features 80 billion total parameters but activates only about 3 billion parameters per token. This sparse execution represents an active-to-total parameter ratio of roughly 3.75 percent, which significantly reduces the computational load per token. The model also boasts an expansive 262,144-token context window, allowing it to process massive documents.
The architecture departs from standard transformer designs by repeating a unique four-layer pattern 12 times across its 48 layers, which have a hidden size of 2,048. Each cycle interleaves three KDA linear attention blocks with one Gated Attention block, attaching a mixture-of-experts feed-forward module to every layer. The KDA operator utilizes a convolution kernel size of 4, alongside 32 query heads and 32 key-value heads, to handle long sequences efficiently. Meanwhile, the Gated Attention layers employ 16 query heads and 2 key-value heads with a head dimension of 256. For the mixture-of-experts component, Yandex uses 512 small experts with an intermediate dimension of 512, routing 10 selected experts and one shared expert per token.
In evaluations, the base model achieved a score of 91.1 on the MATH-500 benchmark, outperforming larger mixture-of-experts competitors on mathematical and factual reasoning tests like HMMT, IMO Answerbench, and Russian factual benchmarks. For practitioners, this release offers a powerful, commercially viable foundation model that does not rely on Llama or Qwen initializations. However, deploying the model requires substantial hardware, as the full BF16 checkpoint spans 49 safetensors files and requires approximately 163 GB of storage before accounting for runtime overhead. Because this is a base model rather than a chat-tuned variant, Yandex has packaged it with a LoRA fine-tuning example, a vLLM Docker path, and multi-token prediction support to facilitate speculative decoding.
This is our own summary of reporting by AlphaSignal
