Kakao Corp Cuts MoE Training Costs With New Scaling Law

Researchers from Kakao Corp have introduced a compute-efficient framework designed to streamline the training of large-scale Mixture-of-Experts (MoE) models. Traditionally, finding the optimal hyperparameters for these massive architectures requires running expensive sweeps over trillions of tokens, which is financially and computationally prohibitive for most organizations. The new two-step method bypasses this bottleneck by predicting the ideal learning rate using small-scale proxy runs.

The first phase of the framework adapts Maximal Update Parameterization, or μP, for MoE architectures that incorporate Multi-head Latent Attention (MLA) and the Muon optimizer. This adaptation ensures that optimal learning rates transfer consistently as the model width scales up. The second phase extends this transferability across the token dimension. By applying linear regression to data from short-budget proxy models, the researchers established a predictive scaling law that projects learning rates onto massive training horizons with a high fidelity of 0.95 R-squared.

To prove the effectiveness of their methodology, the Kakao Corp team successfully pretrained a foundation model from scratch. The model features 155 billion total parameters, with 17 billion active parameters, and was trained over a massive horizon of 10 trillion tokens. The stable training run and subsequent evaluations confirmed that the framework accurately predicted the optimal configurations for the full-scale target model while incurring only minimal ablation costs.

For machine learning practitioners, this development represents a major shift in how large-scale MoE models are optimized. Instead of wasting vast computational budgets on trial-and-error hyperparameter tuning at scale, developers can now rely on cheap, small-scale proxy models to confidently determine training configurations. This drastically lowers the barrier to entry for training state-of-the-art MoE systems.