Multiverse Prunes Llama-3.3-70B Using Physics Solvers
Researchers at Multiverse have developed a physics-inspired method to prune large language models by treating block removal as an Ising optimization problem, boosting compressed model accuracy.
Multiverse has introduced a new depth-pruning technique called Constrained Binary Optimization (CBO) that treats the removal of transformer blocks as a physics problem. Detailed in their paper, LLM Compression by Block Removal with Constrained Binary Optimization, the method maps block selection to an Ising glass, which is a disordered spin system. By calculating a second-order Taylor expansion of the model's loss, the system creates a Hessian matrix where diagonal entries represent individual block importance and off-diagonal entries capture the pairwise couplings between blocks. This allows the system to evaluate billions of configurations cheaply without running or benchmarking the actual models.
The approach shows massive advantages in deep-compression scenarios. When compressing Llama-3.3-70B-Instruct by 50 percent by removing 40 of its 80 blocks, the CBO method maintained an MMLU score near 77, while the strongest baseline fell to the mid-50s—a gap of nearly 23 percentage points. For Qwen3-14B with 12 of its 40 blocks removed, the method led the MMLU benchmark by approximately 10 points. The researchers also tested the method on Llama-3.1-8B-Instruct with 16 of 32 blocks removed. Interestingly, the 17th excited state, which proposed removing an early block, outperformed the ground state on several benchmarks after light retraining.
Computing the Hessian requires only a single run on a small calibration dataset. For smaller search spaces, the team can brute-force the configurations on a single GPU; for example, finding the optimal way to remove 8 of Llama-3.3-70B's 80 blocks involved checking 29 billion configurations and took about two days. For larger spaces, they use open-source tabu solvers to find low-energy states in seconds. To test the method on heterogeneous architectures, they applied it to NVIDIA-Nemotron-3-Nano-30B-A3B-FP8, a hybrid model combining Mamba2, attention, and mixture-of-experts (MoE) layers. Removing two to three MoE layers or two attention layers yielded configurations that beat block influence baselines on the AIME25 and GPQA benchmarks. The code has been open-sourced on GitHub.
This is our own summary of reporting by Hugging Face Blog
