E = T*H/(O+B): A Dimensionless Control Parameter for Mixture-of-Experts Ecology

Problem
This preprint addresses a significant gap in the understanding of Mixture-of-Experts (MoE) models, specifically regarding the conditions that lead to a healthy expert ecology versus the collapse into dead experts. Prior literature lacks a unified framework to predict expert performance based on hyperparameter settings, which can lead to inefficient training and suboptimal model performance. The authors propose a dimensionless control parameter, E = T*H/(O+B), that encapsulates the interplay of routing temperature (T), routing entropy weight (H), oracle weight (O), and balance weight (B) to predict expert behavior.

Method
The core technical contribution is the introduction of the dimensionless parameter E, which is derived from four critical hyperparameters in MoE models. The authors conducted 12 controlled experiments across vision (CIFAR-10, CIFAR-100, TinyImageNet-200) and language (WikiText-2, WikiText-103) tasks, totaling over 11,000 training epochs. The experiments demonstrate that maintaining E >= 0.5 is sufficient to ensure zero dead experts, thus eliminating the need for additional load-balancing auxiliary losses. The authors also explore various phenomena related to expert ecology, such as the resuscitation of dead experts and the impact of task complexity on the critical E threshold.

Results
The findings indicate that E >= 0.5 guarantees a healthy expert ecology, with no dead experts observed in the experiments. The authors report that dead experts can be revived through balance loss, which encourages router re-exploration. They also find that the critical E threshold is influenced by task complexity, and that model overfitting does not correlate with expert health. Notably, a three-tier MoE structure tends to collapse into a two-tier system, and ecological structures remain temperature-invariant across a 50x range. These results provide a robust framework for understanding and diagnosing MoE training dynamics.

Limitations
The authors acknowledge that their findings are based on specific datasets and tasks, which may limit generalizability. They do not address the potential impact of varying model architectures or the scalability of their approach to larger, more complex datasets. Additionally, the experiments do not explore the long-term implications of maintaining E >= 0.5 in diverse real-world applications.

Why it matters
This work has significant implications for the design and training of MoE models, providing a systematic approach to prevent dead experts and enhance model efficiency. By establishing E as a diagnostic tool analogous to the Reynolds number in fluid dynamics, the authors pave the way for future research to optimize hyperparameter settings in MoE architectures. This could lead to more robust and scalable models in both vision and language tasks, ultimately advancing the state of the art in machine learning.

Authors: Qingjun Zhang
Source: arXiv:2605.06415
URL: https://arxiv.org/abs/2605.06415v1