Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE

What happened: A paper proposes Meta Expert, a cross-layer expert sharing method for mixture-of-experts models, and reports an empirical comparison against standard Local Expert designs. Key numbers: Meta Expert models match Vanilla MoE performance using only 70 to 80 percent of the experts, or 75 to 90 percent of the parameters. Why it matters: The authors report lower parameter redundancy, lower cross-entropy loss, and higher downstream task accuracy for Meta Expert models, and argue the approach improves both information storage and retrieval efficiency.

A paper proposes Meta Expert, a cross-layer expert sharing method for mixture-of-experts models, and reports an empirical comparison against standard Local Expert designs.

The authors report lower parameter redundancy, lower cross-entropy loss, and higher downstream task accuracy for Meta Expert models, and argue the approach improves both information storage and retrieval efficiency.