JudgeMyAIAttention-Aware Routing: How Coupling Router and Attention in MoEs Slashes Diverging Generations by 42%
Deep dive into Attention-Aware Routing (AAR) for MoEs: How coupling routing and attention improves GSM8K by +3.37pp, reduces long diverging generations, and forms a coupled residual circuit.