mixture-of-experts

A deep tour of the architectural choices powering today's large language models — decoder-only Transformers, encoder-decoder designs, grouped-query attention, RoPE positional embeddings, and mixture-of-experts routing — with concrete numbers and trade-offs.