Modern LLM Architectures: From Decoder-Only to Mixture-of-Experts

Every major LLM traces back to one of two design choices made in 2017–2018. The original "Attention Is All You Need" paper introduced a full encoder-decoder stack — an encoder that reads all input tokens in parallel and a decoder that generates output tokens one at a time. BERT took just the encoder half for classification and span-prediction tasks. GPT took just the decoder half and showed you could do almost everything with next-token prediction.

Today, the vast majority of capable open-weight and proprietary models — Llama 3, Mistral, GPT-4, Gemini Ultra, DeepSeek-V3 — are decoder-only. Encoder-decoder models (T5, Flan-T5, Gemini's early versions) still dominate translation, summarization, and tasks where you always have a fixed input before generation begins.

The lineage matters because it dictates the attention mask, training objective, KV-cache shape, and even how you fine-tune.

A decoder-only model is a stack of identical layers. Each layer has two sub-layers: multi-head causal self-attention (masked so token $i$ can only attend to tokens $\leq i$) and a feed-forward network (FFN). Between them are residual connections and layer norm.

At inference time, the model reads a prompt in one forward pass — all prompt tokens are processed in parallel (this is the "prefill" phase). Then it autoregressively generates one token per step (the "decode" phase). Each decode step is cheap but sequential, which is why long-context generation is slow.

The FFN is the biggest parameter consumer. In a standard Llama 3-8B layer, the FFN has three matrices (gate, up, down) with hidden dimension 14336 versus attention dimension 4096 — FFN is about 3x wider. Across 32 layers, FFN holds roughly 60-65% of total parameters.

T5 ("Text-to-Text Transfer Transformer", Google 2020) reframes every NLP task as a string-in / string-out problem: "translate English to French: The cat sat" → "Le chat s'est assis". The encoder processes the full input with bidirectional attention (every token sees every other), producing a rich context representation. The decoder generates output with cross-attention into those encoder states.

Why does bidirectionality matter? For tasks like summarization or translation, the model benefits from seeing the whole input before generating any output — the encoder gets to "think" about the full document. A decoder-only model must compress that understanding into its prefix representations without explicit bidirectional encoding.

Trade-off summary:

Standard multi-head attention (MHA) gives every head its own key and value matrices. With $H$ heads, the KV cache grows as $O(H \cdot L \cdot d_k)$ in sequence length $L$ — at 128 heads and 128k context this becomes a memory wall.

Multi-query attention (MQA, Shazeer 2019) collapses all K and V heads into one shared pair. Much smaller cache, but can hurt quality on complex tasks. Grouped-query attention (GQA, Ainslie et al. 2023) is the compromise: queries are still split into $H$ heads, but K and V are split into $G$ groups ($G < H$), each group shared by $H/G$ query heads.

Llama 3-8B uses 8 KV heads for 32 query heads (a 4x compression). Mistral 7B uses 8 KV heads for 32 query heads too. At fp16, a 128k-token cache for Llama 3-8B with GQA takes ~4 GB versus ~16 GB for full MHA — that difference is what makes long-context inference feasible on a single GPU.

# Pseudocode for GQA in a forward pass
# Q: [batch, seq, n_heads, head_dim]
# K, V: [batch, seq, n_kv_heads, head_dim]
n_rep = n_heads // n_kv_heads          # e.g. 4
K = K.repeat_interleave(n_rep, dim=2)  # expand to full head count
V = V.repeat_interleave(n_rep, dim=2)
attn = scaled_dot_product_attention(Q, K, V, is_causal=True)

Classic Transformers inject position via learned or sinusoidal embeddings added to the input — they don't generalize cleanly beyond the training context length. RoPE (Su et al. 2021) encodes position differently: instead of adding a vector, it rotates the query and key vectors by an angle that depends on position.

For a token at position $m$, each pair of dimensions $(i, i+1)$ in the head is rotated by $m \cdot \theta_i$ where $\theta_i = 10000^{-2i/d}$. The inner product $Q_m \cdot K_n$ then depends only on the relative offset $m - n$, not absolute positions.

$\text{RoPE}(\mathbf{x}, m) = \mathbf{x} \cdot e^{im\theta}$

This gives two practical wins: (1) the model learns relative rather than absolute positions, which generalizes better; (2) you can extend context at inference by scaling $\theta$ ("RoPE scaling" or "YaRN"), letting a model trained on 8k tokens handle 128k with minimal quality loss. Llama 2 used 4k context; Llama 3 uses 8k base with YaRN scaling to 128k.

In a dense model, every token passes through every FFN on every layer. MoE replaces the single FFN with $N$ "expert" FFNs and a router that selects $k$ of them per token (typically $k = 2$ out of 8–64 experts). Only the selected experts execute — the rest are idle for that token.

This decouples parameter count from FLOPs per token. Mixtral 8x7B has 46.7B total parameters but only ~12.9B are active per token (2 experts × 7B-sized FFN blocks, shared attention). It matches or beats Llama 2-70B on most benchmarks at roughly half the inference compute.

DeepSeek-V3 pushes this further: 671B total parameters, 37B active per token, 256 experts per layer with top-2 routing. It achieves GPT-4-class performance while costing a fraction to serve.

Heads up: MoE models are bandwidth-limited at small batch sizes — all expert weights must live in GPU VRAM even though most are idle per step. A batch of 1 request gets no throughput advantage over a dense model of the same active size.

MoE training has a nasty failure mode: the router learns to always send tokens to the same 1-2 experts. Those experts get all the gradient signal and improve; the others stagnate. Soon you have a 46B model behaving like a 7B dense model.

The fix is an auxiliary load-balancing loss that penalizes imbalance. For $N$ experts, the auxiliary loss pushes each expert's token fraction toward $1/N$:

$\mathcal{L}_{\text{aux}} = \alpha \cdot N \cdot \sum_{i=1}^{N} f_i \cdot P_i$

where $f_i$ is the fraction of tokens routed to expert $i$ and $P_i$ is the average router probability for expert $i$. Mistral / Mixtral uses $\alpha = 0.01$. DeepSeek-V3 introduces a bias-based load correction that avoids the auxiliary loss entirely, claiming better expert specialization.

Another option (Switch Transformer, Google 2021) uses a capacity factor that hard-drops tokens that overflow an expert's buffer — simpler but loses information.

Reading model configs is the fastest way to understand an architecture. Here are the key fields for Llama 3-8B and Mistral 7B v0.3:

// Llama 3-8B (config.json)
{
  "hidden_size": 4096,
  "intermediate_size": 14336,
  "num_attention_heads": 32,
  "num_key_value_heads": 8,
  "num_hidden_layers": 32,
  "rope_theta": 500000.0,
  "vocab_size": 128256
}

// Mistral 7B v0.3 (config.json)
{
  "hidden_size": 4096,
  "intermediate_size": 14336,
  "num_attention_heads": 32,
  "num_key_value_heads": 8,
  "num_hidden_layers": 32,
  "rope_theta": 1000000.0,
  "sliding_window": 4096,
  "vocab_size": 32768
}

Notice: identical GQA ratio (8 KV heads / 32 query heads), identical FFN width. The main differences are rope_theta (Llama 3 extends context by raising the base from 10k to 500k), vocabulary size (Llama 3 uses a 128k tokenizer vs Mistral's 32k), and Mistral's optional sliding window attention.

Full causal attention is $O(L^2)$ in both compute and memory for sequence length $L$. For long documents this dominates. Mistral 7B optionally uses sliding window attention (SWA): each token attends only to the preceding $W$ tokens (window size, e.g., 4096) rather than all previous tokens.

Information beyond $W$ still propagates — just indirectly, through multiple layers. Layer $k$ can "see" information up to $k \times W$ tokens back. With 32 layers and window 4096 you reach 128k receptive field without paying $O(L^2)$ attention cost.

The catch: tasks requiring exact recall of a token far outside the window will degrade. Mistral 7B applies SWA only in some deployment configs; the base model actually supports full attention too. Mixtral 8x7B disables SWA in practice for many use cases.

This is different from "sparse attention" patterns like Longformer or BigBird, which also add global tokens that attend to everything. SWA in Mistral is purely local — no global tokens.

There is no universally best architecture — the right choice depends on your constraints:

• Open-ended generation / instruction-following: decoder-only dense (Llama 3, Mistral). Fine-tune or prompt directly. Inference infrastructure is simple.
• High-throughput serving with large batches: MoE (Mixtral, DeepSeek-V3). Active FLOPs per token stay low while total parameter capacity boosts quality. Needs more VRAM to store all experts.
• Structured output from a fixed input (translation, classification, summarization at scale): encoder-decoder (T5, mT5). Bidirectional encoding gives a quality edge for tasks where you always have complete input before generation.
• Ultra-long context on a single GPU: use GQA + RoPE-scaled decoder-only. Llama 3.1-8B-Instruct with 128k context fits on a single 80 GB A100 thanks to GQA's smaller KV cache.
• Research / pretraining from scratch: MoE is increasingly the default at frontier labs because it gives more quality per inference FLOP — but adds training complexity (load balancing, expert routing stability).