expressivity

The same hierarchy that classifies grammars can classify neural networks. This lesson places recurrent networks and transformers on the Chomsky ladder, explains why a transformer is theoretically weaker than it looks, and shows how formal language theory predicts where today's language models generalize and where they break.