The Science of Language, Computationally

Computational linguistics is the study of human language using the tools of computation, and it has pursued two goals at once since the 1950s.

The scientific goal is to understand language itself: to state, precisely enough that a machine could execute them, the rules and regularities that let a person produce and understand sentences they have never heard. The engineering goal is to build systems that process language, from spell-checkers to translation to modern language models.

The two feed each other but are not the same. A system can translate well while telling you nothing about how translation works, and a precise theory of some phenomenon may not yet run at scale. This path leans toward the scientific side, because understanding what language is, formally, is what explains why the engineering behaves as it does. The standard reference across both is Daniel Jurafsky and James Martin's textbook "Speech and Language Processing".

Language is not one problem but a stack of them, and linguists divide it into levels, each with its own units and its own computational challenge.

Phonology is the system of sounds and how they pattern. Morphology is the internal structure of words: "unhappiness" is un, happy, ness. Syntax is how words combine into phrases and sentences. Semantics is literal meaning, and pragmatics is meaning in context, where "can you pass the salt" is a request, not a question about ability.

The levels are a genuine engineering convenience: each admits different formal tools. Morphology and phonology turn out to be largely handled by finite-state machines, the subject of a later lesson, while syntax needs the more powerful grammars of the next one. Knowing which level a problem lives on tells you which tool it needs.

The single fact that makes language hard to compute is that almost every utterance is ambiguous, and at every level at once.

A word is lexically ambiguous: "bank" is a riverside or a financial institution. A sentence is structurally ambiguous: in "I saw the man with the telescope", who has the telescope is genuinely undetermined by the grammar. "I saw her duck" is ambiguous between a bird and a downward motion, mixing word sense and structure.

Humans resolve this so fast they rarely notice, using meaning and context. A machine cannot skip the problem. A single sentence of a dozen words can have hundreds of grammatically valid parses, almost all nonsensical, and the computational task is not to generate them but to find the one a human would intend. Ambiguity is why language processing is search, not lookup.

Wilhelm von Humboldt described language as making infinite use of finite means, and Noam Chomsky made the point central to modern linguistics. A speaker commands a finite vocabulary and a finite set of rules, yet can produce and understand sentences never uttered before, without limit.

The engine of that infinity is recursion: structures nest inside structures of the same kind. "The cat slept" can become "the cat the dog chased slept", and "the cat the dog the rat bit chased slept", embedding without any fixed bound.

This rules out the most obvious model of language, a giant table of memorized sentences, on principle rather than for reasons of size. There is no largest sentence to store. Any adequate model must be generative: a compact set of rules that produces the infinite set. Deciding how powerful those rules must be is the whole subject of the next lesson.

Language also has a statistical shape that shows up in every corpus ever counted. Rank the words of a large text by frequency, and the frequency of each is roughly inversely proportional to its rank: the second-most-common word appears about half as often as the first, the tenth about a tenth as often. This is Zipf's law, named for George Kingsley Zipf, who documented it in 1949.

The consequence for computation is severe. A handful of words like "the" and "of" account for a large share of all text, while the vast majority of distinct words are rare. No matter how much data you collect, you keep meeting words, and word combinations, you have never seen before, sitting in an enormous long tail.

This is why systems that rely on having seen something before are perpetually starved. Sparsity is not a flaw in the data; it is the permanent statistical condition of language.

The field has always had two traditions, and the tension between them drives its history.

The formal tradition, rooted in Chomsky's work, models language with explicit rules: grammars, automata, and logic. It asks what structures are possible and what a system can in principle express. The statistical tradition models language with counts and probabilities learned from data, asking not what is possible but what is likely, and it is the ancestor of every modern neural language model.

The pendulum has swung repeatedly, and today's large models sit firmly on the statistical side. But the two are complementary, not rival. Probabilities tell you which of many ambiguous options a human would intend; formal theory tells you what the options are, and crucially what no model of a given power could ever capture. The final lesson shows the two meeting, when formal theory is used to explain the limits of neural networks.

It is tempting to conclude that large neural models have made the formal tradition obsolete. They have not, for a specific reason: a model that learns from data still has a fixed computational power, and that power bounds what it can represent no matter how much it is trained.

Formal language theory is the tool that measures such power. It lets you ask whether a given architecture can, in principle, recognize a particular kind of structure, such as balanced brackets nested to arbitrary depth. If the answer is no, then no amount of data or scale will fix it, only a change of architecture will.

That is why this path spends its middle on grammars and automata before reaching neural networks. The classification they provide is exactly what the last lesson uses to explain, and predict, where today's language models succeed and where they quietly fail.

To fix the map before the detail, here is the field laid out by level, with the computational tool that later lessons attach to each.

The pattern worth noticing is that computational power increases as you move down the table. Sounds and words are largely finite-state, which is why they were the field's first solved problems. Syntax demands strictly more power. Meaning and intent are not fully formalized at all. This ordering is not an accident of history; it reflects genuine differences in structural complexity between the levels.

You now have the shape of the field: a stack of levels, pervasive ambiguity, unbounded recursion, a Zipfian long tail, and two traditions whose meeting is where the interesting questions live.

The next three lessons follow the increasing-power ordering directly. Lesson two builds the formal grammars and the Chomsky hierarchy that classify how much power a language needs, and locates human language within it. Lesson three examines morphology, the level where finite-state methods are a genuine and elegant success story. Lesson four brings it to the present, placing recurrent networks and transformers on the same hierarchy to ask what today's language models can and cannot compute.

The through-line is a single question asked at each level: how much computational power does this piece of language actually require? Everything else is detail hung on that frame.