Ten Domains, and a Profile That Is Not Flat

The framework decomposes intelligence into ten broad abilities, each contributing ten percent of the total.

General Knowledge is the breadth of facts and concepts a person has acquired. Reading and Writing is comprehension and production of text. Mathematical Ability is quantitative reasoning and calculation.

On-the-Spot Reasoning is solving novel problems without relying on prior knowledge, which is the ability closest to what people mean colloquially by intelligence.

Working Memory is holding and manipulating information over the course of a task. Long-Term Memory Storage is acquiring and consolidating new information so it persists. Long-Term Memory Retrieval is getting it back out fluently.

Visual Processing and Auditory Processing are perception and reasoning in those modalities.

Speed is performing simple cognitive operations quickly, covering perceptual speed, reaction time and processing fluency.

Read that list looking for what a language model would find easy. Several are things a large pretrained model does extremely well. Two of them, the memory ones, describe something a fixed set of weights does not do at all, and that split is the whole story.

The paper reports per-domain scores, out of ten each, for two systems. The totals are 27 percent for GPT-4 and around 57 percent for GPT-5.

The totals tell you progress happened. The columns tell you what kind.

Three domains went from partial to full or near-full: reading and writing, mathematics, and general knowledge. Two went from nothing to substantial: visual and auditory processing, which is multimodality arriving. On-the-spot reasoning went from zero to seven, which is the reasoning-model work showing up.

And three barely moved. Long-term memory retrieval sat at four. Speed sat at three. Long-term memory storage was zero in both.

The paper's word for this shape is jagged, and it is worth dwelling on because it contradicts how progress is usually described.

The common picture is a rising tide: models get better, and better means better at everything, with the level rising more or less together. Under that picture a percentage score is a position on a single line, and the interesting question is how fast it moves.

The profile shows something else. The later model is at full marks on two domains and at zero on another. That is not a system that is uniformly more capable. It is a system that is superb along some axes and absent along others, and the gap between its best and worst domains is as large as its overall score.

This has a practical consequence people encounter constantly without naming it. A model that drafts a legal analysis competently and then fails at something a child manages is not being inconsistent. It is operating on two different domains, and its profile is high on one and low on the other.

The jaggedness is also why a single number is a poor summary, which the paper's own framing arguably underplays by leading with a total.

Long-term memory storage scored zero for both systems, and the paper identifies it as perhaps the most significant bottleneck.

The domain tests whether a system can stably acquire, consolidate and store new information from recent experience. Not retrieve it from what it already knows, which is a separate domain and where models do partially score, but take something encountered now and make it part of what the system knows going forward.

A fixed set of weights does not do this. A model finishes training and its knowledge stops changing. Everything after that is inference, and inference does not update anything.

The paper calls the result amnesia, and describes it as forcing the system to re-learn context in every interaction. That is a fair description of what using these systems feels like: explain your situation, get useful work, come back tomorrow and explain it again.

The distinction between this and retrieval is the important part. Retrieval augmentation, long contexts and memory features look like a solution and are a workaround. They put information in front of a model that still knows exactly what it knew before. Storage would mean the model changed.

The domains are not equally hard for the same reasons, and grouping them by what a fix would require explains the shape of the profile.

The domains that improved most are ones scale addresses. More parameters, more data and better training raised knowledge, reading and writing, and mathematics. These respond to doing more of what was already being done.

One domain responded to a change in training objective rather than scale. On-the-spot reasoning went from zero to seven when training on verifiable outcomes was introduced, which is a method change rather than a size change.

Two responded to a change in inputs. Visual and auditory processing went from nothing to substantial when models became multimodal.

And one responds to none of these. Long-term memory storage requires a system whose weights change after deployment, which is an architectural property, not a matter of scale, objective or modality. The paper suggests a module that continually adjusts weights to incorporate experience, and notes that existing systems lack this capacity entirely.

That is why it is a bottleneck rather than a gap. The other domains were waiting for more of something. This one is waiting for something different.

Speed sat at three for both models and receives little attention, which is worth correcting because it is quietly interesting.

The domain covers performing simple cognitive operations quickly: perceptual speed, reaction time, processing fluency. In humans it is a genuine component of cognitive ability, and it correlates with other measures.

Machines are in an odd position on it. On some operations they are enormously faster than any person. On others they are slower, because producing an answer requires generating tokens sequentially, and a reasoning model may spend a long time on something a person answers instantly.

So the score is low not because machines are slow in general but because the domain measures a particular kind of fluency, and the architecture is strange with respect to it: extremely fast where a human is deliberate, and slow where a human is automatic.

This is a good illustration of the mismatch objection from the previous lesson. Speed as a factor makes sense for human cognition, where it reflects a shared underlying property. For a system whose fast and slow operations are unrelated to which ones humans find fast, it may be measuring something without a clear machine analogue.

A per-domain profile is more useful operationally than a total, because it tells you which tasks will go well before you try them.

Tasks that lean on knowledge, text comprehension and production, and mathematics should go well, since those score at or near full marks. That matches experience: drafting, summarising, translating, explaining, and quantitative work are where these systems are strongest.

Tasks that lean on novel reasoning should go moderately well and inconsistently, which matches a score of seven.

Tasks requiring the system to accumulate knowledge of your situation over time should fail, and they do. Anything of the form learn how we work and get better at it over months is asking for the domain scored at zero. Every product in that shape is either doing retrieval and calling it memory, or is fine-tuning on a schedule, which is batch training rather than learning.

That is a useful buying question. When a product claims it learns from your usage, the thing to ask is whether any weights change. If the answer is that it stores documents and retrieves them, the answer is no, and the profile tells you what to expect.

The three courses in this series were written together, and this domain is where they intersect.

Silver and Sutton argue the next capability jump comes from agents living in a continuous stream of experience and learning from it. Long-term memory storage is precisely the capacity that would require, and it scores zero. Two independent lines of argument, one about where progress should come from and one about where current systems are weakest, land on the same missing piece.

That convergence is the most interesting thing about reading them together. It is not a coincidence that a paper proposing learning from experience and a paper measuring cognitive coverage both identify continual learning as the frontier. They are describing the same hole from opposite sides.

It also sharpens the reward problem from the previous course. An agent that genuinely stores what it learns is an agent whose objective has time to drift, and the safety critique becomes more pressing rather than less as this domain improves.

The third course asks a different and equally necessary question: whether the measurements underlying claims like these are measuring what they say.