Institutional Memory: Why Companies Forget What They Know

Why did we choose this vendor in 2023? The person who decided has left. The evaluation lived in a spreadsheet attached to an email thread. The reasoning was spoken aloud in a meeting nobody minuted. What survives is the outcome, a signed contract, with none of the argument that produced it.

So the question gets re-answered from scratch, badly, by people who do not know it was already answered well.

This is not a documentation problem, and treating it as one is why so many attempts fail. It is a structural property of organizations: knowledge is produced by individuals in the flow of work, and the organization only retains whatever escaped into a durable, findable form before that individual moved on.

A company brain is a system designed to raise that survival rate. Understanding what it is up against comes first, because the failure modes are old and well documented.

The foundational distinction comes from the philosopher Michael Polanyi, who wrote in 1966 that we can know more than we can tell.

Explicit knowledge can be written down: a policy, a runbook, a schema, a contract, a postmortem. It transfers by being read. It is what a search index can hold.

Tacit knowledge resists writing. It is the sense of which customer complaints are real and which are noise, the knowledge that a particular integration breaks if deployed on a Friday, the judgement of how far a supplier can be pushed. It transfers by working alongside someone, and it is usually the more valuable of the two.

The consequence is uncomfortable for anyone building a retrieval system. A company brain can only ever hold the explicit layer directly. Its effect on the tacit layer is indirect, by making the explicit layer good enough that experts spend their time on judgement instead of on answering the same lookup question for the fortieth time.

Ikujiro Nonaka, later with Hirotaka Takeuchi, built the standard model of how organizations create knowledge, published through the early 1990s and set out in The Knowledge-Creating Company in 1995. Knowledge is not stored, it is converted, and there are four conversions.

Socialization moves tacit knowledge to tacit knowledge: apprenticeship, pairing, sitting next to someone. Externalization moves tacit to explicit: writing the thing down, which is the hardest and most valuable step. Combination moves explicit to explicit: merging documents into a synthesis. Internalization moves explicit back to tacit: reading something until you no longer need to look it up.

The four form a spiral rather than a cycle, because each pass moves knowledge from individuals to groups to the whole organization.

A company brain is unambiguously a combination engine, and it is only as good as the externalization feeding it.

Institutional memory does not fail all at once. It leaks through four channels, and a system that fixes only one changes nothing.

Turnover. A departing employee takes the tacit layer entirely and the undocumented parts of the explicit layer with them.

Silos. The knowledge exists, in another team's tool, behind a permission boundary, in a format nobody outside that team reads. It is retained and unreachable, which for the person needing it is the same as lost.

Unrecorded decisions. The outcome is recorded because it has to be, in a contract or a merged pull request. The reasoning is not, because nothing forces it. So constraints get violated by people who never learned they existed.

Drift. The document still exists and is still findable, and it is now wrong. This is the worst of the four, because unlike the others it fails silently and confidently.

The standard figure quoted to justify these projects deserves careful handling, because it is usually cited wrongly.

The McKinsey Global Institute's 2012 report The Social Economy estimated that interaction workers spend 19 percent of their time searching for and gathering information. That is roughly an hour a day.

Three caveats belong with it every time it is used. It dates from 2012, before the current generation of search and retrieval tools. It measures interaction workers, McKinsey's term for high-skill knowledge workers such as managers and professionals, not all employees. And it counts searching and gathering together, which includes productive research, not only failed lookups.

It is quoted constantly with none of those qualifiers, and often attributed to studies that never made the claim. If a business case for a company brain rests on a single recycled statistic, that is a signal to go and measure your own organization instead.

None of this is new. Through the 1990s, large organizations ran exactly this project under the name knowledge management. They appointed Chief Knowledge Officers, bought repository software, and asked staff to contribute what they knew. Davenport and Prusak's Working Knowledge, published in 1998, remains the careful account of what worked and what did not.

The repositories filled slowly, then stopped. The reasons were consistent across organizations, and none of them were technical.

Contribution cost fell on the author while the benefit fell on everyone else, so it lost to whatever was urgent. Nothing rewarded it at review time. Content had authors but no owners, so nobody was responsible when it aged. Search was poor enough that even well-stocked repositories did not repay the trip. And in organizations where expertise conferred status, writing it down was a rational thing to avoid.

Every one of those failure modes is still available today.

Be precise about this, because the imprecision is where budgets get wasted.

What changed is the read side, and it changed a great deal. Keyword search required the asker to guess the words the author used. Semantic retrieval matches meaning, so a question phrased in the language of the person asking can reach a document written in the language of the person answering. A language model can then synthesize across several documents and answer in prose rather than returning ten links. The trip to the repository now repays itself, which removes one of the five failure modes above.

What did not change is the write side. A model cannot retrieve reasoning that was never written down. It cannot tell you the constraint that lived only in someone's head. Point it at a corpus of stale, contradictory, half-abandoned documents and it will produce fluent, confident, wrong answers, faster than the old search did and with less signal that something is off.

Four of the five classic failure modes survive untouched.

A company brain is a system that makes an organization's recorded knowledge retrievable in the language people actually ask questions in, subject to who is allowed to see what.

Three clauses, each load-bearing.

Recorded knowledge, not all knowledge. The tacit layer is out of reach by definition, and a project that promises otherwise is mis-scoped from the start.

In the language people ask in, which is the part semantic retrieval delivers and keyword search did not.

Subject to who is allowed to see what, which is the clause that separates this from every public retrieval system and is the subject of the next lesson. A public corpus has no permission model. An internal one has a different answer for every employee, and getting that wrong converts a productivity tool into a data-leak mechanism.

Four things get called a company brain and are not one.

A wiki. A wiki is a place to put documents. It has no retrieval layer worth the name, no connection to the systems where work actually happens, and no permission-aware answer synthesis. It is an input to a company brain, not one.

A general chat assistant. A model with no access to your corpus knows nothing about your organization. It will still answer, which is worse than refusing.

A single search box over one tool. Searching your document store while the decision lives in a ticket and the constraint lives in a chat thread solves the smallest part of the problem.

A model fine-tuned on company documents. Fine-tuning teaches style and format, not facts you can update on Tuesday. It also bakes the corpus into weights with no permission boundary, which makes per-user access control effectively impossible. Retrieval keeps documents outside the model, where they can be re-indexed, corrected, and access-checked.

Scope the ambition deliberately, because the three jobs are not equally achievable and teams routinely promise the third while funding the first.

Lookup. Where is the expense policy, what is the deployment procedure, who owns this service. The corpus already contains the answer as stated text. This works today and delivers most of the early value.

Synthesis. Summarize what we learned from the last three enterprise churns. The answer is not written anywhere; it must be assembled from several documents. Achievable when the underlying material exists and is findable, and this is where retrieval plus a language model genuinely beats search.

Rationale. Why did we choose this vendor, why is this constraint here. This one is not a retrieval problem at all. If the reasoning was never externalized, no system recovers it. It is only solvable by changing what gets written down, which is the third lesson.

Most disappointment comes from expecting the third and building for the first.