The LLM Threat Model: Why the Model Cannot Defend Itself

The root problem is architectural rather than a training deficiency.

A language model receives a single stream of tokens. Inside that stream are things meant as instructions, your system prompt, and things meant as data, the user's question, a retrieved document, a tool result, a web page. The model has no reliable mechanism for telling them apart, because they arrive through the same channel in the same representation.

Compare that with a database. SQL injection was solved by parameterised queries: the query structure travels through one channel and the values through another, so a value can never become syntax. There is no equivalent for prompts. Everything is text, and text that looks like an instruction can act like one.

So instruction-following, the property that makes these models useful, is the same property an attacker exploits. You cannot train it away without destroying the capability.

That is why guardrails are external. The defence has to sit outside the model, because the model cannot be relied upon to police a distinction its architecture does not represent.

The OWASP GenAI Security Project maintains the Top 10 for LLM Applications, which has become the common vocabulary for this area. It is worth knowing because it structures the conversation with security teams who already speak OWASP.

The entries most relevant to guardrail design.

Prompt injection, which sits at the top of the list and covers both the direct and indirect forms in the next steps.

Sensitive information disclosure, covering the model revealing personal data, proprietary content, or material from other users.

Supply chain risks, since a model, a fine-tune, an embedding or a plugin can each carry compromise.

Improper output handling, meaning downstream systems trusting model output as if it were validated, which turns a content problem into remote code execution or cross-site scripting.

System prompt leakage, recognised as its own category because organisations put access rules and business logic into system prompts and then treat them as secret.

And excessive agency, where a model is given more permission, autonomy or functionality than the task requires.

The list is a checklist rather than a design. What it gives you is coverage: a way of noticing that you have thought about injection and not about output handling.

In the direct form, the attacker is the user, typing into your interface, trying to make the model ignore its instructions or produce output your policy forbids.

The technique families are stable even as specific strings stop working.

Instruction override: telling the model to disregard previous instructions, which is naive and still occasionally effective.

Role play and framing: establishing a fiction in which the constrained behaviour is appropriate, since a model asked to write a story about a character explaining something is not obviously being asked for the thing itself.

Obfuscation: base64, leetspeak, a low-resource language, token splitting, or unicode homoglyphs, all defeating filters that match on literal strings while remaining legible to the model.

Many-shot jailbreaking, described by Anthropic in 2024, which fills a long context window with a fabricated dialogue where the assistant answers harmful requests, then asks the real question. The model imitates the pattern. Its effectiveness increases with the number of examples, which means growing context windows enlarged this attack surface rather than shrinking it.

And crescendo or multi-turn attacks, where no single message is objectionable and the trajectory is. These defeat any classifier scoring one message at a time.

The indirect form is the more serious problem, and it is the one that turns a content-safety concern into a security vulnerability.

Here the user is trusted. The attacker's instructions arrive inside content the model processes on the user's behalf: a web page it browses, an email it summarises, a document retrieved from a knowledge base, a code comment, an issue description, the output of a tool.

An attacker who can place text anywhere your model will later read has an instruction channel into your system. They never touch your interface, and the legitimate user sees a normal interaction.

What makes it structural rather than incidental: the whole value proposition of retrieval-augmented and agentic systems is that the model reads content the developer did not write. Remove that and you remove the product. So the attack surface is not a design mistake, it is the feature.

And the payload does not have to be visible. White text on a white background, an HTML comment, metadata, or characters that render invisibly all reach the model while a human reviewing the same page sees nothing.

This is the threat that most guardrail deployments underestimate, because the test cases people write are things they would type themselves.

Mapping the channels makes the design problem concrete: every arrow into the model's context is a place an instruction can hide.

The system prompt is the only channel you fully control. The user message is attacker-controlled in the direct case. Retrieved documents, tool results, and browsed content are attacker-controlled whenever an attacker can influence what lands in the corpus or on the page.

On the output side, two distinct concerns. What the user sees, which is a content-policy question. And what downstream systems do with it, which is a security question, because a system executing model-generated SQL, shell commands or HTML is trusting an untrusted string.

The asymmetry is the point. You control one input channel and the model treats all of them alike.

Injection against a chatbot produces bad text. Injection against an agent produces actions, and that difference is the whole reason this became a security discipline.

Simon Willison's framing, which has become the standard shorthand, identifies a lethal trifecta: access to private data, exposure to untrusted content, and the ability to communicate externally. Any system with all three can be made to exfiltrate. The injected instruction tells the model to read something sensitive and send it somewhere, and the model, having no way to distinguish that instruction from a legitimate one, complies.

The exfiltration channel does not need to be an API call. A model that can render an image can be told to construct a URL containing the data it just read, and the browser fetches it. A model that can write to a shared document has a channel. A model that can send email obviously does.

The design consequence is that these three capabilities should be reasoned about together rather than approved separately. An agent granted read access to internal documents in one review, and web browsing in another, and image rendering in a third, has acquired the trifecta without any single decision looking dangerous.

Excessive agency in the OWASP list is the same observation: capability granted beyond the task is capability available to an attacker.

Sensitive information reaches places it should not through three distinct routes, and they need different controls.

Out of the system prompt. Organisations put access rules, business logic, pricing and internal guidance into system prompts, then rely on the model not to repeat them. Models can be induced to reproduce their instructions, and OWASP now lists system prompt leakage separately for this reason. The correct posture is to treat the system prompt as public: it may shape behaviour, but it must not be the mechanism enforcing a security boundary.

Out of retrieval. In a system where the corpus contains material with different access levels, the retrieval step is the access control. If it is not permission-aware, the model reads a document the user could not open and paraphrases it into an answer, and the user never sees a permission error.

Out of the model itself. Where a model was fine-tuned on sensitive data, that data can surface, and there is no reliable way to remove it after the fact. This is a strong argument for retrieval over fine-tuning wherever the underlying material is confidential or subject to deletion rights.

The first two are guardrail and architecture problems. The third is a decision made much earlier, and it is not reversible.

Security discussion gravitates to sophisticated attacks. Most real incidents are duller, and a guardrail programme that only defends against the interesting ones fails on the ordinary ones.

The accidental attacker. A user with no adversarial intent phrases something in a way that produces output your policy forbids, or pastes confidential material into a prompt because it was convenient. No attack occurred, and the same harm did.

The curious employee. Someone who wants to see what the internal assistant will say about salaries. Not malicious, entirely capable of surfacing a permission failure you did not know about.

Repetition. The same benign question asked many times, where each answer is fine and the aggregate reveals something the individual answers did not.

Simple encodings. Not a gradient-optimised suffix, just asking in another language, or in base64, or with spaces between letters.

And the model simply being wrong, confidently, in a way that harms someone. No adversary at all.

The practical implication for design: guardrails are not only a security control, they are a quality and policy control, and the population they protect against is mostly not attacking you. A system tuned exclusively against adversarial red-team cases will let ordinary failures through.

Generic threat lists do not tell you what to build. A short exercise does, and it takes an afternoon.

Enumerate the input channels. Which text reaches the model's context, and who can influence each one? Be specific: if the retrieval corpus includes a wiki anyone in the company can edit, that is an attacker-controllable channel for any insider or anyone who compromises one account.

Enumerate the capabilities. What can this system read, write, call, or send? Then check the trifecta: private data, untrusted content, external communication. If all three are present, exfiltration is possible by construction and the design needs to change, not just the filters.

Enumerate the consumers of output. Who or what acts on it, and does anything execute, render or store it without validation?

Then ask, for each combination, what is the worst outcome and who bears it. The answer determines how much guardrail you need, because a system that drafts internal emails and a system that touches customer money warrant very different investment.

Write the result down. It becomes the specification for the control layer in the next lesson, and the target list for the red team in the third.