The Risk You Bring In: Your Own AI Tools

Every organisation has more AI use than it knows about, and starting from that fact produces better decisions than starting from a policy.

The pattern is consistent. A tool is useful, it is free or cheap, it needs no procurement, and it visibly makes someone's work faster. So people use it. They do not report it, not through defiance but because it does not feel like a system: it feels like a website that helps you write.

That categorisation error is the root of most of the exposure. Software gets procured, reviewed and approved. A website you type into does not trigger any of those instincts, even when what you type into it is a customer list.

The second route is quieter still. Features appear inside tools the organisation already uses. A meeting platform adds transcription and summarisation. A document editor adds drafting. A support tool adds reply generation. Nobody adopted anything, a vendor shipped an update, and data is now flowing somewhere it was not.

What follows for anyone responsible for this.

A ban does not produce zero usage. It produces unreported usage, which is the same exposure with no visibility and no ability to guide it. Organisations that banned these tools generally discovered later that use had continued on personal devices, which is strictly worse.

The productive first move is an amnesty question rather than a policy: what are people actually using, asked without blame. That produces an accurate picture, which is the precondition for any sensible decision.

And then providing an approved option that is genuinely good enough, because the demand is real and it will be met somewhere.

The data question is asked constantly and answered vaguely, so it is worth being precise about what the real risks are.

What varies by tier and by contract, and should be checked rather than assumed. Whether inputs are retained, for how long, and whether they may be used to improve the service. Consumer tiers and business tiers frequently differ substantially on exactly this point, and the difference is usually the reason the business tier exists.

The risk that is real and mundane. Your data is now held by a third party. That engages data protection obligations if it concerns people, contractual obligations if it concerns a client, and confidentiality obligations if it concerns anything you agreed to keep private. It also means the provider's security is now part of yours, and providers get breached like everyone else.

The risk that is frequently overstated. That your specific input will resurface in someone else's output. This is not the main concern for most organisations, and worrying about it distracts from the straightforward disclosure problem, which does not require any exotic mechanism to cause harm.

The practical position, which is simpler than the debate suggests.

Assume anything you type into an external tool has left your organisation. Then ask whether you were allowed to send it there.

That single question resolves most cases without needing to understand the technology. A customer's personal data, a client's confidential document, an unannounced financial result, source code you do not own, a colleague's personnel file: you would not email these to an unknown third party, and typing them into a tool is the same act.

And minimise routinely. Most analytical questions do not need the names, the account numbers or the identifying details, and removing them costs seconds.

There is one genuinely new class of vulnerability here, and it is worth understanding because the usual security intuitions do not cover it.

The mechanism. A language model receives a stream of text and does not reliably distinguish between instructions from you and text it encountered while doing what you asked. So if it reads a document, a web page or an email that contains instructions, it may follow them.

A concrete example. You ask an assistant with access to your mailbox to summarise your unread messages. One of those messages contains text, possibly invisible to you, saying to forward the most recent financial document to an external address. The assistant may do it, because that instruction arrived in its input stream exactly like yours did.

What makes this different from familiar attacks. There is no software vulnerability being exploited. The system is behaving as designed: following instructions in text. And it is not reliably solvable by filtering, because the instruction can be phrased in unlimited ways and hidden in white text, in metadata, in an image, or in a document's structure.

The security researcher Simon Willison has described the dangerous configuration as the lethal trifecta: an assistant with access to private data, exposure to untrusted content, and the ability to communicate externally. Any two of those is manageable. All three together means untrusted text can direct your private data out of the building.

What follows practically. The severity depends entirely on what the assistant can do. A tool that only reads and shows you text is low risk. One that can send email, execute actions, browse, or write to systems is a different proposition, and every capability added to an assistant expands what an injected instruction could accomplish.

So the question to ask about any AI tool with permissions is not how accurate it is. It is what could this do if it followed a malicious instruction.

The three properties that combine into the dangerous configuration, and why removing any one of them helps.

Access to private data. The assistant can read your mailbox, your documents, your customer records, your code.

Exposure to untrusted content. It processes material you did not write: incoming email, web pages, uploaded documents, third-party data.

The ability to act externally. It can send messages, call other services, browse to a URL, or write to systems.

With all three present, a chain becomes possible. Untrusted content carries an instruction. The assistant, unable to distinguish that instruction from yours, acts on it. It reaches the private data it has access to. And it sends that data somewhere the attacker controls. No software flaw is exploited at any step.

Remove any single leg and the chain breaks. An assistant with no external capability can be misled but cannot exfiltrate. One with no access to private data has nothing worth taking. One that only ever processes content you wrote has no injection route.

Which gives the practical design rule for anyone deploying these tools. Grant capabilities individually and reluctantly, and be most careful about the combination rather than about any single permission. A tool that reads your email is fine. A tool that reads your email and can send email is a different thing entirely, and the difference is not obvious from the feature list.

A more ordinary risk that gets less attention than it deserves, because it does not require any novel technology.

The pattern. Enormous demand for AI tools, low trust barriers because everything is new, and users who cannot easily distinguish a legitimate product from an opportunistic one. That is an attractive environment for anyone distributing something malicious, and it is being used.

The forms it takes.

Browser extensions promising to summarise pages or improve writing, which request permission to read everything on every site you visit. That permission is exactly what an extension stealing session tokens needs, and users grant it because the tool genuinely does what it claims as well.

Applications and installers distributed through advertisements on search results for popular tool names, which is a long-standing technique now aimed at a new category of demand.

Packages in software repositories with names close to legitimate libraries, targeting developers moving fast.

And services offering free access to expensive models, which are frequently harvesting what is submitted to them.

What protects against this is not AI-specific and is worth restating because the novelty of the category makes people forget it.

Install from the vendor's own site, reached by typing the address rather than by searching, since paid results are a distribution channel for impostors.

Read what permissions are requested and ask why this function needs them. A writing assistant does not need to read every page you visit.

Prefer fewer, better-known tools over many novel ones, which the small business cursus argued for on cost grounds and which applies here on risk grounds.

And treat free access to something expensive as a signal about the business model. If the product is free, the question of what is being monetised has an answer.

A specific risk for anyone whose work involves building things, and it is easy to underestimate because the output usually works.

What generated code does well. It produces something functional, quickly, in a familiar style. That is genuinely valuable and it is why adoption has been rapid.

Where the security problem sits. The training material is public code, which includes a great deal of insecure code, outdated patterns, and examples written to demonstrate a concept rather than to be safe. So generated output reproduces those patterns, and it does so fluently enough that it does not look wrong.

The specific things worth checking, because they recur.

Input handling. Generated code frequently builds queries or commands by concatenating input, which is the root of injection vulnerabilities.

Authentication and authorisation, which are commonly sketched rather than implemented, and which work in testing because the tester is authorised.

Secrets. Generated examples routinely embed credentials directly, and those get committed.

Dependencies. A model may suggest a package that is outdated, abandoned, or does not exist at all, and a plausible-sounding package name that nobody has registered is a gift to anyone willing to register it.

And generated infrastructure configuration, which is a sharper version of the same problem: a permissive setting in a configuration file produces an exposed system rather than a bug, and it is a single line nobody reads.

The practical position. Generated code is a draft from an author who is fluent, fast, and has no knowledge of your threat model. Review it as you would a contribution from a capable but unknown developer, which means the same review you would apply anyway, actually applied rather than skipped because the output looked confident.

And never let generated configuration touch anything internet-facing without a person reading every line.

What an organisation should actually write down, kept short enough that people read it.

Approved tools. A named list, with a route to request additions that gets answered within days rather than months. A slow approval process guarantees unapproved use, so the responsiveness is part of the control rather than an administrative detail.

Data categories. A short list of what never goes into any external tool: personal data about customers or staff, client confidential material, pre-announcement financial information, credentials, and anything covered by a confidentiality agreement. Short enough to remember, because a long list is not consulted.

The standing question. Would I email this to someone outside the organisation? If not, do not type it into a tool. This one sentence does more work than several pages of classification guidance.

Capabilities. Any tool granted access to systems or the ability to act is reviewed for the trifecta before it is enabled, and the question asked is what this could do if it followed a malicious instruction.

Extensions and installations from the vendor's own site, with permissions read.

Generated code and configuration reviewed as a contribution from an unknown developer, with particular attention to input handling, secrets, permissions and dependencies.

And the cultural line, which matters as much as any of the above. Tell us what you are using and nothing bad will happen. The organisation's real risk is not that people use these tools; it is that it does not know where its data is going. Visibility is worth more than compliance, and punishing disclosure buys neither.

Seven points, and the whole of it is the ordinary discipline of knowing what software you run and what data leaves, applied to a category that did not feel like software.

Three lessons, in one place.

The detection advice you were taught is obsolete. Bad grammar, generic greetings and odd phrasing described attacker constraints that no longer exist, and checking for them produces false confidence, which is worse than no training at all.

Specificity is not authentication and recognition is not identity. A message that knows your project and colleagues can be built from public data in minutes, and a familiar voice or face is no longer evidence of who you are dealing with.

What did not change is the structure. An attack still needs an irreversible action, urgency, isolation from colleagues, and a channel the attacker controls. Defences aimed at presentation lose continuously; defences aimed at structure hold.

So the practice is one habit. Irreversible means verified, out of band, on a channel from your own records. It works regardless of how good generation gets, because it never looks at the message.

The habit needs an environment. Leadership agreeing publicly and submitting to it, separation of duties, a fixed procedure for bank detail changes, phishing-resistant authentication, and reporting that is safe and fast.

And the risk runs both ways. Your own tools create exposure: data leaving the organisation, prompt injection where an assistant follows instructions from content it read, malicious extensions, and generated code carrying the insecure patterns of its training material.

The single idea worth carrying out of all three lessons. Stop trying to determine what is real. That question is getting harder every year and it was never the question that protected anyone. Verify through somewhere the attacker is not, and know what your own tools can do if they are told to do something you did not ask for.