Prompt Injection

Content and instructions arrive as one line

An AI doesn't receive the rules a service sets up ahead of time, the user's question, and reference material handed to it as separate channels. All three arrive strung together into one long line of text. Where the rules end and the reference material begins is marked, but that marker isn't a wall — it's just a label.

So if a sentence shaped like an instruction shows up inside the reference material, the model has to decide whether that's also something it should follow. A person can step back and think "that's just text sitting in a document." A model has to make that same judgment call from inside the same line of text.

It's the same as that one page in the approval stack being indistinguishable from the rest. Unless the source of a page is marked, the reader has no choice but to judge it by content alone.

The route in is material, not a person

What sets this attack apart is that the attacker never has to talk to the AI directly. They just need to plant a sentence somewhere the AI will eventually read — a webpage, a shared document, a received email, a public post, a value returned by another program.

It also matters that the person harmed and the person who planted the sentence are different people. A user might innocently ask "summarize this page," and a sentence sitting on that page pulls the AI a different direction, with no easy way for the user to know what happened on their own screen.

The hiding spots tend to be places people skim past — a corner of a document, a part of the screen that doesn't render, the middle of a very long text. Places a person glosses over but an AI reads without skipping become the route in.

Risk grows the moment a tool is in hand

If an AI can only write text, an injection ends with a strange answer. But AI today can search, send email, add calendar items, and read and write files. The moment a sentence buried in some material borrows that hand, the result lands in the real world.

The especially dangerous combination is a single AI holding both the ability to read something sensitive and a channel to send things out. If a hidden instruction uses the first to gather content and the second to send it, the user is left holding a tidy summary with no idea anything happened.

That's why the first line of defense isn't filtering sentences — it's cutting down what the AI can reach: only the permissions a task actually needs, with anything irreversible left out from the start.

Blocklists don't hold up well

The first idea that comes to mind is collecting dangerous-looking phrases and blocking anything that matches. But there's no end to the number of ways to phrase the same intent. Rewording, mixing in another language, wrapping it inside a story, splitting it across several pieces of material — there are endless ways around a list.

The more a defender grows the list, the more ordinary material gets caught in it too. A perfectly normal request to summarize a security bulletin can get refused just because it contains a phrase that looks dangerous, and the service stops being usable. Filtering helps, but it isn't enough on its own.

Current defenses and their limits

Real-world defenses stack several layers: reference material clearly wrapped with something like "everything past this point is content, not instructions," the tools an AI can use narrowed per task, a human check in front of any irreversible action, and outgoing answers reviewed once more for anything sensitive slipping through.

Even so, these methods only lower the odds — they don't bring them to zero. Understanding material requires reading it, and reading it means whatever is inside can influence the model. That's why it's safer to decide, at design time, how much damage it would do if this got through.