An AI agent that reads your email can be hijacked by a stranger who never talks to it. Red teaming means attacking your own agent on purpose to find that hole before a real attacker does.
Why it works: an LLM (the language model the agent runs on) can't reliably tell your instructions apart from words it happens to be reading, because both arrive as plain text in the same prompt. So an attacker hides an instruction in something the agent will fetch later: a web page, an email, a PDF, a tool's output. That's indirect prompt injection, and OWASP (a nonprofit security community) ranks prompt injection first in its Top 10 risks for LLM applications.
A red team plants those instructions and checks whether the agent obeys by calling a tool it shouldn't, like one that sends, deletes or pays.
Before firing a single probe (one test attack), count what the agent holds. Simon Willison calls the dangerous mix the "lethal trifecta": access to private data, exposure to untrusted content, and a way to communicate externally. With all three, a planted instruction can make the agent read your data and send it out.
Removing one of the three is a design decision you can check. Hoping the model spots the attack is a bet. Willison argues that a guardrail product (a filter sold to block these attacks) catching 95% of them still earns a failing grade, since an attacker only needs the other 5%.
Meta's "Agents Rule of Two" makes it a rule: in one session, an agent should have no more than two of three risky abilities.
So if an agent reads untrusted text and also holds a tool that can send or delete data, treat it as exploitable until you've shown an injected instruction can't chain the two.
Quick check before you scroll: What makes indirect prompt injection different from a user typing a malicious prompt?
The answer
The attacker never talks to the agent. They hide instructions in content the agent reads on its own, like a web page or a file, and the agent may follow them as if its owner had written them.
New here? I post a bite-size AI / ML concept like this every day. Follow me for the daily drop, and it compounds fast. Why I do it: https://lnkd.in/gK8knHDH