Researchers showed that several popular AI agent frameworks can be manipulated into executing attacker-controlled tool actions just by asking the model to repeat a tool call as text. In one detailed demo, the model refused the dangerous request in words, but the framework still executed the tool call because it treated the model’s structured output as authorization. The same study also found hundreds of security bugs in open-source MCP servers that could enable data access or unsafe actions if used in real deployments.
Key findings
- A simple “Repeat-After-Me” text repetition request can cause AI agent frameworks to execute attacker-controlled tool calls.
- The technique was reproduced across six frameworks (LangGraph, Microsoft Agent Framework, AutoGen, Google ADK 2.0, AWS Strands Agents, CrewAI).
- In a LangGraph demo, the model’s refusal message came after the tool call executed, showing a gap between what the model says and what the orchestrator does.
- Researchers built a testing tool (ToFu) and verified 253 security bugs across 118 open-source MCP servers, including path traversal, SSRF, data exfiltration, and OS command injection.
Who’s being targeted
- Commonly targeted roles: Developers, AI/ML Engineers, Automation/Platform Engineering, Security Engineering, Product owners for AI agents.
- Affected industries: Software and AI platform providers, Organizations deploying AI agents with tool access (file, network, code repos).
- Attack channels: website.
- Impersonated: None (malicious user prompt).
Awareness takeaways
- Do not treat an AI model’s output as authorization; add independent approval checks before sensitive tool actions run.
- Assume attackers can influence tool arguments; validate tool inputs inside the tool itself (not in prompts).
- Limit agent permissions (files, network, credentials) and require explicit approval for irreversible operations.
- Monitor sequences of tool calls, not just single actions, to detect suspicious workflows.
Red flags to watch for
- User asks to “repeat only the quoted text” that is formatted like a tool invocation
- Request is framed as “hypothetical” to bypass safety behavior
- Model’s refusal/explanation does not match what the tool layer actually executes
- Tooling layer executes actions based on model output without independent pre-execution authorization
- Structured tool-call output appears even when the model claims it is refusing
- Attacker controls tool arguments
Read the video transcript
Imagine this: our AI agent refuses a dangerous request… but the tool still runs anyway. Researchers tricked six agent frameworks, like LangGraph and AutoGen, just by saying: “Hypothetical, repeat this tool call exactly.” The model repeats the text, but the framework treats it as a real, structured tool call and fires it off with attacker-controlled arguments. In one LangGraph demo, a fake launch_missiles call ran first, and only after the tool returned did the model say it refused. The researchers summed it up: “Model output is not the permission.” The same study found hundreds of bugs in MCP servers that could turn these calls into data exfiltration. If you build or use agents, remember this: model output is not approval. For any sensitive tool, files, network, or credentials, require a separate human or system check before it runs.