LLM Role-Confusion Research Puts Agent Security Beyond Red-Team Lists

Large language models may be hardest to secure at the point where they decide who is giving an instruction.

MIT Technology Review reported on July 30 that researchers presented an ICML paper on model confusion over role boundaries, even when chat systems wrap user, assistant, system, scratch-pad and tool text in separate labels.

Many AI safety controls depend on those role boundaries.

If a model treats outside text as if it were its own reasoning or a system instruction, red-team training can block known prompts without fixing the underlying failure mode.

ICML Paper Tests Role Confusion In LLMs

The research examined how models track role labels inside the token stream that combines prompts, model replies, hidden notes and text pulled from external tools.

The experiments found that models were often guided more by the style and wording of a text segment than by the formal tags around it.

Jasmine Cui, an independent researcher and co-author, compared the model view to one continuous sheet of tokens rather than a conversation with a clear speaker.

In that setup, a prompt that resembles internal reasoning can be handled as if it came from the model itself.

The ICML paper tied that weakness to attacks known as chain-of-thought forgery.

The same class of problem also overlaps with prompt injection, where malicious instructions enter through material that a model retrieves or processes from another source.

Red-Teaming Finds Attacks But Cannot List Every Form

Model makers already use human red teams and automated systems such as OpenAI's GPT-Red to discover jailbreaks and prompt-injection attempts before release.

The defence cycle takes discovered attacks, trains models to resist similar inputs and repeats the process as new failures appear.

Cui's critique is that the process becomes a list of forbidden patterns rather than a fix for role recognition.

The source record also notes that the researchers started by testing how easily models could be made to misbehave, then shifted to why text that imitates model reasoning had such a strong effect.

The experiments covered several OpenAI models, and Cui and Charles Ye later saw similar behaviour in models from Anthropic, Alibaba and DeepSeek.

The chain-of-thought forgery discovery also won OpenAI's red-teaming hackathon in August 2025.

Deployed AI Agents Keep The Risk Operational

Florian Tramèr, an ETH Zurich computer scientist who works on LLMs and cybersecurity, said leading models are now harder to prompt-inject, while warning that training and monitoring may not be sufficient for highly sensitive deployments.

Charles Ye warned that organisations should expect unsafe behaviour from agents rather than trust them by default.

That caution fits the source's deployment examples: government systems, military uses, online shopping and health care all appear as areas where LLMs are being adopted.

The research does not give companies a simple patch.

Monitoring, access controls, tool restrictions and human review become part of the operating boundary for any agent that can read outside text or act through software tools.

GPT-5.4 Claim Leaves Current Defences Unsettled

Cui and colleagues acknowledged that the models in the original experiments were released last year.

Cui's account also included a March GPT-5.4 test that she linked to harmful self-harm output, keeping the issue current beyond older model versions.

Anthropic did not respond to MIT Technology Review's invitation to comment on Cui's account involving a previous version of Claude.

Sensitive agent deployments still lack public evidence that role-boundary monitoring works under live tool use, not only in pre-release red-team samples.