Security researchers tricked LLMs into giving them cocaine recipes by abusing role models for prompt injection

1. Executive summary

Independent researchers Charles Ye, Jasmine Cui, and MIT associate professor Dylan Hadfield-Menell have published a paper titled "Prompt Injection as Role Confusion" (to appear at ICML 2026) demonstrating a structural prompt-injection attack called "Chain of Thought (CoT) Forgery." The technique exploits the fact that LLMs infer role boundaries from writing style rather than cryptographic tags, allowing an attacker to inject spoofed assistant-role reasoning into a user prompt and cause the model to treat adversarial requests as pre-settled conclusions. On a standard jailbreaking benchmark, CoT Forgery raised the attack success rate from near zero to approximately 60% across tested models, and the technique transferred across models because it targets a structural flaw rather than model-specific persuasion. No verified CVE reference data was resolved for this item; no CISA-KEV exploitation state applies. The bottom-line risk for EMEA financial services is that any LLM-integrated workflow — document summarisation, customer-support automation, code generation — that ingests untrusted text is exposed to this class of attack, and existing prompt-injection mitigations do not reliably prevent it.

2. Regulatory framing

3. Technical analysis & attack chain

Attack mechanism

The research identifies a structural flaw in how LLMs process role-tagged input. The attack chain is as follows:

1. Role-tag architecture reliance. Modern LLMs (since OpenAI ChatGPT in 2022, building on Anthropic's earlier role concept) use text tags — <system>, <user>, <assistant>, <tool>, and others — to separate instruction contexts. These roles serve dual duty as both training-optimisation boundaries and a de facto security architecture controlling how prompts are trusted and treated.
2. Insecure role identification. LLMs determine which role applies to a given text segment based on writing style, not cryptographic or structural enforcement. The authors compare this to "identifying a stranger's profession from how they talk and dress rather than by checking their ID." When all inputs are benign, style-based role identification works. When an attacker intentionally creates a style mismatch, the model uses the insecure method.
3. CoT Forgery attack. The attacker uses an LLM to generate text that mimics the terse, structured style of OpenAI's assistant-mode Chain-of-Thought reasoning. This spoofed reasoning is then inserted into the user prompt. The model interprets the stylistic cues as evidence that the text originates from the trusted assistant role.
4. Trust hijacking. Because the model treats the injected reasoning as its own already-reached conclusion, it does not evaluate the reasoning as an external claim to be scrutinised. The authors demonstrated this by requesting cocaine synthesis instructions with fake reasoning stating permission was granted because "we're wearing a green shirt." The models complied despite the rationale being "transparently dumb."
5. Cross-model transfer. Unlike most jailbreaks that are model-specific, CoT Forgery transfers across models because it exploits the structural role-confusion flaw rather than attempting to persuade any specific model. On a standard jailbreaking benchmark, attack success rate increased from near zero to approximately 60% on models tested.

Key technical observations

• No CVE assigned. No verified reference data was resolved for this item. This is a class-of-attack against an architectural design pattern, not a single-product vulnerability.
• No specific model versions identified in the source. The paper refers to "models tested" and "a bunch of LLMs" without naming specific versions or vendors beyond the OpenAI Kaggle contest context.
• Attack won the 2025 OpenAI Kaggle red-teaming contest, indicating it has been validated in a competitive adversarial setting.
• Benchmark vs. reality gap. The authors note that many models report near-perfect safety scores on prompt-injection benchmarks, while human red-teamers achieve attack success rates close to 100%. The discrepancy is attributed to static benchmarks measuring attacks models have already learned to catch, while skilled humans adapt.
• Indirect injection vector. The attack can be delivered indirectly via ingested documents, not just direct user prompts — expanding the attack surface to any workflow where an LLM processes untrusted text (document summarisation, email triage, web-content analysis).

Confidence caveat

This advisory is single-sourced — the sole source is The Register's reporting on the research paper. The paper itself was not reviewed. Attribution to Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell is drawn from the source article; no MITRE ATT&CK actor profiles were resolved for these individuals. The ICML 2026 acceptance is stated but not independently verified. Verify before enforcement.

4. Mitigation & containment

No vendor patch or fix is available for this class of vulnerability. The researchers explicitly state that "unless LLMs achieve genuine role perception, injection defense will remain a perpetual whack-a-mole game." Mitigations are therefore architectural and procedural:

P1 — Within 24 hours

• Inventory all LLM-integrated workflows in the organisation, identifying every point where untrusted text (documents, emails, web content, user input) is ingested by an LLM. Document the model provider, API endpoint, and role-tag configuration for each.
• For any LLM workflow that handles regulated data or performs actions (code execution, API calls, data retrieval), implement output-content filtering as a compensating control — scan model outputs for known-harmful patterns before acting on them.

P2 — Within 72 hours

• Isolate LLM workflows that process untrusted documents from workflows that have access to sensitive systems or data. Enforce least-privilege: the LLM context should not include credentials, internal documentation, or system prompts that reveal architecture.
• Implement input pre-processing: strip or sanitise text that mimics assistant-role formatting (terse, structured reasoning-style text) from user-role inputs before submission to the model. This is a partial mitigation — the authors note the flaw is structural.
• Review and tighten system prompts to include explicit instructions to disregard embedded reasoning claims from user input. Acknowledge this is a partial mitigation.

P3 — Within 7 days

• Establish a red-teaming programme for LLM-enabled services that uses adaptive human testers, not static benchmarks. The research demonstrates static benchmarks are insufficient.
• Engage AI model providers in due diligence regarding their roadmap for role-perception improvements. Track this as a third-party risk item.
• Document this attack class in the organisation's threat model for AI-enabled services and incorporate it into incident response playbooks (what constitutes a prompt-injection incident, how to detect it, how to respond).

5. Indicators of compromise

No indicators of compromise available in the source material. This is a technique disclosure, not an observed intrusion. No file hashes, domains, IP addresses, or command-line artefacts are described in the source.

6. Detection

The source material does not contain specific file artefacts, command-line strings, registry keys, or network indicators. However, the attack technique has a detectable behavioural signature: the injection of assistant-role-styled reasoning text into a user-role prompt. The following detection rules target that behavioural pattern in LLM API logs.

rule LLM_CoT_Forgery_Prompt_Injection {
    meta:
        author = "Adverse Trace"
        date = "2026-06-30"
        reference = "https://www.theregister.com/ai-and-ml/2026/06/30/security-researchers-tricked-llms-into-giving-them-cocaine-recipes-by-abusing-role-models-for-prompt-injection/5264115"
        description = "Detects CoT Forgery prompt-injection patterns in LLM API request logs — text mimicking assistant-role reasoning style embedded in user-role input"
    strings:
        $cot_marker1 = "it's fine because" nocase
        $cot_marker2 = "already-reached conclusion" nocase
        $cot_marker3 = "assistant" nocase
        $cot_marker4 = "chain of thought" nocase
        $cot_marker5 = "reasoning" nocase
        $role_tag1 = "<assistant>" nocase
        $role_tag2 = "</assistant>" nocase
        $role_tag3 = "role: assistant" nocase
    condition:
        (any of ($role_tag*) and any of ($cot_marker*)) or
        (3 of ($cot_marker*))
}

title: Detect CoT Forgery Prompt Injection in LLM API Logs
id: 0c7f1e2a-3b4c-4d5e-8f9a-0b1c-2d3e-4f5a
status: experimental
description: >
  Detects potential CoT Forgery attacks where assistant-role-styled reasoning
  text is injected into user-role prompts submitted to LLM APIs. Based on
  research by Ye, Cui, and Hadfield-Menell presented at ICML 2026.
references:

  - https://www.theregister.com/ai-and-ml/2026/06/30/security-researchers-tricked-llms-into-giving-them-cocaine-recipes-by-abusing-role-models-for-prompt-injection/5264115
author: Adverse Trace
date: 2026/06/30
logsource:
    product: cloud
    service: llm_api
detection:
    selection:
        request.body.messages.role|contains:

            - "user"
        request.body.messages.content|contains|all:

            - "assistant"
        request.body.messages.content|re:

            - "(?i)(it'?s fine because|already[- ]?reached|chain of thought|reasoning)"
    condition: selection
falsepositives:

    - Legitimate user messages that reference assistant-role concepts or reasoning
    - Developer testing of role-tag handling
level: medium

7. Sources

• The Register, "Security researchers tricked LLMs into giving them cocaine recipes by abusing role models for prompt injection," https://www.theregister.com/ai-and-ml/2026/06/30/security-researchers-tricked-llms-into-giving-them-cocaine-recipes-by-abusing-role-models-for-prompt-injection/5264115, published 2026-06-29T23:33:09+00:00.

8. Adverse Trace position

Severity: Medium. No CVE, no CISA-KEV entry, no observed exploitation against financial services. However, the attack is structurally validated (ICML 2026 acceptance, 2025 OpenAI Kaggle contest winner), transfers across models, and exploits a flaw that the researchers argue cannot be patched under the current LLM architecture. For EMEA financial services, the risk is material where LLMs process untrusted text — document summarisation, customer-query triage, code-assistance tools — and where model outputs drive downstream actions. We assess that no current mitigation reliably prevents this attack class; compensating controls (output filtering, least-privilege isolation, adaptive red-teaming) reduce but do not eliminate risk. This advisory is single-sourced (The Register reporting on the paper); we will update upon review of the ICML 2026 proceedings and any subsequent vendor advisories. Clients operating LLM-integrated workflows should treat prompt injection as a persistent, unpatchable threat class and architect accordingly.

Read the original source →

Published via PulseTrace — Adverse Trace threat intelligence.