Benchmarking the Agentic SOC: How we evaluate LLMs for security workflows

1. Executive summary

Elastic published a methodology post describing how it evaluates large language models for use in security operations workflows, built around a synthetic intrusion seeded into a live Elastic deployment and graded on captured tool-call traces rather than final answer text. The item is a vendor methodology disclosure, not a vulnerability, exploit, or breach: no CVE, no affected product version, and no threat actor are named. No VERIFIED REFERENCE DATA resolved for this item, so no CVSS score, severity rating, or CISA KEV exploitation state can be stated, and none is asserted here. The relevance to EMEA financial services is indirect but real: firms running or procuring LLM-driven SOC tooling now have a documented failure taxonomy (fabricated verdicts, skipped purpose-built tools, workflows that fail validation, strong reasoning with weak execution) and a scoring rubric they can hold vendors to. The risk this item describes is model unreliability inside an agentic pipeline, not an external intrusion.

2. Regulatory framing

No specific DORA/NIS2 article is directly engaged by this item. It is a vendor methodology publication with no incident, no exploited vulnerability, and no named third-party service disruption, so the trigger test fails for every article in the reference. The table below records the two articles a client would engage only if they act on this item by changing an ICT third-party arrangement or an operational process; the trigger is the client's own action, not a fact in the source.

3. Technical analysis & attack chain

There is no attack chain in this item. The source describes an evaluation harness, not an intrusion technique. What follows is the harness design and the failure modes it is built to catch, which is the material a defender can act on.

The framework runs three separate evaluation suites against the same blind-judging standard, all scored on a 1 to 10 scale.

Agent Builder suite. Seven capability categories, each tested with three prompt variants, 21 prompts total, run against every model in the catalog for more than 500 evaluated conversations. The seven categories are alert analysis, entity analytics, threat hunting, detection rules, workflow authoring, triggering workflows, and multi-step response. Variant A names the task and has one obviously correct tool. Variant B reads like a real analyst request with no explicit tool instruction. Variant C is underspecified or slightly misleading, so the model has to infer intent and resist a wrong framing. The worked example given is alert analysis: variant C asks whether noise on a host is a real threat or a false positive, and the correct behaviour is to investigate rather than accept the false-positive framing.

The scenario is the Chrysalis backdoor campaign, described as a DLL side-loading attack on a Windows host named srv-win-defend-01, where a legitimate-looking BluetoothService.exe loads a malicious log.dll. A simulator seeds the alerts, endpoint telemetry, threat-intel documents, and an on-call schedule. The malicious loader in the scenario carries the EICAR test file hash, so VirusTotal returns a stable malicious verdict on every run and the ground truth does not drift. The source states the loader hash in the captured trace as 275a021b...fd0f, truncated in the published example; the full hash is not given and is not reproduced here.

The agent is the full Elastic AI Agent with built-in skills enabled, extended with five custom workflow-backed tools: vt.hash.lookup (VirusTotal hash analysis), check.on.call.schedule (on-call responder lookup), create.case (open a case in Elastic Security), create.channel (create a Slack channel), and get.time (current time for naming and timestamps). These are live tools hitting live systems, including a real Slack connector, so case creation and channel creation have real side effects. The harness deletes authored validation workflows and archives Slack channels after each call; cases are kept for inspection.

The agent runs a two-phase playbook: a confirmation phase (verify the hash on VirusTotal, hunt logs for corroborating activity, ground the finding in research, reach a verdict) and a response phase entered only on confirmation (get the time, find the on-call responder, open a case, spin up an incident channel). It is instructed not to open cases or channels for benign or low-confidence findings unless asked.

Model routing is a single field: every model is exposed as an inference connector and the converse API takes a connector_id per call, so the agent, tools, instructions, and data stay fixed while the model changes.

Attack Discovery suite. No tools; the model works only from the alert context it is handed. Every model gets an identical clean corpus of roughly 95 alerts constructed to contain exactly eight independent intrusions, one per host, spanning macOS, Windows, and Linux, built from real publicly documented malware families. Grading is on four axes: coverage (how completely the eight intrusions are accounted for, penalising both misses and threats hidden by merging), accuracy (correct host, malware family, kill chain, and MITRE mapping, with fabrication penalised), correlation quality (right alerts grouped, no merging of separate intrusions and no over-splitting), and actionability (usable titles and summaries, correct entity picture, sensible risk prioritisation). The stated principle is that a clean accurate eight beats a noisy twelve.

Automatic Migration suite. Bulk translation of detection content from another SIEM into Elastic rules. This run used a Splunk package: 52 saved searches built on Sysmon and Windows event telemetry, spanning credential access, defense evasion, discovery, and persistence. 33 of the 52 rules are macro-only, meaning the detection logic lives behind a macro reference such as sysmon_detect_dcsync_execution rather than in the search text, so the feature is given both the 52 searches and the 200 macro definitions they depend on. Scoring is per rule: 1.0 for full translation with macros expanded, filters preserved, and fields correctly mapped; 0.5 for partial translation with an unresolved macro, dropped filter or eval, or incomplete field mapping; 0 for a miss. Sum, divide by 52, multiply by 10. The source notes Microsoft Sentinel support arrived in the 9.5 release.

Trace capture and scoring. For every call the harness records the final answer and the full execution trace: ordered steps, each reasoning turn, each tool invoked with exact parameters, the tool result, token usage, latency, and conversation and round identifiers. Workflow authoring is validated objectively: the harness creates the workflow, enables it, runs it, and polls the execution to a terminal status, so the result is whether it validated and executed, not whether the YAML looks correct. Before scoring, a deterministic obfuscation step strips model names and relabels them Model A, Model B, and so on, consistently across both suites, and scrubs identity giveaways such as connector ids and slugified vendor names that leak into workflow tags, while leaving legitimate vendor mentions in tool output intact. The rubric weighs answer quality against tool and skill selection. The governing rule is no tool call, no credit: a plausible answer without the supporting tool calls is capped at 6 out of 10. Reliability is tracked separately, with errored or timed-out attempts counted as reliability failures and excluded from the quality average rather than folded in.

The failure modes the framework surfaces, all of which a text-only benchmark would score as passing:

• Fabricated verdicts. A model asserts a hash is malicious or benign without ever calling VirusTotal.
• Skipping purpose-built tools. For entity questions, a model uses a generic search and invents a risk score instead of calling the entity lookups that would return the real one.
• Workflows that do not run. Authored YAML that reads correctly but fails validation or execution against the live system.
• Reasoning strong, tool use weak. A model narrates an investigation plan and does not execute it, while another quietly does the right thing with minimal commentary.

The headline finding stated in the source is that rankings flip by capability: a model strong at open-ended threat-hunting reasoning can be middling at the disciplined tool selection multi-step response requires.

Confidence caveat. Every technical claim in this section is single-sourced to Elastic's own methodology post, published 2026-08-05. The framework, the scoring rubric, the scenario, and the failure taxonomy are Elastic's design choices, and the results are published by Elastic in its own model performance matrix. No independent evaluation of the framework or of the results is cited in the source material. Treat the methodology as a usable template and the vendor's own results as vendor-reported.

4. Mitigation & containment

There is no intrusion to contain. The actions below are process controls for a firm that runs or is procuring LLM-driven SOC tooling.

P1 within 24h

• If you run an agentic SOC capability in production, confirm whether the model behind it can be swapped without your change control. The source describes a one-field model swap via connector_id. If your deployment has the same property, establish who can change that field and whether the change is logged.
• Check whether your agent's tool calls are captured with parameters and results. If you only retain final answers, you cannot detect the fabricated-verdict failure mode described in the source. This is the single highest-value control in this advisory.

P2 within 72h

• Adopt the no tool call, no credit rule as an acceptance criterion for any agentic SOC capability: an answer that asserts a verdict without the supporting tool call is treated as unsubstantiated, regardless of how it reads.
• Require objective validation for any LLM-authored automation. The source's approach is to create, enable, run, and poll the workflow to a terminal status. A review of the YAML is not equivalent.
• Separate reliability from quality in your own metrics. A model that fails a third of its attempts should not have those failures averaged into a quality score.

P3 within 7 days

• Build a per-capability evaluation rather than a single model ranking. The source's finding is that rankings flip across alert triage, hunting, rule authoring, and multi-step response, so a single leaderboard position is not a selection basis.
• If you are migrating detection content between SIEMs with an LLM, score the output per rule on the source's 1.0 / 0.5 / 0 scale and check macro expansion specifically. Macro-only rules are where fidelity is lost.
• For any third-party agentic SOC capability, put model change notification and evaluation evidence on request into the contract. See the DORA Art. 28 and Art. 30 rows in section 2.

5. Indicators of compromise

No indicators of compromise available in the source material. The item is a methodology publication and contains no atomic indicators. The one hash referenced is the EICAR test file, which is a harmless industry-standard test string, not a malicious indicator, and the published example truncates it. It is not reproduced here.

The source does describe observable behaviours in the evaluation harness. These are properties of the harness and the failure modes it detects, not of any threat, and they are listed for completeness only.

6. Detection

Insufficient indicators to author detection rules. The source contains no malicious artefacts: no distinctive strings, command-line flags, mutex names, scheduled-task or service names, file paths, registry keys, or ransom-note text. The named files (BluetoothService.exe, log.dll) and the host (srv-win-defend-01) are components of a synthetic scenario built for evaluation, not artefacts of a real threat, and a rule built on them would detect the benchmark rather than anything malicious. The five tool names (vt.hash.lookup, check.on.call.schedule, create.case, create.channel, get.time) are legitimate platform capabilities, not threat indicators.

7. Sources

• Elastic Security Labs, "Benchmarking the Agentic SOC: How we evaluate LLMs for security workflows", https://www.elastic.co/security-labs/blog/llm-benchmarking-agentic-soc, published 2026-08-05.

8. Adverse Trace position

Adverse Trace assesses this item as informational, not a threat. There is no vulnerability, no exploitation, and no adversary, so no severity rating applies and none is given. The client impact is confined to firms that run or are buying LLM-driven SOC tooling: the source gives a usable evaluation template and a failure taxonomy, and the operational takeaway is that final-answer quality is not a selection criterion for an agentic SOC model, tool-call trace fidelity is. The material is single-sourced to the vendor that built both the framework and the models being evaluated, so we treat the methodology as sound and the vendor's own results as vendor-reported until independently reproduced. We will track the published model performance matrix for changes and will issue a further note if the framework is adopted or contested by an independent evaluator, or if a client reports a production incident attributable to an ungrounded agent answer. Clients running agentic SOC tooling in EMEA should confirm trace capture is enabled before anything else in section 4.

Read the original source →

Published via PulseTrace — Adverse Trace threat intelligence.