~/f4n6 $ grep -r "The Harness Is Part of the Model" ./investigations/ --include="*.md"
DFIR

The Harness Is Part of the Model

Jeff Davies 17 Nov 2026 9 min read

The Harness Is Part of the Model
Benchmarking agentic forensic analysis on local hardware, and finding that the model is only one of the variables that matter.

TL;DR

I built a deterministic agentic DFIR evaluation harness and ran 1,202 graded case-runs across 17 campaigns. The models were DeepSeek V4.1 Flash, GLM 5.3 and GLM 5.3 Flash, under different reasoning efforts, step budgets and harness policies.

  • Max effort doesn't make the model smarter, it makes it slower to finish. DeepSeek at max effort with 40 steps solved 4/12 hard cases. At low effort with 100 steps it solved 12/12. Max effort needed 34–56 steps where low effort needed 17–36.
  • Token efficiency is not accuracy. GLM 5.3 used 3–7× fewer prompt tokens and 12–22× fewer output tokens, and finished 8–17× faster. It got the whole case right 21/30 times. DeepSeek went 30/30.
  • GLM's failures are specific. It reconstructs the attack every time. It loses on telling benign decoys apart and on exact counts, which are exactly the fields a forensic report depends on.
  • Harness policy matters as much as the model. In one run the agent looped for 100 steps and submitted nothing. With a forced-submit nudge at 80% of the budget, the same case came back correct.

The short version: cheap reading lets a model be thorough, a generous step budget lets it finish, and the harness decides whether you get either.

Why this experiment exists

Forensic work has two hard constraints that push you away from cloud AI.

Data residency. Client evidence does not go to a third party. Case data, IOCs, hostnames and unredacted logs are often restricted by contract or by law.

Exactness. A confident wrong answer is worse than no answer. If the report says 359 files were exfiltrated and the real number is 369, the report is wrong, and so is everything built on it.

So we run models locally, on four NVIDIA DGX Sparks: about 512 GB of unified memory, and a lot more capacity than bandwidth.

The question wasn't "which model wins a leaderboard". It was: on my hardware, on my kind of task, which configuration produces a report I would sign my name to, and at what cost?

Most public benchmarks can't answer that. They use single-turn prompts, give partial credit, and rarely test whether a model stays coherent across a 60-step tool-using investigation. So I built a harness.

Method

The harness. cyber-eval is a small runner that:

  • gives each task a dataset of evidence (JSONL, 3,000–8,000 rows) and tools to query it: search, count, filters, pagination;
  • runs a multi-step agent loop, where the model picks tools, reads results and must finally call submit_findings with a structured answer;
  • grades the answer field by field. Partial credit is recorded, but "exact" means every field is correct;
  • logs full transcripts, token counts, steps, tool calls and latency for every run.

The grader has no model in it. A pass is a fixed, reproducible comparison against a known answer, and every answer key is generated from the data and checked by a reference solver.

The scenarios. Synthetic but realistic DFIR cases, with benign decoys deliberately mixed in:

Suite

Scenarios

Examples

A

4

BEC fraud; enterprise intrusion (large evidence sets, up to an hour of agent time)

B

6

AWS CloudTrail key leak → privilege escalation → S3 exfil; departing-engineer insider theft with a benign leaver as a decoy; Linux webshell → sudo privesc → SSH-key lateral movement → database exfil

Each scenario ran three times at temperature 0.2, giving 30 agentic runs per configuration. Across 17 campaigns (about 35 model and harness combinations once effort and step budget vary) that came to 1,202 graded runs. The totals below count exact cases only.

The variables.

  • Model: DeepSeek V4.1 Flash (the 1017 build), GLM 5.3 (full), GLM 5.3 Flash (self-hosted and hosted).
  • Reasoning effort: max, high, low, off.
  • Step budget: 40 or 100 investigation steps. Some scenarios set their own cap (the insider cases use 60).
  • Harness policy: with and without forced-submit.

One step is one model turn, and a single turn can make several tool calls.

Result 1: max effort needs far more runway

The least intuitive finding, and the one that changed how I run everything. DeepSeek V4.1 Flash on Suite A:

Configuration

Exact cases

Max effort, 40 steps

4/12 (33%)

Max effort, 100 steps

9/12 (75%)

Low effort, 40 steps

9/12 (75%)

Low effort, 100 steps

12/12 (100%)

Two things are going on.

Max effort was step-starved. At 40 steps, its passes and failures were indistinguishable: both ran to the ceiling, making around 105 tool calls. Given 100 steps, it rose from 4/12 to 9/12. The effort setting wasn't the main problem; the runway was.

Low effort gets there in fewer steps. Low effort solved the same cases in 17–36 steps, where max effort needed 34–56. Max effort wasn't smarter, just slower to commit. Note that max effort with 100 steps and low effort with 40 steps landed on the same score.

The 12/12 came from combining both: low effort so the model doesn't ruminate, and a generous step budget so it has room to finish.

Practical takeaway: in any agentic loop, measure steps to completion, not just pass rate. A model that needs 39 of its 40 steps is one unlucky run away from failing.

Result 2: token efficiency is not accuracy

By any token measure, GLM 5.3 is the more efficient model:

Per agentic case

DeepSeek V4.1 Flash

GLM 5.3 full

Prompt tokens

~717,000

~97,000–222,000

Output tokens

~44,000

~2,000–3,700

Steps

27

11

Median wall-clock

1,118 s

64–143 s

GLM ran the same investigation on 3–7× fewer prompt tokens and 12–22× fewer output tokens, and finished in one or two minutes instead of about twenty. If you optimise for cost per case, GLM wins by a country mile.

And yet:

Model

Suite A exact

Suite B exact

Combined

DeepSeek V4.1 Flash

12/12

18/18

30/30 (100%)

GLM 5.3 full

11/12

10/18

21/30 (70%)

GLM 5.3 Flash (best of its configs)

9/12

3/18

12/30 (40%)

The GLM 5.3 Flash row is the best result across its local and hosted configurations, so treat it as an upper bound.

DeepSeek's extra ~500k prompt tokens and ~40k output tokens per case are not waste. They are the exhaustive search that finds every field: the difference between investigating thoroughly and investigating efficiently.

Result 3: what GLM actually gets wrong

This is the most useful part, because the failure is specific. GLM never failed to reconstruct the attack. Every run found the privilege escalation, the webshell, the exfil path and the lateral movement. What it lost was:

  • Telling benign decoys apart. It flagged the benign bulk downloader instead of the departing engineer, or named the wrong benign automation host.
  • Exact counts. 370 vs 359 off-hours downloads, 229 vs 300, exfil bytes off by 6%.
  • Completeness. One missing list item: cron left out of the persistence mechanisms, web02 missing from the lateral hosts, one anti-forensics step dropped.

These are not random hallucinations. They are exactly the fields that separate a narrative from an evidentiary finding.

Because they are nameable, they are checkable. You can write a deterministic verifier for counts and decoys. You can't write one for "the model misunderstood the whole intrusion". So these near-misses matter less than they look, provided something downstream checks them.

Result 4: the harness is part of the model

Two policy changes moved results more than any prompt or parameter change I tried.

Forced submit. At 80% of the step budget, the harness injects: "Stop searching and call submit_findings now with your best current findings."

One GLM 5.3 Flash run had used all 100 steps and 200 tool calls and submitted nothing. Rerun with the nudge, it returned the correct answer in 96 tool calls and 284 seconds.

To be straight about the rest: three runs had looped to the wall. On the rerun, two passed and one scored 14/15. But the nudge fired on only one of the three. The other two finished inside the budget because the server was less loaded. The policy rescued one run, not three.

It doesn't cost accuracy. DeepSeek's insider scenarios consistently ran to about step 49 of 60, just past the nudge point. The nudge fired six times across the suite, and every one of those runs still produced a 15/15 answer. It turned "ran out of road" into "submitted in time".

Neither of these is a model improvement. They are harness engineering.

So why does DeepSeek win?

Because its architecture makes reading cheap, and agentic DFIR is mostly reading.

DeepSeek V4.1 Flash is a 552B-parameter mixture-of-experts model with an unusual split. Its causal encoder–decoder design runs only the encoder half when reading input, so about 8B parameters are active per token during prefill. Generating output runs the full model, about 16B active per token. DeepSeek built it this way for agent loops, which read far more than they write (Baseten, AI Weekly).

Our workload is a textbook case: about 717,000 tokens read and 44,000 written per case. Measured single-stream on our hardware:

Prefill

Decode

DeepSeek V4.1 Flash

5,894 tok/s

72.8 tok/s

GLM 5.3 Flash

2,514 tok/s

114.9 tok/s

GLM 5.3 Flash actually writes faster. DeepSeek reads more than twice as fast, and reading is where the bill is. The numbers hang together: 44,000 output tokens at 72.8 tok/s is about 600 seconds of generation, which with prefill and tool time accounts for most of the 1,118-second median.

So DeepSeek can afford to re-read the evidence again and again, and that exhaustive search is what finds every field. GLM stops earlier, saves tokens, and accepts a small accuracy loss. For a blog post that trade is fine. For a report with a client's name on it, "98% of fields correct" is still a report you can't ship.

The throughput table covers GLM 5.3 Flash. GLM 5.3 full, the model in Result 2, still needs the same measurement.

Limitations (read these before quoting numbers)

  • Synthetic scenarios. They are realistic and deliberately full of decoys, but they are not real cases. A model that tops this table has not been validated on your evidence.
  • Small n. Three runs per cell at temperature 0.2. The direction of the big differences is solid; a difference of one or two cases in a cell is noise.
  • One grader, one harness. The graders are deterministic and field-level, but they encode our definition of correct.
  • Two model families, one hardware platform. Everything ran locally on four Sparks, plus a hosted GLM endpoint. Different quantisation, batching or serving stacks will move the absolute numbers.
  • Leaderboards don't predict this. On the Artificial Analysis Intelligence Index (read 7 November 2026), DeepSeek V4.1 Flash 1017 ranks #2 among open-weight models and GLM 5.3 ranks #3, less than a point apart. On our decoy tests they went 30/30 and 21/30. Run your own grader.

What I'd recommend if you're building this

  1. Low effort, generous steps, forced submit. Low effort with 100 steps solved 12/12 hard cases, against 4/12 for max effort with 40. That compares two settings at once, so test each on your own workload.
  2. Separate investigation from verification. The fast, efficient model is the one that slips on counts and decoys, and those failure classes are known. Let it investigate, then check those fields with a deterministic script or a slower, more precise model.
  3. Grade at field level and publish exact-case rates. Partial credit is how "93% accurate" quietly becomes a wrong report.
  4. Measure before you buy. The differences between these models and settings were larger than most hardware upgrades would give. Profile your own workload first.

What's next

Two model families can't settle the bigger question: is accuracy mostly about total capacity, with cheap reading deciding how much of it you can afford to use? That is the hypothesis to test next.

The open-weight landscape moved fast this autumn. Kimi K3.1 Pro tops the agentic open-weight rankings, but at around 1.5 TB of weights it needs a different class of machine. The realistic candidates for four Sparks are the sparse models that fit on two Sparks:

  • MiMo-V2.6-Flash: 309B total, 15B active
  • MiniMax M3.1 Flash: 428B total, 23B active
  • K2-Horizon-375B: 375B total, 23B active, Apache-licensed

(Sizes and licences from GPUwerk's autumn-2026 release audit, which checked 21 releases against Hugging Face.)

Each will go through the same harness, the same scenarios and the same grader, because on this evidence the leaderboard won't tell us the answer.

The results, scripts and evaluation harness are internal to f4n6's research. If you'd like the scenario pack, the grader definitions or the throughput bench script, get in touch.

Sources

Post this to LinkedIn
Formatting is converted automatically — headings, bullets, a link back & hashtags. Paste straight in.
J
Jeff Davies
// read next

Related