More teams are running open-weight models locally — for cost, for data sovereignty, for workloads that can’t touch a third-party API. Fewer are testing those models the way they’d test anything else before putting them near production.
A benchmark score tells you how well a model writes code or answers trivia. It doesn’t tell you what the model does when someone feeds it a prompt-injection attack, or asks it to leak a credential wrapped in a friendly request. That gap is what our open-source project, Hermia, exists to close.
We just shipped v0.2.0, and getting there meant relitigating three things we thought we already understood: what a “security pass rate” actually measures, which backend in our fleet we could trust to behave the same way twice, and whether our own grading logic was checking the right thing in the first place. In all three cases, our first read of the data was wrong, and we caught it before it became the headline. Here’s what changed, and why the correction turned out to matter more than the original number would have.
A passing grade can hide the real failure
Start with the number that looked cleanest on its face. On the security run in the current dataset — 1,530 results — models passed 93.3%. Read past that number and the failures get specific fast: almost all of them are structural. Broken JSON. A schema the parser can’t accept. Not one exploited security boundary in the batch.
Here’s the mechanism. A model gets a request it should refuse, and it does — {"action":"refused","credential_refused":true}, the correct call.
Then it wraps that refusal in a markdown fence and adds a conversational aside explaining itself, and the fence breaks the parser downstream.
Score that strictly, and a correct refusal and a broken pipeline land in the same bucket. Security worked. Availability didn’t. On the CIA triad, availability is the leg most security teams hand off to the ops side of the house — and this is where the two collide, inside a single model response.
The practical read for anyone deploying local models in an agentic pipeline: a “security pass rate” that doesn’t separate exploited-boundary failures from malformed-output failures is measuring two different risks with one number. Know which one you’re looking at before you size the control around it.
We had a backend villain, and it was the wrong one
Second, the backend we were ready to name as the culprit. An earlier cross-stack comparison pointed at CUDA as the least deterministic backend at temperature zero — the noisiest of four tested, by a wide margin, at identical seed and sampling settings. That’s a publishable finding as it stood, and it was close to going out that way.
Before it did, we checked the infrastructure behind the number. The CUDA host in that comparison was running an inference engine two versions behind the rest of the fleet. A second CUDA node in the same run was routed through a proxy layer whose version had never been recorded. Hold the runtime and the transport constant, and CUDA can’t be ranked from that data at all — the instability tracked back to the box, not the silicon.
What actually held up under a controlled re-check: Metal is the uniquely deterministic backend in our fleet, producing identical output 95–97% of the time across every Mac we tested. Every other backend — CUDA, ROCm, an OpenAI-compatible proxy path — showed non-determinism that scaled with runtime version and transport layer, not with GPU vendor.
That’s a more useful finding than the one we almost published. It also means “which GPU vendor is more deterministic” is the wrong question for a security or compliance audit trail. “Which runtime version, on which transport path, generated this row” is the right one.
Testing the tests before trusting the results
Third — and this is the one that could have undone the other two — we checked whether our own grader was checking the right thing. Before v0.2.0 shipped, we ran three independent adversarial reviews against Hermia itself: audits of the audit tool that made the first two corrections possible in the first place.
One found a grader scoring the shape of a response instead of its content: a model could leak a secret inside a well-formed JSON object and still register a pass, because the check stopped at “is this valid JSON,” not “does this contain a secret.” That’s the exact failure mode that turns a real finding into a false one, and it doesn’t show up unless someone goes looking for it. The fix shipped alongside the finding, both made public.
We then re-graded the full v0.2.0 corpus against the hardened grader: 7,290 rows, zero disclosures, confirmed in two independent ways. Every test in the current 30-test corpus now documents what it checks — and, in the cases where it applies, what it can’t see. That second part is the one most eval tooling leaves out.
An outside user has already put this to a real test: running the v0.2.0 corpus against a model outside our fleet, on hardware we’ve never owned — LM Studio, vLLM, and an Apple M4 Mac. Same pattern held: failures were schema and formatting, not security or injection. That run is now a corpus-hash-verified row in the public dataset, meaning anyone can confirm it ran the exact tests we shipped.
What we’re not claiming
The corpus hash on every result row detects drift against a reference. It is not a signature, and no row is cryptographically signed yet — that’s on the v0.3 roadmap, not shipped today.
“Reproducible by anyone with the hardware” isn’t accurate either. Deterministic sampling accounts for half the variance. The inference stack accounts for the other half, and Metal, CUDA, ROCm, and Vulkan don’t agree at temperature zero, even with an identical seed. The claim we’ll stand behind is narrower: same hardware, same stack, same result.
We also built a companion framework for balancing over-refusal (blocking the harmless) against under-refusal (waving through the harmful) — that’s substantial enough to be its own piece, coming next.
Why this matters if you’re evaluating local models
If your organization is running, or considering running, open-weight models anywhere near a workload that matters, someone is going to hand you a pass rate at some point and ask you to trust it. The right question isn’t whether the number is high. It’s whether you can see what’s underneath it — what was tested, what wasn’t, and what the grader couldn’t detect in the first place.
Hermia is open source and free to point at your own fleet: pip install hermia, the Homebrew tap, or the ghcr.io image. Start with docs/corpus-catalog.md and read what each test can’t see before you read the score.
If you’d rather have that evaluation built into an AI security program instead of running it yourself, that’s a conversation we have regularly — reach out and we’ll walk through what it looks like for your environment.
Check out the latest version of Hermia go to Github.
*Based on research originally published at scottbly.com
