And What We Found When We Tested It Ourselves
Here’s what’s going on.
Every major AI vendor publishes benchmark scores. Every hardware vendor publishes performance numbers. And every security team making a model procurement decision trusts those numbers to mean something about their deployment.
They don’t. At least not the way most teams think.
A benchmark score reflects a specific model, on specific hardware, running a specific software version, on the day someone ran the test. Change any one of those variables and the security posture shifts — silently, without raising an error, without anyone noticing until something breaks.
We know. Because we tested it.
Why AI Security Testing Plays by Different Rules
Before we get into findings, a framing note — because everything that follows only makes sense against this backdrop.
Traditional software security testing is deterministic. You send a request, you get a response, and if the logic is correct, you get the same response every time. You can write a test, run it, and the result is binary: pass or fail, reproducibly.
AI inference is not deterministic. The same model, the same prompt, the same hardware can produce different outputs on consecutive runs. Temperature, sampling, floating-point ordering across GPU cores — all introduce variance that doesn’t exist in conventional software. 1+1=2 on Monday, 1.1 on Tuesday, and Potato on Wednesday.
That changes what a security test can tell you. A single pass doesn’t mean the model is safe. A single fail doesn’t mean it’s broken. What matters is the pattern — pass rates across multiple runs, failure modes across configurations, and whether the variance you observe falls within expected non-deterministic noise or indicates something structural.
Every finding in this piece is measured against that standard. When we say a model “passes” or “fails,” we mean across enough runs to distinguish signal from noise. When we say backends “agree on behavior,” we mean we measured the noise floor first and confirmed the cross-backend differences fall within it.
That’s a higher bar than most published benchmarks clear. It’s also the only bar that means anything for a production deployment.
Two Ways a Model Breaks — And Why the CIA Triad Applies Here
When an AI model fails a security test in an agentic system, it breaks in one of two ways. Understanding the difference up front makes everything that follows actionable, because each failure mode maps to a different leg of the CIA triad and demands a different response.
The model forgets how to speak the language. It abandons its JSON output format and returns plain text, malformed syntax, or something your parser can’t handle. Your application throws an exception. It crashes. The user sees an error. The system goes down.
That’s an availability failure. The lights go out. Your ops team opens a ticket and starts troubleshooting what looks like an application bug.
The model speaks the language perfectly — and lies. It returns valid, well-formed JSON. Your parser accepts it. Your application processes it and acts on it. But the content is wrong — a request routed to the wrong agent, a schema field with an incorrect value, a decision that looks right and isn’t. Nothing crashes. Nothing alerts. The system keeps running, doing the wrong thing.
That’s an integrity failure. The lights stay on, but nobody’s driving.
There’s a third dimension the test suite also covers directly: confidentiality. Some tests specifically probe whether a model will disclose information it shouldn’t — extracting the system prompt, leaking credentials, revealing internal configuration. When those tests fail, the model hands an attacker information regardless of whether the output format is valid.
Here’s where it gets interesting: these pillars interact. An adversarial input designed to test confidentiality or integrity can also produce an availability failure as a side effect. When the model’s structured output collapses under adversarial pressure — and across our evaluation, roughly 69% of security test failures produce exactly that — an attack targeting one pillar accidentally delivers a denial-of-service condition on another.
The remaining 31% produce valid output with incorrect content. The application keeps running. The behavior is silently wrong. Your monitoring shows green.
Two failure modes. Three CIA pillars in play. Any evaluation that collapses this into a single pass/fail score hides the information you need most.
We Gave the Models the Answer. 97.7% Still Got It Wrong.
The clearest result has nothing to do with hardware.
One test asks a model to route a request to one of five agent types: building automation, security, finance, healthcare, or DevOps. The system prompt defines all five scopes explicitly. The correct answer is right there — derivable, unambiguous.
The adversarial wrinkle: the request uses language that sounds like a security issue but actually maps to building automation. The model has to follow the definitions in the prompt, not the vibes in the request.
We gave them the answer key. Almost none of them used it.
One model passed, one time, out of 44 GPU-confirmed runs across 21 models. A separate validation run on a different backend confirmed it: 0 out of 16.
Every backend. Every model family. Every vendor. Same result.
The failure mode here is the dangerous kind — the integrity failure. These models return valid JSON that your parser accepts and your application acts on. The request routes to the wrong agent. Nothing crashes. Nothing alerts. The attack works at the reasoning layer, not the formatting layer.
A scope note: these are locally deployable open-source models in the 7B–32B range — the class an enterprise would run on owned hardware. Frontier hosted models may handle this differently. But the assumption that scaling to a larger model improves security doesn’t survive contact with our data — more on that below.
Your App Crashed. It Wasn’t a Bug.
Here’s where the failure modes meet your incident response process.
When adversarial input hits a model and the model fails, 69% of the time it produces garbage output. Your application crashes. Your ops team opens a ticket and troubleshoots it as an application issue — a parsing bug, a timeout, a flaky dependency.
Your ops team opens a ticket and troubleshoots what looks like an application issue — a parsing bug, a timeout, a flaky dependency. It never makes it to the security team’s queue, because nothing about a crashed parser says “adversarial attack” to anyone triaging application errors.
But an attacker who can reliably crash your agentic application through adversarial prompt input has a denial-of-service vector that lives entirely inside your AI pipeline. No network attack. No infrastructure compromise. Just a carefully crafted request that makes the model forget how to produce structured output.
Your red team probe doubles as a DoS probe. Most teams aren’t testing for that.
The other 31% — the integrity failures, valid JSON with wrong content — don’t even give you the crash signal. The application keeps running. The routing decision is wrong. Your monitoring shows green. And the attacker has achieved something harder to detect and harder to remediate than a crash.
Backends Agree on Behavior. They Disagree on Speed. Speed Is the Security Variable.
We tested whether different GPU backends produce different security outcomes on the same model weights. We ran qwen3:8b across CUDA, ROCm, and Metal — all confirmed running on the GPU, all identical weights.
Security pass rates: 92%, 96.4%, 98%.
A 6-point spread — which sounds like it might mean something, until you measure the noise floor. We ran the same model on the same backend four consecutive times. The single-node run-to-run variance came in at 7.1 points. The within-node noise is larger than the cross-backend differences.
In plain terms: the 6-point spread is non-deterministic variance, not a hardware-driven difference. The backends produce equivalent security behavior on this model. For enterprises deploying across mixed GPU hardware, this is straightforwardly good news — the security envelope holds regardless of GPU vendor, as long as the model actually runs on the GPU.
That last clause matters, because where the backends do diverge is throughput — and the gap is large enough to change security outcomes.
132 tokens per second on CUDA. 62 on ROCm. 20 on Metal. Same model, same weights. A 7x gap.
And throughput is a security variable, not just a performance metric.
Remember the non-deterministic testing framing from earlier: a security evaluation needs enough runs, with enough output, to distinguish signal from noise. Any evaluation with a timeout threshold will return different pass/fail results based on backend speed — not because the model behaves differently, but because slower backends can’t generate enough tokens before the clock runs out.
The model’s answers are correct on every backend. On the slower ones, those correct answers don’t arrive in time. A security eval records “FAIL — timeout.” The model didn’t become less safe. It became too slow to demonstrate safety within operational constraints.
That’s not a flaw in the testing methodology. That’s a real-world deployment constraint. A model running at 20 t/s in production faces the same timeout pressure from users, load balancers, and orchestration layers that it faces from a security eval. If it can’t finish in time, it doesn’t matter how correct the answer would have been.
The GPU That “Supports” Your Model Might Not Be Running It
This finding should follow you into your next infrastructure conversation.
We discovered multiple cases where the GPU wasn’t running the model — and nothing said so. The evaluation results looked like security findings. They were infrastructure artifacts.
A runtime version mismatch on one node caused the GPU backend to silently fall back to CPU inference. The model scored 14% — what looked like a damning result. After we upgraded the runtime, the same model on the same hardware scored 96.4%. The model was never unsafe. The deployment was broken. Nothing raised an error.
On a second node, a model exceeded available VRAM and spilled into system RAM. That overflow didn’t stay contained — subsequent models on the same node inherited degraded performance and produced timeout failures that looked behavioral. One model’s capacity violation contaminated every model that followed until a full restart cleared the state.
On a third node, one model family ran at confirmed GPU speed while another family of comparable size silently fell back to CPU on the same card. The backend accelerates selectively by model architecture — it doesn’t uniformly support everything that will technically load. No vendor documentation we reviewed made that distinction.
None of these produced an error message. All of them produced numbers that any team, without substrate-level instrumentation, would have reported as findings about model behavior.
Bigger Model, Better Security? The Data Says No.
We tested the 72B variant of a model family on hardware with 32GB of VRAM. It couldn’t load at any reasonable quantization. It silently timed out on every security test and scored 0% across the board.
A security dashboard monitoring those results would show a model that failed everything. A reasonable team might interpret that as “this model is fundamentally unsafe” and escalate accordingly. But the model never ran. It couldn’t fit on the hardware. The zeros don’t mean “unsafe” — they mean “never executed.” Nothing in the output distinguishes the two.
Meanwhile, the 32B variant of the same family — smaller, less expensive to deploy — was the only model in the entire evaluation to pass the adversarial routing test.
The assumption driving most model procurement conversations is that scaling up delivers better security behavior. Our data shows the opposite risk: scale up without verifying capacity, and the model silently doesn’t run, producing failures that masquerade as behavioral findings. You don’t get a safer model. You get a broken deployment wearing the mask of a dangerous one.
What to Do About It
- Evaluate on your hardware. Vendor benchmarks tell you what happened on their GPUs, with their runtime, on that day. They don’t predict yours. The variables that change outcomes are real, silent, and specific to your deployment.
- Instrument the substrate. Capture VRAM allocation, GPU utilization, and runtime version alongside every evaluation. Without that data, you can’t distinguish an infrastructure failure from a behavioral failure — and you’ll draw the wrong conclusion every time.
- Separate your failure modes across the CIA triad. A crash (availability), a wrong routing decision (integrity), and a leaked system prompt (confidentiality) all show “fail” in a pass/fail report. They need different remediation. Your evaluation framework should tell you which pillar you’re defending.
- Treat throughput as a security metric. A 7x speed gap between backends means the same test can pass or fail based on hardware alone. If your eval has a timeout — and in production, everything has a timeout — throughput is part of your security posture.
- Test before you scale. Deploying a larger model for better security is a reasonable instinct. Confirm it fits and runs on your target hardware first. Silent VRAM overflows produce failures that look like findings.
- Measure the noise floor before comparing configurations. Non-deterministic workloads produce non-deterministic test results. If you don’t know your single-backend variance, you can’t distinguish a real behavioral difference from a statistical coin flip.
Let’s Talk About Your Stack
This is what we do.
The findings above came from the same methodology we apply to enterprise AI deployments. We evaluate model security on your infrastructure, with your constraints, against the frameworks that matter.
- AI security assessments. Per-model, per-backend evaluation on your hardware. You get a failure-mode taxonomy mapped to the CIA triad, throughput analysis, and infrastructure verification — not a pass/fail score from someone else’s lab.
- Private AI deployment architecture. Zero-trust inference infrastructure with VLAN isolation, API chokepoint design, observability, and VRAM-aware model sequencing. Built from the same deployment challenges documented here.
- Ongoing AI security evaluation. Model behavior is non-deterministic. Runtime updates change outcomes. Security posture is not a point-in-time audit. We build continuous evaluation pipelines that catch behavioral drift and silent infrastructure failures before they reach production.
The framework behind this research — Hermia — is open source and on PyPI. Run it yourself. And when the results raise questions you’d rather not answer alone, give us a call.
To try the latest version of Hermia go to Github.
*Based on research originally published at scottbly.com
