Most AI security evaluations test the model. They don’t test the stack — the hardware, drivers, quantization, and routing decisions sitting underneath it. That gap is exactly what Scott Bly, SIS Cyber Director of Security Technologies, stumbled into while trying to cut cloud costs by running local AI models, and it led him to build Hermia, an open-source LLM security evaluation framework now available under an MIT license.
In this conversation with Dan Barahona, Scott breaks down what happened when he started testing models across full inference stacks instead of in isolation. The findings challenge some common assumptions:
- Size doesn’t equal safety. A 27B parameter model failed a test that a 14B model passed.
- Answer keys don’t guarantee correct behavior. In an adversarial routing scenario, models given the correct answer still failed 95% of the time.
- There are two distinct failure modes under pressure — and one of them can crash your application outright.
- The same model can behave completely differently depending on hardware, drivers, and quantization settings — sometimes swinging from 96% to 14% success on identical tests.
- The scale of the problem is enormous: roughly 15 trillion possible inference stack combinations, each a potential source of silent behavioral drift.
Scott walks through Hermia’s design, demos its terminal interface, and explains the 28 tests it runs to surface these issues — plus where the framework is headed next.
Try it yourself:
Hermia is free and open-source. Install with pipx install hermia, or explore the code on GitHub.
