LLM Vulnerability Scanners Compared: What to Look For
LLM vulnerability scanners compared: what Garak, Promptfoo, PyRIT and DeepTeam actually cover, their limits, and how to choose one for agent security.
An LLM vulnerability scanner automates part of the adversarial testing an AI system needs before it ships — sending crafted probes at a model or agent and reporting which ones succeed. Used well, a scanner turns a slow, manual red-team exercise into a repeatable step you can run on every deployment. Used naively, it produces a green dashboard that hides everything the tool cannot see. The difference is understanding what these scanners actually cover, and what they structurally miss.
This post compares the leading open-source LLM vulnerability scanners by capability rather than by hype, explains the evaluation criteria that matter, and is honest about the gap between automated scanning and a real assessment. For a broader tool comparison across categories, see our AI red teaming tools comparison. For pricing on a full engagement, see the red teaming pricing guide.
What a Scanner Can and Cannot Do
An LLM vulnerability scanner is a probe engine. It maintains a library of adversarial inputs — jailbreaks, injection payloads, encoding tricks, toxicity elicitation — and a way to judge whether the model’s response constitutes a failure. Some judge with heuristics, some with a classifier, some with another model. That is the whole mechanism, and it is genuinely useful.
What a scanner does well: broad, repeatable coverage of known attack classes; regression testing so a fix stays fixed; fast triage that catches the obvious problems before a human spends time on the subtle ones. What a scanner cannot do: reason about your specific business logic, chain a multi-step exploit across tools, understand that leaking one particular internal document is catastrophic while another is harmless, or invent a novel attack that is not in its payload library. Scanners find classes of vulnerability; humans find your vulnerability.
Treat the scanner as the automated first pass — Promptfoo’s own documentation suggests automated scanning catches roughly 60–70% of what manual testing would find. The remaining 30–40% is where breaches live, and it is exactly the part a dashboard will not warn you about.
The Leading Scanners
These are widely used, actively maintained open-source tools. Capabilities evolve, so treat this as a starting map, not a frozen spec — always confirm current coverage in each project’s documentation.
NVIDIA Garak. A probe-based scanner from NVIDIA, structured around probes (attack generators) and detectors (failure classifiers). Strong on a broad catalogue of known vulnerability classes — jailbreaks, prompt injection, toxicity, data leakage, encoding attacks — and easy to point at a model endpoint for a quick baseline. Best as a fast, model-level first sweep.
Promptfoo. A red-teaming and evaluation CLI with an owasp and owasp:agentic preset that maps probes to the OWASP LLM and agentic Top 10. Its strength is configuration-as-code: you declare your system, providers, and test suites in YAML and re-run them in CI. Good coverage across agentic risk categories and strong for regression testing and comparison across models.
Microsoft PyRIT. The Python Risk Identification Toolkit, built by Microsoft’s AI Red Team. More framework than turnkey scanner — it gives you orchestrators, converters, and scorers to build automated attack pipelines, including multi-turn attacks. Best when you need to script sophisticated, adaptive attack flows rather than run a fixed catalogue.
DeepTeam (Confident AI). An open-source framework covering a broad set of agentic vulnerability types and multiple attack methods, including multi-language attack generation. Useful for agentic-specific coverage and for teams already using its evaluation ecosystem.
Giskard. An AI testing platform with prompt-injection detection and RAG evaluation. Strong where retrieval pipelines are the risk surface, and integrates evaluation with security probing.
Each of these appears in our practical OWASP agentic testing guide alongside the specific ASI risks they help cover.
Evaluation Criteria That Actually Matter
When choosing a scanner, the marketing feature list is less useful than these questions.
Attack coverage and taxonomy mapping. Does the tool map its probes to a recognised framework — OWASP LLM Top 10, OWASP agentic Top 10, MITRE ATLAS? Framework mapping is what makes results defensible to an auditor and comparable across tools. A scanner with 500 probes and no taxonomy is harder to reason about than one with fewer probes mapped to ASI01–ASI10.
Agent-awareness. Can it test tool-using agents, or only single-turn model prompts? Many scanners were built for chatbots. Agentic risks — tool poisoning, privilege abuse, multi-agent failures — need a tool that can drive the agent through real tool calls, not just prompt the model. This is the single biggest differentiator for agent security.
Detection quality. How does it judge success or failure, and what are its false-positive and false-negative rates? A model-based judge is flexible but can be fooled; a heuristic detector is deterministic but brittle. Understand the judge, because it decides what “pass” means.
Multi-turn and adaptive attacks. Real attacks are conversations, not one-shot prompts. Can the tool sustain a multi-turn attack that builds context before striking? PyRIT and DeepTeam lead here; single-turn scanners will miss anything that requires setup.
CI/CD integration. Can you run it as code, in a pipeline, on every deploy, with results diffed against a baseline? Security testing you run once is worth far less than testing that runs on every change and blocks regressions.
Extensibility. Can you add probes for your own business logic and threat model? Off-the-shelf payloads never cover your specific risks. The best tools let you extend the catalogue.
Reporting. Does it produce output you can act on — reproduction steps, severity, and remediation pointers — or just a pass/fail count? A count is triage; reproduction steps are what engineering needs to fix the finding.
Building a Scanner Stack
No single scanner is complete, so mature teams combine them. A practical stack:
- Fast baseline with Garak against the model endpoint to catch obvious model-level weaknesses.
- Framework-mapped coverage with Promptfoo’s
owasp:agenticpreset, run in CI, diffed against a baseline on every deploy. - Adaptive, multi-turn campaigns with PyRIT or DeepTeam for the attack classes that need setup and persistence.
- RAG-specific probing with Giskard where retrieval is a primary surface.
- Custom probes encoding your own business logic and highest-value targets.
Run the first two continuously; run the deeper campaigns periodically and after major changes. Then — and this is the part no scanner replaces — layer human red teaming on top for the exploit chains automation cannot reason about.
Where Scanners End and Assessment Begins
The honest limit of any scanner is that it tests what someone already thought to encode. The vulnerabilities that cause real incidents are usually specific to your architecture: an over-provisioned tool that turns a minor injection into a database leak, a confused-deputy chain across two MCP servers, a business-logic flaw where the agent is technically behaving correctly and still causing harm. These are found by humans who understand your system, using the scanner output as a starting point rather than a conclusion.
That is why a professional assessment combines automated scanning for breadth with manual red teaming for depth. The scanner tells you that you are not obviously broken; the human tells you whether you are broken.
FAQ
What is an LLM vulnerability scanner? It is a tool that automatically sends adversarial inputs — jailbreaks, prompt injections, encoding attacks, and more — at a language model or agent and reports which succeed. It automates broad, repeatable coverage of known attack classes but cannot reason about your specific architecture or invent novel exploits.
Which LLM vulnerability scanner is best? There is no single best tool; they are complementary. Garak is strong for a fast model-level baseline, Promptfoo for framework-mapped coverage in CI, PyRIT and DeepTeam for adaptive multi-turn attacks, and Giskard for RAG pipelines. Choose based on whether you need model-level or agent-level testing, and combine tools for real coverage.
Can a scanner replace a red team? No. Automated scanning typically catches around 60–70% of what manual testing finds — the known, encodable attack classes. The remaining 30–40%, where most real incidents originate, requires human red teamers who understand your business logic and can chain multi-step exploits. Scanners are the first pass, not the whole assessment.
How often should I run a scanner? Continuously for the fast, framework-mapped suites — ideally in CI on every deploy, with results diffed against a baseline so regressions are caught immediately. Run deeper, adaptive campaigns periodically and after any major change to the model, tools, or architecture.
Get Beyond the Dashboard
A green scanner dashboard is a starting point, not an assurance. Our assessments run a multi-tool automated stack for breadth, then layer human red teaming for the exploit chains scanners cannot reason about — all mapped to OWASP and NIST AI RMF and delivered as an audit-ready report.
Want help choosing and integrating tools? Book a 30-minute call, email security@aivyuh.com, or see our enterprise engagement models.
Related reading: