A Complete Guide to AI Red-Teaming With Garak

If you're deploying LLMs in production, you need to know how they fail before your users find out. That's where red-teaming comes in.

Garak—short for Generative AI Red-teaming and Assessment Kit—is an open-source framework that checks if an LLM can be made to fail in ways you don't want.[reference:72][reference:73] It probes for hallucinations, data leakage, prompt injection, misinformation, toxicity generation, jailbreaks, and many other weaknesses.[reference:74] Originally developed by Leon Derczynski's group, Garak is now backed by NVIDIA.[reference:75]

What Garak Does

Garak operates by running probe batteries against a target model[reference:76]:

• Prompt injection probes test whether an LLM can be tricked into ignoring its system instructions
• Jailbreak probes attempt to bypass safety filters and elicit harmful or restricted content
• Data leakage probes check if the model inadvertently reveals training data or sensitive information
• Hallucination probes assess whether the model fabricates information

Garak needs to know what model to scan. By default, it will try all the probes it knows on that model, using the vulnerability detectors recommended by each probe.[reference:77]

Getting Started With Garak

Garak works with both local models and remote endpoints[reference:78]. For deployed applications, you can point Garak at a REST endpoint rather than just the base model—which is critical because many vulnerabilities only appear when the model is integrated with tools and context.[reference:79]

Step 1: Installation

Step 2: Basic Scan

Step 3: Targeted Probing

Advanced Red-Teaming Patterns

For best results, use different models for different roles[reference:80]:

• Red-team attacker: One model generates attack prompts
• Target: The model being tested
• Detector judge: A third model evaluates whether the attack succeeded

Using the same model for multiple roles can lead to false negatives—the model may fail to recognize its own vulnerabilities.

Garak also supports multi-turn attacks through integration with frameworks like PyRIT, enabling more sophisticated red-teaming scenarios that test how models behave over extended conversations.[reference:81]

The Red-Teaming Workflow

A 2026 playbook for LLM red-teaming describes it as three loops: probe, classify, triage.[reference:82]

1. Probe: Generate attacks using Garak's probe library
2. Classify: Label model responses to identify which attacks succeeded
3. Triage: Prioritize vulnerabilities based on severity and fix them

This creates a continuous feedback loop where you're constantly testing and improving your model's defenses.

Why Garak Matters

Garak is the only mature open-source probe library for LLM security.[reference:83] In an era where AI systems are increasingly integrated into critical infrastructure, the ability to systematically test for vulnerabilities isn't optional—it's essential.

For engineering teams, Garak provides a way to move from reactive security (responding to incidents) to proactive security (finding problems before they're exploited). The framework's open-source nature means the community is constantly adding new probes as new attack vectors are discovered.

Whether you're building a customer service chatbot, a coding assistant, or an autonomous agent, Garak should be part of your pre-deployment checklist. The question isn't whether your model has vulnerabilities—it's whether you know what they are before someone else finds them.