An interactive walk-through of the OWASP GenAI Incident Response Guide, designed for everyone.
What is an AI Incident?
Think of AI as a brilliant but very literal-minded new employee. It’s amazing at its job, but it can be tricked, make weird mistakes, or be used by pranksters (and criminals). An “AI Incident” is the fire drill plan for when the AI does something unexpected or harmful.
It’s less about killer robots and more about preventing costly, embarrassing, or harmful mistakes from happening in the first place.
The Overly-Helpful Chatbot
An Air Canada chatbot invented a refund policy, forcing the airline to honor it. This wasn’t a hack; the AI’s own “hallucination” caused a financial loss.
The Chatbot That Learned Too Much
Microsoft’s “Tay” chatbot was designed to learn from Twitter. Trolls taught it to be racist and hateful in under a day, a classic case of “Data Poisoning.”
The Incident Response Lifecycle
Responding to an AI incident follows a clear path. Click on any stage to learn more about it.
Prepare
Laying the groundwork.
Detect
Spotting the trouble.
Respond
Taking action.
Key AI Threats to Understand
AI has unique vulnerabilities. Click on a threat to see what it means in simple terms.
Phase 1: Prepare
Preparation is about knowing what you have and who’s in charge before an incident occurs. You can’t protect what you don’t know about.
Create an AI Inventory
The first step is to simply list all AI systems in use. For each one, ask:
- What does it do?
- How critical is it to the business?
- What data does it access?
- Who is the designated owner?
Assemble Your AI Response Team
An AI incident requires new skills alongside traditional roles. Your team should include:
- Data Scientist: The “AI Whisperer” who can diagnose the model.
- ML Engineer: The mechanic who can fix the AI pipeline.
- Ethics Advisor: The conscience ensuring a responsible response.
- Plus: Security, Legal, IT, and PR.
Phase 2: Detect
Detecting an AI incident means looking for new kinds of clues. You need to monitor the *conversation* between users and the AI, not just network traffic.
A key part of detection is understanding the top risks. The OWASP Top 10 for LLMs highlights the most common vulnerabilities to watch for.
Monitor Prompts
Look for strange user inputs, repeated “jailbreak” attempts, or questions designed to trick the AI.
Analyze Outputs
Is the AI revealing secrets, generating harmful content, or behaving erratically? Sudden changes are a red flag.
Watch Resources
A sudden spike in cloud costs or processing power can indicate a malicious, resource-hogging prompt.
Phase 3: Respond
When an incident is confirmed, a swift and structured response is critical to minimize damage. This involves assessing severity, containing the problem, and recovering safely.
Assess Severity & Contain
First, determine the blast radius. How bad is it? Then, contain the problem immediately.
- Isolate: Take the AI model offline if possible.
- Block: Block the malicious user or IP address.
- Rollback: Switch to a previous, known-good version of the model.
Eradicate & Recover
Once contained, remove the root cause and restore service safely.
- Cleanse Data: If data was poisoned, find and remove the bad data, then retrain.
- Patch: Improve guardrails and fix the vulnerability that was exploited.
- Validate & Monitor: Test extensively to ensure the fix works, and monitor closely after relaunch.