1/ Last month I poisoned four of the strongest AI coding agents — 288 runs. The worst one got hijacked 54.2% of the time.
So I built the gate I wanted afterwards: vibe-guardian, a free open-source scanner that blocks the vulnerabilities AI ships — and calls zero LLMs.
Both halves 🧵
2/ The lineup, each in its native agent framework:
• Opus 4.8 (Claude Code) • GPT-5.5 (Codex) • DeepSeek-V4-Pro (opencode) • GLM-5.2 (opencode)
Indirect prompt injection: the payload never touches my prompt. It's buried in the stuff the agent reads while working.
3/ Base-defense hijack rates:
• DeepSeek-V4-Pro: 54.2% 🔴 • GPT-5.5: 8.3% • Opus 4.8: 0%* • GLM-5.2: 0%*
*0% = observed within the confidence interval, NOT "proven zero."
The single worst carrier: dependency docs (29.2%), ahead of fake error messages (16.7%). The agent can't tell your words from an attacker's.
4/ But injection is only half the danger.
The other half is quieter and far more common: the AI doesn't need to be tricked. It just ships the vulnerability itself — because vibe coding is fast and careless.
5/ The three it writes constantly:
• Hardcoded secrets → pushed to GitHub, scraped within minutes • String-concatenated SQL → one crafted input dumps your table • Supabase tables with no RLS → anyone with your frontend reads/writes everything. service_role key in client code = your master key taped to the front door.
6/ Here's the thing both experiments taught me.
In the injection test, what stopped the hijacks wasn't a smarter model. It was ONE deterministic rule + an opt-out. That alone crushed 54.2% → 29.2%, and 8.3% → 0%.
Lesson: don't trust the model's judgment. Add a gate that doesn't depend on it.
7/ So I built that gate: vibe-guardian.
Two checkpoints: • While the AI writes (Claude Code plugin): every file scanned instantly; on a finding, the AI is told what's wrong and fixes it on the spot. • At commit (git pre-commit hook): vulnerable code never enters your history.
8/ Real scan, not a mockup. I wrote the kind of file AI generates all the time:
3 errors (hardcoded OpenAI key, string-built SQL, service_role in client) → commit BLOCKED. 3 warnings (2 tables missing RLS, innerHTML XSS) → flagged, not blocked.
Only high-confidence rules block. No alert fatigue, no --no-verify temptation.
9/ The irony I love most:
A tool built to defend against unreliable AI calls zero LLMs. 100% local regex, zero dependencies, millisecond scans, your code never leaves your machine.
Sometimes the best defense against careless AI is to not let any AI into the gate at all.
10/ Four things worth pinning:
- Treat all AI-written code as unsafe until it passes a gate
- Secrets go in env vars, never source
- If you don't turn on RLS, no one does it for you
- The gate must be a dumb machine, not another AI that can be talked around
11/ Honest limits: vibe-guardian catches vulnerable code the AI writes — it does NOT stop prompt injection itself (different battle). It's a first gate, not a replacement for full SAST. The injection test compares model + native framework, not raw models; different attacks → different numbers.
12/ It's open source and free: → github.com/wbw20000/vibe-guardian → or just: npx vibe-guardian scan
Full injection write-up — 288 runs, the worst carrier (dependency docs), the worst instruction class (phone-home, 87.5% on DeepSeek base), and the one guardrail line that crushed 54.2% → 29.2%: → crawdpad.com/library/x/ai-coding-agent-prompt-injection