Vibe Security Radar

About Vibe Security Radar

AI coding tools are writing a growing share of production code. Some of it ships with security vulnerabilities. We track the cases where vulnerable code in public advisories (CVEs, GHSAs, RustSec, and others) was authored by an AI tool.

This is a research project from Georgia Tech SSLab (Systems Software & Security Lab, School of Cybersecurity and Privacy). Our goal is to understand how AI-assisted development affects software security, not through benchmarks or synthetic tasks, but by studying real vulnerabilities that were reported and fixed in the wild.

78 AI-linked vulnerabilities tracked46,831 advisories analyzed

Coverage: May 1, 2025 – Mar 24, 2026

Core methodology

Our approach has three steps: find the fix, trace the blame, and verify the cause.

1. Find the fix commit

We aggregate vulnerability data from four advisory databases (OSV, GitHub Advisory Database, Gemnasium, NVD) and extract the commit that fixed each vulnerability. Over 95% of lookups use local bulk data, with no API calls required.

2. Trace who introduced the bug

Using SZZ-style git blame on the fix commit, we trace backward to the commit that introduced the vulnerable code. Squash-merged PRs are decomposed into individual sub-commits so attribution is per-commit, not per-PR. Each bug-introducing commit is then scanned for AI tool signatures: co-author trailers, bot emails, and commit message markers from 54+ tools.

3. Verify causality

An AI signature in a commit is not enough. First, a screening pass checks whether the blamed commit is plausibly related to the security issue. Then a deep investigator with full git tool access examines the entire vulnerability (all fix commits, all blame candidates), running up to 50 tool calls per case. Rather than rating each commit in isolation, it answers one question: did AI-authored code contribute to causing this vulnerability? This catches patterns that per-commit analysis misses: an AI commit that altered a calling convention, making previously safe code exploitable, or a squash-merge where the AI-tagged sub-commit never touched the vulnerable file. If the primary model fails, a Claude Agent SDK fallback with independent repository access retries the investigation. The pipeline is conservative: attribution is dropped when causality is uncertain.

Pipeline details

Click any phase to see the algorithm in detail.

  1. Phase A

    Fix commit discovery

    ▸ details

    Pull fix commit SHAs from advisory databases and reference URLs, with LLM-assisted search as a last resort.

    Advisory sources are queried in priority order: OSV bulk data and the local GitHub Advisory Database clone first (no API calls), then Gemnasium DB commit URLs and fixed-version tag resolution, followed by NVD reference URL parsing for GitHub commits and PRs. If none yield a fix commit and LLM mode is enabled, two additional strategies are attempted: a multi-version tag search that uses an LLM to rank candidate commits between version tags, and a description-based search that extracts terms from the CVE description and scores git log matches. Earlier tiers short-circuit later ones.

  2. Phase B

    Bug-introducing commit discovery

    ▸ details

    Clone the repo, diff each fix commit, and run SZZ-style git blame to trace who introduced the vulnerable code.

    Four blame strategies run in parallel on each fix commit: (1) Deleted-line blame: when a fix removes or modifies code, blame those lines to find who wrote them (strongest causal signal). (2) Context blame: for add-only fixes, blame the surrounding lines in the parent commit. (3) Function history: when context blame finds nothing, trace function-level history via git log -L. (4) Pickaxe search: when vulnerability analysis identifies a dangerous pattern (e.g., a specific function call), search git history with git log -S to find who first introduced it. Before blaming, an LLM identifies which files in the fix are security-relevant, so blame runs only on those rather than every file the fix touched.

  3. Phase B+

    Squash-merge decomposition

    ▸ details

    Large squash-merge commits get broken apart via the GitHub API to find which specific sub-commit introduced the vulnerable code.

    When a fix or bug-introducing commit is a squash-merge (detected by the (#NNN) pattern in the commit message) with over 30 changed files, the pipeline fetches the original PR sub-commits via the GitHub API. Each sub-commit is scored by file relevance to the CVE. For fix commits, only CVE-relevant files are blamed instead of all 1000+ files. For bug-introducing commits, the pipeline identifies which sub-commit actually touched the blamed file and records it as the culprit. This matters for AI attribution: if a PR has 17 commits and only one carries a Copilot co-author trailer, but that commit never touched the vulnerable file, the AI signal is dropped.

  4. Phase C

    AI signal detection

    ▸ details

    Check each bug-introducing commit for AI coding tool signatures. CI/CD bots are filtered out.

    The pipeline scans commit metadata for co-author trailers (e.g., Co-Authored-By: Copilot), author and committer email domains, commit message keywords, and tool-specific patterns. Associated PR bodies are also checked for attribution text. For squash-merged BICs, individual sub-commits are inspected separately and confidence is scaled by the proportion of AI-authored commits in the PR. A lone Copilot commit in a 20-commit PR yields lower confidence than one where every commit carries AI signatures. Known CI/CD bots (Dependabot, Renovate, GitHub Actions, etc.) are excluded, and an anachronism filter rejects signals where the commit predates the tool's public release.

  5. Phase D

    Screening verification

    ▸ details

    A lightweight LLM screen decides whether AI-signaled commits are plausibly related to the vulnerability, gating the expensive deep investigation.

    Rather than sending every AI-signaled CVE to the deep investigator, a lightweight LLM screen runs first. It receives the vulnerability description, the fix diff, and a summary of all AI-signaled bug-introducing commits (including decomposed sub-commits and blamed files). The screener asks one question: could any of these AI commits have contributed to this vulnerability? If not (say the AI commits touched frontend auth code but the vulnerability is a backend SSRF), the CVE is filtered out without incurring a full investigation. CVEs that pass proceed to Phase E. This filter achieves ~80% precision (validated by independent audit of rejected cases) and catches cases where blame cast a wide net and happened to land on unrelated AI-authored code.

  6. Phase E

    Deep investigation

    ▸ details

    A single LLM investigator sees the entire vulnerability at once, runs a multi-step investigation with tool access, and answers a CVE-level question: did AI-authored code contribute to introducing this vulnerability?

    The investigator receives all fix commits, all blame candidates, and the CVE description. It has access to git log, file read, blame, diff, and pickaxe search, running up to 50 tool calls per investigation. It can trace chains of related commits, follow code across renames, and discover bug-introducing commits that blame missed entirely. Rather than rating individual commits, it answers a vulnerability-level question: was AI-authored code part of the causal chain? This verdict determines what appears on the site. Per-commit assessments are retained as supporting evidence but do not drive the final call.

  7. Phase F

    Fallback verification

    ▸ details

    When the primary deep investigator fails (timeout, model error), a Claude Agent SDK subprocess with git MCP tools retries the investigation independently.

    The deep investigator (Phase E) uses a model fallback chain. When the primary model exhausts its tool-call budget or errors out, the pipeline falls back to a Claude Agent SDK subprocess with its own git tools (log, blame, diff, file read) via MCP. This is a fundamentally different execution path (a full CLI subprocess rather than API calls), so it often succeeds where the primary model failed. If the SDK fallback also fails, remaining models in the chain are tried. The fallback verdict replaces the failed investigation.

How we attribute vulnerabilities to AI

An AI signature in a bug-introducing commit is not enough. We ask the question at the vulnerability level: did AI-authored code help cause this? That could mean the AI wrote the vulnerable lines directly, altered a calling convention that made existing code exploitable, or added a feature without the security checks it needed. If the AI commits were not part of the causal chain, we drop the attribution.

This matters especially for squash-merged PRs. Suppose a PR has 20 commits and one carries a Copilot co-author trailer, but that commit only updated a README, while a different human-written commit in the same PR introduced the vulnerability. We check file-level overlap between each sub-commit and the blamed file, and the deep investigator independently verifies whether AI-authored code was actually part of the causal chain.

AI tools monitored

We detect signatures from 54 AI coding tools via co-author trailers, bot email addresses, and commit message markers. CI/CD bots (Dependabot, Renovate, etc.) are filtered out.

Data sources

Limitations & what we miss

This is an observational study with inherent blind spots. We are transparent about what we can and cannot measure.

Detection blind spots

  • Our detection relies entirely on metadata signals: co-author trailers, bot emails, commit message markers. Code written with AI assistance but committed without these markers is invisible to us. This is the single biggest limitation: many developers use AI tools in ways that leave no trace (copy-pasting from ChatGPT, using tools that don't add co-author trailers, or stripping markers before pushing). Our numbers represent a strict lower bound on AI-linked vulnerabilities.
  • Different AI tools leave varying amounts of metadata. Claude Code and GitHub Copilot have well-established co-author conventions; others are harder to detect. This creates uneven coverage across tools.
  • We are developing LLM-based code fingerprinting to identify AI-generated code from its stylistic and structural patterns, independent of commit metadata. This would catch cases where AI involvement is evident from the code itself but leaves no trace in the git history.

Attribution accuracy

  • Git blame tracks line authorship, not semantic causality. A line may be blamed on commit X even when the real root cause is a design decision in commit Y. The deep investigator's CVE-level analysis catches most of these cases, but not all.
  • Squash-merge decomposition depends on the GitHub API returning sub-commits. Force-pushed PRs or rebased branches may lose the original commit history, making per-commit attribution impossible.
  • The investigator is a single LLM with tool access. Borderline cases where causality is genuinely ambiguous can go either way. We do not claim 100% accuracy on any individual case.

Coverage scope

  • We cover publicly disclosed vulnerabilities (CVEs, GHSAs, RustSec, etc.) in public repositories, including CI/CD configuration issues like GitHub Actions injection. When advisory databases lack a fix commit, the pipeline uses LLM-assisted search (version-tag ranking and description-based git log matching) to discover one. Roughly 10% of our confirmed cases rely on these LLM-inferred fix commits. Closed-source bugs and unpatched vulnerabilities remain out of scope.
  • Our analysis starts from May 2025. Vulnerabilities disclosed or fixed before that date are not covered, even if AI tools were involved.
  • We do not analyze whether AI tools are more or less likely to introduce vulnerabilities than human developers. This project measures incidence, not relative risk.

Methodological constraints

  • Our approach is inherently retrospective: we find AI-authored vulnerabilities after they are reported and fixed. We cannot predict which AI-generated code will become vulnerable.
  • The pipeline is conservative by design: we would rather miss a true positive than report a false positive. This means our count underestimates the real number of AI-linked vulnerabilities.
  • We use LLMs to judge whether AI-authored code caused a vulnerability, which creates an inherent circularity. The investigator may have systematic blind spots when evaluating AI-generated code patterns. We mitigate this with multi-model verification and conflict resolution, but cannot fully eliminate the risk.
  • We can audit for false positives (cases we flagged incorrectly), but have no systematic way to measure our false negative rate: how many AI-caused vulnerabilities we miss entirely. The true count is unknowable with metadata-only detection.

Contact

Found a false positive? Think we missed something? Have a question about our methodology? Email hanqing@gatech.edu.