[HN Gopher] N-Day-Bench - Can LLMs find real vulnerabilities in ...
       ___________________________________________________________________
        
       N-Day-Bench - Can LLMs find real vulnerabilities in real codebases?
        
       N-Day-Bench tests whether frontier LLMs can find known security
       vulnerabilities in real repository code. Each month it pulls fresh
       cases from GitHub security advisories, checks out the repo at the
       last commit before the patch, and gives models a sandboxed bash
       shell to explore the codebase.  Static vulnerability discovery
       benchmarks become outdated quickly. Cases leak into training data,
       and scores start measuring memorization. The monthly refresh keeps
       the test set ahead of contamination -- or at least makes the
       contamination window honest.  Each case runs three agents: a
       Curator reads the advisory and builds an answer key, a Finder (the
       model under test) gets 24 shell steps to explore the code and write
       a structured report, and a Judge scores the blinded submission. The
       Finder never sees the patch. It starts from sink hints and must
       trace the bug through actual code.  Only repos with 10k+ stars
       qualify. A diversity pass prevents any single repo from dominating
       the set. Ambiguous advisories (merge commits, multi-repo
       references, unresolvable refs) are dropped.  Currently evaluating
       GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro, GLM-5.1, and Kimi K2.5.
       All traces are public.  Methodology:
       https://ndaybench.winfunc.com/methodology  Live Leaderboard:
       https://ndaybench.winfunc.com/leaderboard  Live Traces:
       https://ndaybench.winfunc.com/traces
        
       Author : mufeedvh
       Score  : 14 points
       Date   : 2026-04-13 21:54 UTC (1 hours ago)
        
 (HTM) web link (ndaybench.winfunc.com)
 (TXT) w3m dump (ndaybench.winfunc.com)
        
       | Rohinator wrote:
       | Very curious how Claude Mythos will perform here
        
       | mbbutler wrote:
       | It would be helpful to add in some cases that do not contain any
       | vulnerabilities to assess false-positive rate as well.
        
         | mufeedvh wrote:
         | This is a good idea.
         | 
         | Will incorporate false-positive rates into the rubric from the
         | next run onwards.
         | 
         | At winfunc, we spent a lot of research time taming these models
         | to eradicate false-positive rates (it's high!) so this does
         | feel important enough to be documented. Thanks!
        
         | cortesoft wrote:
         | Any code that is certain that it doesn't have any
         | vulnerabilities is going to be pretty trivial to verify.
        
       ___________________________________________________________________
       (page generated 2026-04-13 23:00 UTC)