Post B5AnF35QePLEekNhPU by zhuowei@notnow.dev
 (DIR) More posts by zhuowei@notnow.dev
 (DIR) Post #B5AnF35QePLEekNhPU by zhuowei@notnow.dev
       0 likes, 0 repeats
       
       With all this hype about upcoming AIs that can hack anything, I decided to see how well LLMs can audit code for security issues today.I downloaded an older version of a codebase I’ve been researching. It’s a well-written modern C codebase, and I chose an older version with known vulnerabilities.Then I installed an LLM coding tool. (Gemini 3.1 Pro (High) in Antigravity, if you’re wondering)I asked the LLM:Please find potential security issues in the <project> directoryAnd after a few minutes, it replied with:a few out-of-bound reads - which I’ve already found myself with fuzzing, two spec violations, which the LLM claimed makes the codebase vulnerable to a well-known bug.That well-known bug was indeed present in this version of this code, but these two spec violations don’t seem to be part of that bug. The later version of this codebase - which patches that well known bug - didn’t change them.a non-reentrant function - it doesn’t run concurrently anyways (this codebase runs on a single thread event loop)I then asked:Are there any out-of-bound writes?After another minute, it found one:A call that incorrectly passed an ID instead of an array index. This was a real security issue, since it was patched in the later version of this codebase.I asked again:Good catch. Are there any other issues?It then gave me another two:A stack overflow that seems genuine. However, it’s not patched in the later version of the codebase, so I can’t tell if the AI is wrong, or if it found a 0-day.The same issue above, but reached with an integer underflow instead of a large inputSo…2 definite issues2 kinda-issues with a wrong explanation1 issue that’s not a security issue but is not good practice1 issue that might be a 0-day or might be completely wrong.That’s honestly way better than I expected - there wasn’t a flood of false positives I had to wade through. And this is with a current generation LLM.So… yeah, I can see why people are hyped. But still:Why did I have to keep telling it to continue looking?The LLM wasn’t able to check by itself if its potential issues are genuine.Project Zero BigSleep and Claude Mythos/Glasswing both gave the AI agent a harness where it can write code to test its findings and see if they’re real issues.Is the future of security researchers just writing harnesses for GenAI, the same way everyone wrote harnesses for fuzzers back when the first coverage-based fuzzers came out?(sha256 of the conversation transcript is 36a9508bc06bb0d10d67294fabe0f46f3e71b676662fcf57e40956674c8428e5 - I’ll post the whole transcript once I check whether it actually found a real unpatched issue)
       
 (DIR) Post #B5AnF3STGjlzoDA6KG by astraleureka@social.treehouse.systems
       0 likes, 0 repeats
       
       @zhuowei >Why did I have to keep telling it to continue looking?the Claude docs show that it has a fairly low limit of tool invocations per prompt, and has to be told to continue. I guess maybe so it doesn't get stuck in endless tool exec loops or blow out the token budget for "thinking" modes?