[HN Gopher] Potential issues in curl found using AI assisted tools
___________________________________________________________________
Potential issues in curl found using AI assisted tools
Author : robhlam
Score : 353 points
Date : 2025-10-02 13:29 UTC (9 hours ago)
(HTM) web link (mastodon.social)
(TXT) w3m dump (mastodon.social)
| simonw wrote:
| Here are 55 closed PRs in the curl repo which credit "sarif data"
| - I think those are the ones Daniel is talking about here
| https://github.com/curl/curl/pulls?q=is%3Apr+sarif+is%3Aclos...
|
| This is notable given Daniel Stenberg's reports of being
| bombarded by total slop AI-generated false security issues in the
| past: https://www.linkedin.com/posts/danielstenberg_hackerone-
| curl...
|
| Concerning HackerOne: "We now ban every reporter INSTANTLY who
| submits reports we deem AI slop. A threshold has been reached. We
| are effectively being DDoSed. If we could, we would charge them
| for this waste of our time"
|
| Also this from January 2024:
| https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-f...
| octocop wrote:
| The models used have improved quite well since then, I guess
| his change of opinion shows that.
| simonw wrote:
| I think it's more about how people are using it. An amateur
| who spams him with GPT-5-Codex produced bug reports is still
| a waste of his time. Here a professional ran the tools and
| then applied their own judgement before sending the results
| to the curl maintainers.
| tptacek wrote:
| I keep irritating people with this observation but this was
| the status quo ante before AI, and at least an AI slop
| report shows clear intent; you can ban those submitters
| without even a glance at anything else they send.
| whizzter wrote:
| It's probably also the difference of idiots hoping to cash
| out/get credit for vulnerabilities by just throwing ChatGPT
| at the wall compared to this where it seems a somewhat
| seasoned researcher is trialing more customized tools.
| Twirrim wrote:
| No, he's still dealing with a flood of crap, even in the last
| few weeks, off more modern models.
|
| It's primarily from people just throwing source code at an
| LLM, asking it to find a vulnerability, and reporting it as-
| read, without having any actual understanding of if it is or
| isn't a vulnerability.
|
| The difference in this particular case is it's someone who
| is: 1) Using tools specifically designed for security audits
| and investigations. 2) Takes the time to read and understand
| the vulnerability reported, and verifies that it is actually
| a vulnerability before reporting.
|
| Point 2 is the most significant bar that people are woefully
| failing to meet and wasting a terrific amount of his time.
| The one that got shared from a couple of weeks ago
| https://hackerone.com/reports/3340109 didn't even call curl.
| It was straight up hallucination.
| tomjakubowski wrote:
| Some of those bugs, like using the wrong printf-specifier for a
| size_t, would be flagged by the compiler with the right warning
| flags set. An AI oracle which tells me, "your project is
| missing these important bug-catching compiler warning flags,"
| would be quite useful.
|
| A few of these PRs are dependabot PRs which match on "sarif", I
| am guessing because the string shows up somewhere in the
| project's dependency list. "Joshua sarif data" returns a more
| specific set of closed PRs.
| https://github.com/curl/curl/pulls?q=is%3Apr+Joshua+sarif+da...
| flohofwoe wrote:
| This is exactly what I'd want from an 'AI coding companion'.
|
| Don't write or fix the code for me (thanks but I can manage that
| on my own with much less hassle), but instead tell me which
| places in the code look suspicious and where I need to have a
| closer look.
|
| When I ask Claude to find bugs in my 20kloc C library it more or
| less just splits the file(s) into smaller chunks and greps for
| specific code patterns and in the end just gives me a list of my
| own FIXME comments (lol), which tbh is quite underwhelming - a
| simple bash script could do that too.
|
| ChatGPT is even less useful since it basically just spend a lot
| of time to tell me 'everything looking great yay good job high-
| five!'.
|
| So far, traditional static code analysis has been much more
| helpful in finding actual bugs, but static analysis being clean
| doesn't mean there are no logic bugs, and this is exactly where
| LLMs should be able to shine.
|
| If getting more useful potential-bugs-information from LLMs
| requires an extensively customized setup then the whole idea is
| getting much less useful - it's a similar situation to how static
| code analysis isn't used if it requires extensive setup or manual
| build-system integration instead of just being a button or menu
| item in the IDE or enabled by default for each build.
| walthamstow wrote:
| Cursor BugBot is pretty good for this, we did the free trial
| and it was so popular with our devs that we ended up keeping
| it. Occasional false positives aside, it's very useful. It
| saves time for both the PR submitter and the reviewer.
| simonw wrote:
| Suggestion: run a regex to remove those FIXME comments first,
| then try the experiment again.
|
| I often use Claude/GPT-5/etc to analyze existing repositories
| while deliberately omitting the tests and documentation folders
| because I don't want them to influence the answers I'm getting
| about the code - because if I'm asking a question it's likely
| the documentation has failed to answer it already!
| trenchpilgrim wrote:
| I use Zed's "Ask" mode for this all the time. It's a read only
| mode where the LLM focuses on figuring out the codebase instead
| of modifying it. You can toggle it freely mid conversation.
| flaviolivolsi wrote:
| I found GPT-5 to be very much less sycophantic than other
| models when it comes to this stuff, so your mention of
| 'everything looking great yay good job high-five' surprises me.
| Using it via Codex CLI it often questions things. Gemini 2.5
| Pro is also good on this.
| vidarh wrote:
| I've "worked" with Claude Code to find a long standing set of
| complex bugs over the last couple of days, and it can do so
| much more. It's come up with hypotheses, tested them, used gdb
| in batch mode when the hypotheses failed in order to trace what
| happened at the assembly level, and compared with the asm dump
| of the code in question.
|
| It still needs guidance, but it quashed bugs yesterday that
| I've previously spent many days on without finding a solution
| for.
|
| It can be tricky, but they definitely can be significant aid
| for even very complex bugs.
| Sharlin wrote:
| This is a point I see discussed surprisingly little. Given that
| many (most?) programmers like designing and writing code
| (excluding boilerplate), and not particularly enjoy _reviewing_
| code, it certainly feels backwards to make the AI write the
| code and relegate the programmer to reviewing it. (I know, of
| course, that the whole thing is being sold to stakeholders as
| "LoC machine goes brrrr" - code review? what's that?)
| alexchantavy wrote:
| There are a lot of good AI code reviewers out there where
| they learn project conventions based on prior PRs and make
| rules from them. I've found they definitely save time and
| catch things I would have missed - things like cubic.dev or
| greptile etc etc. Especially helpful for running an open
| source project where code quality can have high variance and
| as a maintainer you may feel hesitant to be direct with
| someone -- the machine has no feelings so it is what it is :)
| SAI_Peregrinus wrote:
| Creativity is fun. AIs automate that away. I want an AI that
| can do my laundry, fold it, and put it away. I don't need an
| AI to write code for me. I don't mind AI code review, it
| sometimes has a valid suggestion, and it's easy enough to
| ignore most of the rest of the time.
| diggan wrote:
| > Creativity is fun. AIs automate that away.
|
| I've been developing with LLMs on my side for months/about
| a year now, and feels like it's allowing me to be more
| creative, not less. But I'm not doing any "vibe-coding",
| maybe that's why?
|
| The creative parts (for me) is coming up with the actual
| design of the software, and how it all fits together, what
| it should do and how, and I get to do that more than ever
| now.
| dingnuts wrote:
| I'm still faster than the cheap bots.
|
| The creative part for me includes both the implementation
| and the design, because the implementation also matters.
| The bots get in the way.
|
| Maybe I would be faster if I paid for Claude Code. It's
| too expensive to evaluate.
|
| If you like your expensive AI autocomplete, fine. But I
| have not seen any demonstrable and maintainable
| productivity gains from it, and I find understanding my
| whole implementation faster, more fun, and that it
| produces better software.
|
| Maybe that will change, but people told me three years
| ago that we would be at the point today where I could not
| outdo the bot;
|
| with all due respect, I am John Henry and I am still
| swinging my hammer. The steam pile driving machine is
| still too unpredictable!
| r_lee wrote:
| Claude code is too expensive to evaluate?
|
| It's 20 bucks a month
| diggan wrote:
| > The creative part for me includes both the
| implementation and the design
|
| The implementations LLMs end up writing are predicable,
| because my design locks down what it needs to do. I
| basically know exactly what they'll end up doing, and
| how, but it types faster than I do, that's why I hand it
| off while I go on to think about the next design
| iteration.
|
| I currently send every single prompt to Claude, Codex,
| Qwen and Gemini (looks something like this:
| https://i.imgur.com/YewIjGu.png), and while the all most
| of the time succeed, doing it like this makes it clear
| that they're following what I imagined they'd do during
| the design phase, as they all end up with more or less
| the same solutions.
|
| > If you like your expensive AI autocomplete
|
| I don't know if you mean that in jest, but what I'm doing
| isn't "expensive AI autocomplete". I come up with what
| has to be done, the design for achieving so, then hand
| off the work. I don't actually write much code at all,
| just small adjustments when needed.
|
| > and I find understanding my whole implementation faster
|
| Yeah, I guess that's the difference between "vibe-coding"
| and what I (and others) are doing, as we're not giving up
| any understanding or control of the architecture and
| design, but instead focus mostly on those two things
| while handing off other work.
| grantWilliams wrote:
| Most software is developer tools and frameworks to manage
| electrical state in machines.
|
| Such state management messes use up a lot of resources to
| copy around.
|
| As an EE working in QA future chips with a goal of
| compressing away developer syntax art to preserve the least
| amount of state management possible to achieve maximum
| utility; sorry self selecting biology of SWEs, but also not
| sorry.
|
| Above all this is capitalism not honorific obligationism.
| If hardware engineers can claim more of the tech economy
| for our shareholders, we must.
|
| There are plenty of other creative outlets that are much
| less resource intensive. Rich first world programmers are a
| small subset of the population and can branch out then and
| explore life rather than believe everyone else has an
| obligation to conserve the personal story of a generation
| of future dead.
| collingreen wrote:
| I was thinking this again just yesterday. Do my laundry
| correctly and get it put away. Organized my storage. Clean
| the bathroom. Do the dishes. Catalog my pantry, give me
| recipes, and keep it correctly stocked. Maybe I'm just a
| simple creature but like, these are the obvious problems in
| my life I'll pay to have go away so why are we taking away
| the fun stuff instead?
| CuriouslyC wrote:
| Is AI automating creativity away if you come up with an
| idea and have it actually implement it?
| Sharlin wrote:
| Yes, because ideas are not worth much if anything. If you
| have an idea of a book, or a painting, and have someone
| else implement it, you have not done creative work.
| Literally, you have not _created_ the work, brought it to
| existence. The _creator_ has done the creativity.
| CuriouslyC wrote:
| I guess that depends on how much oversight you engage in.
| A lot of famous masters would oversee apprentices and
| step in for difficult tasks and to finish the work, yet
| we still attribute the work to those masters. Most of the
| work in science is done by graduate students, but we
| still attribute the lion's share of the credit to PIs.
| JeremyHerrman wrote:
| depends on what abstraction level you enjoy being creative
| at.
|
| Some people like creative coding, others like being
| creative with apps and features without much care to how
| it's implemented under the hood.
|
| I like both, but IMO there is a much larger crowd for
| higher level creativity, and in those cases AIs don't
| automate the creativity away, they enable it!
| jes5199 wrote:
| codex can actually do useful reviews on pull requests, as of
| the last few weeks
| ratelimitsteve wrote:
| honestly? this but zoom out. machines are supposed to do the
| grunt work so that people can spend their time being creative
| and doing intangible, satisfying things but we seem to have
| built machines to make art, music and literature in order to
| free ourselves up to stack bricks and shovel manure.
| dylan604 wrote:
| To me, it's the natural result of gaining popularity that
| enough people have started to use after the hype train rolled
| through and are now giving honest feedback. Real honest
| feedback can feel like a slap in the face when all you have
| had is overwhelming positive feedback from those aboard the
| hype train.
|
| The writing has been on the wall with so called
| hallucinations where LLMs just make stuff up that the hype
| was way out over its skiis. The examples of lawyers being
| fined for unchecked LLM outputs being presented as fact type
| of stories will continue to take the shine off and hopefully
| some of the raw gungho nature will slow down a bit.
| zdragnar wrote:
| I saw an article today from the BBC where travellers are
| using LLMs to plan their vacations and getting into trouble
| going places (sometimes dangerously remote ones) to visit
| landmarks that don't even exist:
|
| https://www.bbc.com/travel/article/20250926-the-perils-of-
| le...
|
| I'm mildly bearish on the human capacity to learn from its
| mistakes and have a feeling in my gut that we've taken a
| massive step backwards as civilization.
| dylan604 wrote:
| I could almost understand a lawyer working late the night
| before a brief is due and just run out of time to review
| the output of the LLM. How do you not look up travel
| destinations before heading out? That's just something I
| can't wrap my head around in any way of trying to be kind
| and seeing the other side of something
| mhitza wrote:
| In an application I'm working on, I use gpt-oss-20B. In a
| prompt I dump in the OWASP Top 10 web vulnerabilities, and a
| note that it should only comment on "definitive
| vulnerabilities". Has been pretty effective in finding
| vulnerabilities in the code I write (and it's one of the
| poorest-rated models if you look at some comments).
|
| Where I still need to extend this, is to introduce function
| calling in the flow, when "it has doubts" during reasoning,
| would be the right time to call out a tool that would expand
| the context its working with (pull in other files, etc).
| airstrike wrote:
| Tool calling is the best lever for getting value out of LLMs
| diggan wrote:
| > (and it's one of the poorest-rated models if you look at
| some comments).
|
| Yeah, don't listen to "wisdom of the crowd" when it comes to
| LLM models, there seems to be a ton of fud going on,
| especially on subreddits.
|
| GPT-OSS was piled on for being dumb in the first week of
| release, yet none of the software properly supported it at
| launch. As soon as it was working properly in llama.cpp, it
| was clear how strong the model was, but at that point the
| popular sentiments seems to have spread and solidified.
| CharlesW wrote:
| > _When I ask Claude to find bugs in my 20kloc C library it
| more or less just splits the file(s) into smaller chunks and
| greps for specific code patterns and in the end just gives me a
| list of my own FIXME comments (lol), which tbh is quite
| underwhelming - a simple bash script could do that too._
|
| Here's a technique that often works well for me: When you get
| unexpectedly poor results, ask the LLM what it thinks an
| effective prompt would look like, e.g. _" How would you prompt
| Claude Code to create a plan to effectively review code for
| logic bugs, ignoring things like FIXME and TODO comments?"_
|
| The resulting prompt is too long to quote, but you can see the
| raw result here:
| https://gist.github.com/CharlesWiltgen/ef21b97fd4ffc2f08560f...
|
| From there, you can make any needed improvements, turn it into
| an agent, etc.
| OGWhales wrote:
| I've found this a really useful strategy in many situations
| when working with LLMS. It seems odd that it works, since one
| one think its ability to give a good reply to such a question
| means it already "understands" your intent in the first
| place, but that's just projecting human ability onto LLMS. I
| would guess this technique is similar to how reasoning modes
| seems to improve output quality, though I may misunderstand
| how reasoning modes work.
| ako wrote:
| Works for humans the same? Even if you know how to do a
| complex project, it helps to first document the approach,
| and then follow it.
| einarfd wrote:
| This is a great idea, and worth doing. An other option in
| Claude code, that can be worth trying, is the planning mode,
| which you start with ctrl+tab. Have it plan out what it's
| going to do, and keep iterating it, until the plan seems
| sound. Tbh. I wish I've found the planning mode earlier, it's
| been such a great help.
| llleeeooo wrote:
| Indeed, in many machine learning models, classification is
| always easier than generation. Maybe that's consistent with
| chatgpts intelligence level
| notatoad wrote:
| i've had great success with both chatGPT and claude with the
| prompt "tell me how this sucks" or "why is this shit". being a
| bit more crass seems to bump it out of the sycophantic mode,
| and being more open-ended in the type of problems you want it
| to find seems to yield better results.
|
| but i've been limiting it to a lot less than 20k LoC, i'm
| sticking with stuff i can just paste into the chat window.
| freedomben wrote:
| I've had reasonably good success with asking Claude things
| like: "There's a bug somewhere that is causing slow response
| times on several endpoints, including <xyz>. Sometimes response
| times can get to several seconds long, and don't look
| correlated with CPU or memory usage. Database CPU and memory
| also don't seem to correlate. What is the issue?" I have to
| iterate a few times but it's hinted me a few really tricky
| issues that would have probably taken hours to find.
|
| Definitely optimistic for this way to use AI
| chmod775 wrote:
| I really didn't expect a story about curl and AI to be positive
| for once.
|
| Some history: https://hn.algolia.com/?q=curl+AI
| dwedge wrote:
| Yeah this is really fair play to Daniel Stenberg that he still
| approached these AI generated bug reports with an open mind
| after all the problems he's had.
| athorax wrote:
| I think the big difference is that these aren't AI generated
| bug reports. They are bugs found with the assistance of AI
| tools that were then properly vetted and reported in a
| responsible way by a real person.
| Legion wrote:
| Basically using AI the way we have used linters and other
| static analysis tools, rather than thinking it's magic and
| blindly accepting its output.
| 1970-01-01 wrote:
| Notice it was 'a set of tools'
|
| They're using it correctly. It's a system of tools, not an
| autopilot.
| ragnese wrote:
| Well, that's how Mr. Stenberg described it, but he wasn't the
| one using them. I don't know how the contributor feels about
| his AI tool(s).
| mos_basik wrote:
| I haven't read it yet, but later in the mastodon thread,
| stenberg says "this is [the contributor's] (long) blog post
| on his work: https://joshua.hu/llm-engineer-review-sast-
| security-ai-tools...".
| tptacek wrote:
| It's weird that the discussion has collapsed down to
| "autopilots" vs. "abstention". I'm thrilled to be converging on
| an understanding that it instead "people who understand what
| they're trying to do" vs. "vibe coders".
| nxobject wrote:
| In defense of the cynics, I get the impression in a situation
| where (a) there's a lot of company marketing hype in such a
| competitive market that begs cynicism, _and_ (b) we 're
| constantly learning the boundary of trained LLMs can actually
| do (and can't), as well as unusual emergent workflows, that
| really do make a difference.
| Timshel wrote:
| I did not read it, but this article from the contributor should
| contain more details: https://joshua.hu/llm-engineer-review-
| sast-security-ai-tools... (mentioned in
| https://mastodon.social/@bagder/115241413210606972).
| bgwalter wrote:
| If something is found by Valgrind, we can reproduce it ourselves.
| Here we get private bug reports found by "his set of AI assisted
| tools".
|
| The set seems to be:
|
| https://joshua.hu/llm-engineer-review-sast-security-ai-tools...
|
| So he likes ZeroPath. Does that get us any further? No, the
| regular subscription costs $200 and the free one-time version
| looks extremely limited and requires yet another login.
|
| Also of course, all low hanging fruit _that these tools detect_
| will be found quickly in open source (provided that someone can
| afford a subscription), similar to the fact that oss-fuzz has
| diminishing returns.
| simonw wrote:
| Presumably the bug reports were private because some of them
| might relate to curl security.
|
| You can see the fixes that resulted from this in the PRs that
| mention "sarif" in the curl repository:
| https://github.com/curl/curl/pulls?q=is%3Apr+sarif+is%3Aclos...
| cubefox wrote:
| This should probably link to the original blog post by Joshua
| Rogers:
|
| https://joshua.hu/llm-engineer-review-sast-security-ai-tools...
| ("Hacking with AI SASTs: An overview of 'AI Security Engineers' /
| 'LLM Security Scanners' for Penetration Testers and Security
| Teams")
| simonw wrote:
| The PDF slide deck that accompanies that post includes some
| screenshots of the tools that were used:
| https://joshua.hu/files/AI_SAST_PRESENTATION.pdf
| tempodox wrote:
| Now that is how LLM assistance for coding can be useful. Would be
| interesting to know which set of tools was used exactly. How
| might one reproduce this kind of assistance for other code bases?
| simonw wrote:
| See Joshua's post for details: https://joshua.hu/llm-engineer-
| review-sast-security-ai-tools...
|
| Tools included ZeroPath, Corgea and Almanax.
| excitedrustle wrote:
| And those good uses extend from appsec to cloudsec (IaC) as
| well.
|
| I'm working on open-source tool [1] to look for policy
| violations in cloud infra. LLMs are great at dealing with cloud
| security policies that are frequently subjective and under-
| specified. They can "understand" the intent of the policy and
| use tools to pull in the necessary context to fully evaluate a
| potential violation.
|
| We look at two examples in this blog post
|
| https://blog.fraim.dev/ai_eval_vs_rules/
|
| "No publicly exposed admin ports" and "IAM policies follow
| principle of least privilege".
|
| [1] https://github.com/fraim-dev/fraim
| konart wrote:
| Love this one:
|
| https://mastodon.social/@icing@chaos.social/1152440641434357...
|
| >tldr
|
| >The code was correct, the naming was wrong.
| alganet wrote:
| Something sounds fishy in this. Has these bugs really been found
| by AI? (I don't think they were).
|
| If you read Corgea's (one of the products used) "whitepaper", it
| seems that AI is not the main show:
|
| > BLAST addresses this problem by using its AI engine to filter
| out irrelevant findings based on the context of the application.
|
| It seems that AI is being used to post-process the findings of
| traditional analyzers. It reduces the amount of false positives,
| increasing the yield quality of the more traditional analyzers
| that were actually used in the scan.
|
| Zeropath seems to use similar wording like "AI-Enabled Triage"
| and expressions like "combining Large Language Models with AST
| analysis". It also highlights that it achieves less false
| positives.
|
| I would expect someone who developed this kind of thing to setup
| a feedback loop in which the AI output is somehow used to improve
| the static analysis tool (writing new rules, tweaking existing
| ones, ...). It seems like the logical next step. This might be
| going on on these products as well (lots of in-house rule
| extensions for more traditional static analysis tools, written or
| discovered with help of AI, hence the "build with AI" headline in
| some of them).
|
| Don't get me wrong, this is cool. Getting an AI to triage a
| verbose static analysis report makes sense. However, it does not
| mean that AI found the bugs. In this model, the capabilities of
| finding relevant stuff are still capped at the static analyzer
| tools.
|
| I wonder if we need to pay for it. I mean, now that I know it is
| possible (at least in my head), it seems tempting to get open
| source tools, set them to max verbosity, and find which prompts
| they are using on (likely vanilla) coding models to get them to
| triage the stuff.
| simonw wrote:
| Looks like you're reacting to the Hacker News title here, which
| is currently " Daniel Stenberg on 22 curl bugs found by AI and
| fixed"
|
| That's an editorialized headline (so it may get fixed by dang
| and co) - if you click through to what Daniel Stenberg said he
| was more clear:
|
| > Joshua Rogers sent us a _massive_ list of potential issues in
| #curl that he found using his set of AI assisted tools.
|
| AI-assisted tools seems right to me here.
| alganet wrote:
| If the title changes, it is still a valid critique of the
| tools, how they might work, and a possible way of getting
| them for free.
|
| Also, think about it: of course I read Joshua's report.
| Otherwise, how could I have known the names of the products
| he used?
| bgwalter wrote:
| I don't think many people here are interested in how
| something works. They want to see the headline "Curl
| developer finally convinced by AI!" and otherwise drop
| anecdotes about Claude Code etc.
|
| All comments that want to know more are at the bottom.
| robhlam wrote:
| It's clear my attempt to keep the gist of what Daniel said
| while keeping under the title character count didn't hit the
| mark.
|
| How would you have worded it?
| simonw wrote:
| Always tricky! In this case maybe the following:
|
| Daniel Stenberg on 22 curl bugs reported using AI-assisted
| security scanners
| robhlam wrote:
| That doesn't really convey that these bug reports were
| for real issues and greatly appreciated unlike the slop
| that Daniel is known for complaining about which I think
| that's the real story here.
|
| I will spend longer considering my title next time.
|
| Cheers!
| bgwalter wrote:
| I suppose the downvoters all have subscriptions to the tools
| and know exactly how the tools work while leaving the rest of
| us in the dark.
|
| Even Joshua's blog post does not clearly state which parts and
| how much is "AI". Neither does the pdf.
| refulgentis wrote:
| I believe there's a little more going on than everyone
| knowing every detail already, or presumably, being wrong to
| downvote.
|
| Full case study of a downvoter at work:
|
| I've worked on this stuff for 5 years between Google and
| leaving Google in 2023. It has came a long way, certainly,
| but the shape was clear in 2022. I used GPT-3, i.e. pre-
| ChatGPT, to get a bunch of color science code ported to
| multiple languages ~automatically, allowing us to ship
| Material 3 rapidly on everything from Android to Chrome to
| Search.
|
| At this point, long comments trying to find a way to make it
| not actually true and nitpick are passe. (n.b. to reader, I
| forgot why I had that reaction, suffice it to share the
| opening: "Something sounds fishy in this. Has these bugs
| really been found by AI? (I don't think they were).")
|
| We're _well_ past "what's the trick here". c.f. tptacek's
| viral article a month or two ago. (https://fly.io/blog/youre-
| all-nuts/)
|
| I feel all of my 37 years, and those of my interlocutor, when
| I see the # of people on HN, of all places, still trying to
| wiggle their way to these being exceptional cases.
|
| I don't even understand at this point, I haven't had to write
| substantive code since the GPT-5 release (August 2025), and
| maybe 20% as much since Sonnet 3.5. (Jan 2025) It's a very
| real thing, for many of us, and I can't imagine the mindset
| that would have me _avoiding_ substantive help with my work.
|
| At some point, the ur-rejection stance of "I don't believe"
| is so far behind and out of touch it will be seen as
| quarrelsome or nitpicking rather than substantive. Not quite
| now, but, well, closer than we think: _waves at situation_.
|
| Random aside: one formative moment for me was perusing Visual
| Basic 6 and Visual C++ 6 at Barnes & Noble. Late 90s. ~10
| years old. Dad was in tech too. He wistfully wished he had
| time to look at such things and hit the new wave (he was a
| "DBA", database administrator?, if that exists anymore)
|
| It left me with a lifelong fear of avoiding curiosity, as it
| would sentence me to work daily on things that were
| interesting 30 years ago.
|
| Maybe that helped me here.
|
| But, all that aside, this whole situation is a sigil we're
| past "what individual curiosity makes you into AI" and on our
| way to "please stop being skeptical that electronic
| calculators work and fucking use it, we can't afford to have
| you do this by hand"
| alganet wrote:
| Do you believe AI is at the core of these security
| analyzers? If so, why the personal story blogpost? You can
| just explain me in technical terms why is that so.
|
| Claiming to work for Google does not work as an authority
| card for me, you still have to deliver a solid argument.
|
| Look, AI is great for many things, but to me these products
| sounds like chocolate that is actually just 1% real
| chocolate. Delicious, but 99% not chocolate.
| refulgentis wrote:
| I don't mean to aggravate you. I do mean to offer some
| insight in the mindset of the people the person I was
| replying to was puzzled by. I'm calmed by the fact that
| if we're both here, we both value one of the HN sayings
| I'm very fond of: come with curiosity.
|
| > Do you believe AI is at the core of these security
| analyzers?
|
| Yes.
|
| > If so, why the personal story blogpost?
|
| When I am feeling intensely, and people respond to me as
| I'm about to respond to you, I usually get _very_
| frustrated. Apologies in advance if you suffer from that
| same part of being human, I don 't mean anything about
| you or your positions by this:
|
| I don't know what you mean.
|
| Thus, I may be answering wrong with the following: the
| person I replied to indicated all downvoters must know
| every detail, and as the, well lets use your phrasing,
| personal story blogpost, I just assume you mean my
| comment, leads with: "I believe there's a little more
| going on than everyone knowing every detail already, or
| presumably, being wrong to downvote. Full case study of a
| downvoter at work:"
|
| > Claiming to work for Google
|
| I claimed the opposite! I'm a jobless hack :) (quit in
| 2023)
|
| > does not work as an authority card for me,
|
| Looking at it, the thing isn't "I worked at Google
| therefore AI good" it's "I worked at Google and on a
| specific well-known project, the company's design
| language, used AI _pre-ChatGPT_ to great effect. It 's
| unclear to me why _this_ use case would be unbelievable
| years later "
|
| > you still have to deliver a solid argument.
|
| What are we arguing? :) (I'm serious! Apologies, again,
| if it comes off as flippant. If you mean I need to
| deliver a solid argument the tools _must_ have AI, I
| assume if said details were available you would have
| found them, you seem well-considered and curious. I meant
| to explain the mind of a downvoter who yet cannot recite
| details as yet unavailable to the public to the person I
| replied to, not to verify the workflow step by step.)
| alganet wrote:
| The argument is that these high-quality security
| analyzers seem to use AI as a triage mechanism, and the
| quality of the analysis is still capped by the quality of
| the static analysis tool.
|
| One of the tools provide a whitepaper, that you can read
| here:
|
| https://corgea.com/blog/whitepaper-blast-ai-powered-sast-
| sca...
|
| It seems to explicitly put AI in this coadjuvant role,
| contradicting the HN title "found by AI".
|
| Neither me or the other commenter actually dismissed AI
| as useless. I can't speak for him, but to me, it seems
| actually useful in this arrangement. However, not "I'll
| pay for a subscription" levels of useful.
|
| Since it's just triage, it seems that trying to reproduce
| the idea using free tools might be worth a shot (and
| that's the idea of finding out where the AI component
| lies in the system). What I said is very doable (plug the
| output of traditional tools into vanilla coding LLMs
| prompts). It also looks a lot like this Corgea schematic:
|
| https://framerusercontent.com/images/EtFkxLjT1Ou2UTPACObJ
| bR2...
|
| I mean, it's very brave to explain a downvote, but in
| this case, it seems that you missed the opportunity to
| make sense.
| tptacek wrote:
| I had a conversation in a chat room yesterday about AI-
| assisted math tutoring where a skeptic said that the
| ability of GPT5 to effortlessly solve quotient
| differentials or partial fraction decomposition or
| rational inequalities wasn't indicative of LLM
| improvements, but rather just represented the LLMs
| driving CAS tools and thus didn't count.
|
| As a math student, I can't possibly care less about that
| distinction; either way, I paste in a worked problem
| solution and ask for a critique, and either way I get a
| valid output like "no dummy multiply cos into the tan
| before differentiating rather than using the product
| rule". Prior to LLMs, there was no tool that had that UX.
|
| In the same way: LLMs are probably mostly not off the top
| of their "heads" (giant stacks of weight matrices)
| axiomatically deriving vulnerabilities, but rather just
| doing a very thorough job of applying existing program
| analysis tools, assembling and parallel-evaluating large
| numbers of hypothesis, and then filtering them out. My
| interlocutor in the math discussion would say that's just
| tool calls, and doesn't count. But if you're a
| vulnerability researcher, it doesn't matter: that's a DX
| that didn't exist last year.
|
| As anyone who has ever been staffed on a project triaging
| SAST tool outputs before would attest: it _extremely_
| didn 't exist.
| silisili wrote:
| > I have already landed 22(!) bugfixes thanks to this, and I have
| over twice that amount of issues left to go through
|
| Sounds like it was a lot more than 22, assuming most are valid.
| retr0reg wrote:
| I work in a ML security R&D startup called Pwno, we been working
| on specifically putting LLMs into memory security for the past
| year, we've spoken at Black Hat, and we worked with GGML
| (llama.cpp) on providing a continuous memory security solution by
| multi-agents LLMs.
|
| Somethings we learnt alone the way, is that when it comes to
| specifically this field of security what we called low-level
| security (memory security etc.), validation and debugging had
| became more important than vulnerability discovery itself because
| of hallucinations.
|
| From our trial-and-errors (trying validator architecture,
| security research methodology e.g., reverse taint propagation),
| it seems like the only way out of this problem is through
| designing a LLM-native interactive environment for LLMs, validate
| their findings of themselves through interactions of the
| environment or the component. The reason why web security
| oriented companies like XBOW are doing very well, is because how
| easy it is to validate. I seen XBOW's LLM trace at Black Hat this
| year, all the tools they used and pretty much need is curl. For
| web security, abstraction of backend is limited to a certain
| level that you send a request, it whether works or you easily
| know why it didn't (XSS, SQLi, IDOR). But for low-level security
| (memory security), the entropy of dealing with UAF, OOBs is at
| another level. There are certain things that you just can't tell
| by looking at the source but need you to look at a particular
| program state (heap allocation (which depends on glibc version),
| stack structure, register states...), and this ReACT'ing process
| with debuggers to construct a PoC/Exploit is what been a pain-in-
| the-ass. (LLMs and tool callings are specifically bad at these
| strategic stateful task, see Deepmind's Tree-of thoughts paper
| discussing this issue) The way I've seen Google Project Zero &
| Deepmind's Big Sleep mitigating this is through GDB scripts, but
| that's limited to a certain complexity of program state.
|
| When I was working on our integration with GGML, spending around
| two weeks on context, tool engineering can already lead us to
| very impressive findings (OOBs); but that problem of
| hallucination scales more and more with how many "runs" of our
| agentic framework; because we're monitoring on llama.cpp's main
| branch commits, every commits will trigger a internal multi-agent
| run on our end and each usually takes around 1 hours and hundreds
| of agent recursions. Sometime at the end of the day we would have
| 30 really really convincing and in-depth reports on OOBs, UAFs.
| But because how costly to just validate one (from understanding
| to debugging, PoC writing...) and hallucinations, (and it is
| really expensive for each run) we had to stop the project for a
| bit and focus solving the agentic validation problem first.
|
| I think when the environment gets more and more complex,
| interactions with the environment, and learning from these
| interactions will matters more and more.
| bwfan123 wrote:
| > I think when the environment gets more and more complex,
| interactions with the environment, and learning from these
| interactions will matters more and more
|
| Thanks for sharing your experience ! It correlates with this
| recent interview with Sutton [1]. That real intelligence is
| learning from feedback with a complex and ever changing
| environment. What an LLM does is to train on a snapshot of what
| has been said about that environment and operate on only on
| that snapshot.
|
| [1] https://www.dwarkesh.com/p/richard-sutton
| qustrolabe wrote:
| Oh so AI usage news could be positive after all. Not to undermine
| huge issue of slop reports spam, but I'm so happy to see
| something besides doomerism
| lowbloodsugar wrote:
| When I read "we consider nread == 0 as reading a byte and we
| shouldn't" I immediately think of all the things that look like
| bugs but are there because some critical piece of infrastructure
| relies on that behavior. AI isn't going to know about that unless
| you tell it, and the problem is that there's plenty of folks who
| have job security precisely because they don't write that down.
| dude250711 wrote:
| _" Using its human tool AI sent us a *massive* list of potential
| issues in #curl that it found."_
| kissgyorgy wrote:
| I guess the AI slop an actual useful finding are now in balance
| for curl.
| noelwelsh wrote:
| More interesting to me is how to stop these bugs from occurring
| in the first place. The example given in the thread is the kind
| of bug that C (and mutation) excels at creating.
| viraptor wrote:
| The linked blog post https://joshua.hu/llm-engineer-review-
| sast-security-ai-tools... shows that most of the used tools can
| be run in ci and comment on the PRs.
| daxfohl wrote:
| So whereas previously, repo owners were getting flooded with AI-
| generated PRs that were complete slop, now they're going to be
| flooded with PRs that contain actual bugfixes. IDK which problem
| is worse!
| blixt wrote:
| It wasn't immediately obvious to me what the AI tools were? He
| mentioned that multiple other tools failed to find anything, so
| I'm very curious to hear what made this strategy so superior.
| NooneAtAll3 wrote:
| there's a blog link https://joshua.hu/llm-engineer-review-sast-
| security-ai-tools... that has Products chapter
|
| I guess mastodon link is simply a confirmation that bugs were
| indeed bugs, even with wrong code snippets?
| srcreigh wrote:
| Link should be updated to this
|
| https://joshua.hu/llm-engineer-review-sast-security-ai-tools...
| tiahura wrote:
| Perhaps Anthropic, OpenAI, and Google could compete by auditing
| and monitoring the top projects?
| runningmike wrote:
| There are some good SAST scanners and many bad commercial
| scanners.
|
| Many people advocate for the use of AI technology for SAST
| testing. There are even people and companies that deliver SAST
| scanners based on AI technology. However: Most are just far from
| good enough.
|
| In the best case scenario, you'll only be disappointed. But the
| risk of a false sense of security is enormous.
|
| Some strong arguments against AI scanners can be found on
| https://nocomplexity.com/ai-sast-scanners/
| renox wrote:
| Once Claude found a bug in my code but I had to explain the
| structure of the data. Then and only then it found the bug.
| Eggpants wrote:
| I have to admit, I expected a couple of "You should rewrite it in
| Rust" hipster posts by now... Maybe they caught on that those
| types of posts were not having the effect they thought they
| would? I kid, I kid... mostly
| redbell wrote:
| Somehow related:
|
| _You did this with an AI and you do not understand what you 're
| doing here_: https://news.ycombinator.com/item?id=45330378
| cadamsdotcom wrote:
| AI is non-deterministic as we know.
|
| That makes its results unpredictable.
|
| So don't have AI create your bugs.
|
| Instead have your AI look for problems - then have it create
| deterministic tools and let tools catch the issues in a
| repeatable, understandable, auditable way. Have it build short,
| easy to understand scripts you can commit to your repo, with
| files and line numbers and zero/nonzero exit codes.
|
| It's that key step of transforming AI insights into detection
| tools that transforms your outcomes from probabilistic to
| deterministic. Ask it to optimize the tools so they run in
| seconds. You can leave them in the codebase forever as linters,
| integrate them in your CI, and never have that same bug again.
___________________________________________________________________
(page generated 2025-10-02 23:01 UTC)