[HN Gopher] Hallucinations in code are the least dangerous form ...
___________________________________________________________________
Hallucinations in code are the least dangerous form of LLM mistakes
Author : ulrischa
Score : 66 points
Date : 2025-03-02 19:15 UTC (3 hours ago)
(HTM) web link (simonwillison.net)
(TXT) w3m dump (simonwillison.net)
| al2o3cr wrote:
| My cynical side suspects they may have been looking for a
| reason to dismiss the technology and jumped at the first
| one they found.
|
| MY cynical side suggests the author is an LLM fanboi who prefers
| not to think that hallucinating easy stuff strongly implies
| hallucinating harder stuff, and therefore jumps at the first
| reason to dismiss the criticism.
| williamcotton wrote:
| What do you mean by "harder stuff"? What about an experimental
| DSL written in C with a recursive descent parser and a web
| server runtime that includes Lua, jq, a Postgres connection
| pool, mustache templates, request-based memory arena, database
| migrations and much more? 11,000+ lines of code with ~90%
| written by Claude in Cursor Composer.
|
| https://github.com/williamcotton/webdsl
|
| Frankly us "fanbois" are just a little sick and tired of being
| told that we must be terrible developers working on simple toys
| if we find any value from these tools!
| dzaima wrote:
| Some free code review of the first file I clicked into - http
| s://github.com/williamcotton/webdsl/blob/92762fb724a9035...
| among other places should probably be doing the conditional
| "lexer->line++"; thing. Quite a weird decision to force all
| code paths to manually do that whenever a newline char is
| encountered. Could've at least made a
| "advance_maybe_newline(lexer);" or so. But I guess LLMs give
| you copy-paste garbage.
|
| Even the article of this thread says:
|
| > Just because code looks good and runs without errors
| doesn't mean it's actually doing the right thing.
| williamcotton wrote:
| Thanks for taking a look! The lexer and parser is probably
| close to 100% Claude and I definitely didn't review it
| completely. I spent most of the time trying out different
| grammars (normally something you want to do _before_ you
| start writing code) and runtime features! "Build the web
| server runtime and framework into the language" was an idea
| kicking around in my head for a few years but until Cursor
| I didn't have the energy to play around with the idea.
| ianbutler wrote:
| Okay so this is a personal opinion right? Like where is the
| objectivity in your review?
|
| What are the hardline performance characteristics being
| violated? Or functional incorrectness. Is this just "it's
| against my sensibilities" because at the end of the day
| frankly no one agrees on how to develop anything.
|
| The thing I see a lot of developers struggle with is just
| because it doesn't fit your mental model doesn't make it
| objectively bad.
|
| So unless it's objectively wrong or worse in a measurable
| characteristic I don't know that it matters.
|
| For the record I'm not asserting it is right, I'm just
| saying I've seen a lot of critiques of LLM code boil down
| to "it's not how I'd write it" and I wager that holds for
| every developer you'll ever interact with.
| KoolKat23 wrote:
| I agree, it seems a lot of the complaints boil down to
| academic reasons.
|
| Fine it's not the best and perhaps may run into some
| longer term issues but most importantly it works at this
| point in time.
|
| A snobby/academic equivalent would be someone using an
| obscure language such as COBOL.
|
| The world continues to turn.
| dzaima wrote:
| OP didn't put much effort into writing the code so I'm
| certainly not putting in much effort into a proper review
| of it, for no benefit to me no less.
|
| I'm pretty sure the code not having the "if (...)
| lexer->line++" in places is just a plain simple repeated
| bug that'd result in wrong line numbers in places.
|
| And human-wise I'd say the simple way to not have made
| that bug would've been to make/change abstractions such
| that it's hard to get it wrong, vs the code allowing
| getting it wrong by default with no indication that
| there's a thing to be gotten wrong.
| elanora96 wrote:
| I'm a strong believer that LLMs are tools and when wielded by
| talented and experienced developers they are somewhere in the
| danger category of Stack Overflow and transitive
| dependencies. This is not a critique of your project, or
| really the quality of LLMs, but when I see 90% of a 11,000+
| loc project written in Claude, it just feels sort of
| depressing in a way I haven't processed yet.
|
| I love foss, I love browsing projects of all quality levels
| and vintages and seeing how things were built. I love
| learning new patterns and sometimes even bickering over their
| strengths and weaknesses. An LLM generated code base hardly
| makes me even want to engage with it...
|
| Perhaps these feelings are somewhat analogous to hardcopies
| vs ebooks? My opinions have changed over time and I read and
| collect both. Have you had similar thoughts and gotten over
| them? Do you see tools like Claude in a way where this isn't
| an issue?
| williamcotton wrote:
| I mean, when I'm working on something that I don't expect
| to be more than a throw-away experiment I'm not too worried
| about the code itself.
|
| The grammar itself still seems a bit clunky and the next
| time I head down this path I imagine I'll go with a more
| hand-crafted approach.
|
| I learned a _lot_ about integrating Lua and jq into a
| project along the way (and how to make it performant),
| something I had no prior experience with.
| Snuggly73 wrote:
| ..."request-based memory arena"...
|
| there are some very questionable things going on with the
| memory handing in this code. just saying.
| williamcotton wrote:
| Request-based memory arenas are pretty standard for web
| servers!
| semi-extrinsic wrote:
| Honest question: this looks like a library others can use to
| build websites. It contains features related to
| authentication and security. If it's 90% LLM generated, how
| do you sleep at night? I'd be dead scared someone would use
| this, hit a bug that leaks PII (or worse) and then sue me
| into oblivion.
| williamcotton wrote:
| "WebDSL is an _experimental_ domain-specific language and
| server implementation for building web applications. "
|
| And it's MIT: THE SOFTWARE IS PROVIDED "AS
| IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
| IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
| MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
| NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
| COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
| LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
| OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH
| THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
| SOFTWARE.
| davesque wrote:
| I thought he was going to say the really danger is hallucination
| of facts, but no.
| throwaway314155 wrote:
| > The real risk from using LLMs for code is that they'll make
| mistakes that aren't instantly caught by the language compiler or
| interpreter. And these happen all the time!
|
| Are these not considered hallucinations still?
| dzaima wrote:
| Humans can hallucinate up some API they want to call in the
| same way that LLMs can, but you don't call all human mistakes
| hallucinations; classifying everything LLMs do wrong as
| hallucinations would seem rather pointless to me.
| ForTheKidz wrote:
| Maybe we should stop referring to undesired output
| (confabulation? Bullshit? Making stuff up? Creativity?) as
| some kind of input delusion. Hallucination is already a
| meaningful word and this is just gibberish in that context.
|
| As best I can tell, the only reason this term stuck is
| because early image generation looked super trippy.
| thylacine222 wrote:
| Analogizing this to human hallucination is silly. In the
| instance you're talking about, the human isn't hallucinating,
| they're lying.
| dzaima wrote:
| I definitely wouldn't say I'm lying (...to.. myself? what?
| or perhaps others for a quick untested response a chatroom
| or something) whenever I write some code and it turns out
| that I misremembered the name of an API. "Hallucination"
| for that might be over-dramatic but at least it it's a
| somewhat sensible description.
| fweimer wrote:
| I don't think it's necessarily a hallucination if models
| accurately reproduce the code quality of their training data.
| sublinear wrote:
| > Compare this to hallucinations in regular prose, where you need
| a critical eye, strong intuitions and well developed fact
| checking skills to avoid sharing information that's incorrect and
| directly harmful to your reputation
|
| Ah so you mean... actually doing work. Yeah writing code has the
| same difficulty, you know. It's not enough to merely get
| something to compile and run without errors.
|
| > With code you get a powerful form of fact checking for free.
| Run the code, see if it works.
|
| No, this would be coding by coincidence. Even the most
| atrociously bad prose writers don't exactly go around just saying
| random words from a dictionary or vaguely (mis)quoting
| Shakespeare hoping to be understood.
| Velorivox wrote:
| Not just that, "it works" is a very, very low bar to have for
| your code. To illustrate, the other day I tested an LLM by
| having it create a REST API. I asked for an end point where I
| could update a particular field of the record (think liking a
| post).
|
| Then I decided to add on more functionality and asked for the
| ability to update all the other fields...
|
| As you can guess, it gave me one endpoint per field for that
| entity. Sure, "it works"...
| trollbridge wrote:
| There are human developers who do the same thing...
| drekipus wrote:
| > Even the most atrociously bad prose writers don't exactly go
| around just saying random words from a dictionary or vaguely
| (mis)quoting Shakespeare hoping to be understood.
|
| I actually do this (and I'm not proud of it)
| jccalhoun wrote:
| I am not a programmer and i don't use Linux. I've been working on
| a python script for a raspberry pi for a few months. Chatgpt has
| been really helpful in showing me how to do things or debug
| errors.
|
| Now I am at the point that I am cleaning up the code and making
| it pretty. My script is less than 300 lines and Chatgpt regularly
| just leaves out whole chunks of the script when it suggests
| improvements. The first couple times this led to tons of head
| scratching over why some small change to make one thing more
| resilient would make something totally unrelated break.
|
| Now I've learned to take Chatgpt's changes and diff it with the
| working version before I try to run it.
| Tostino wrote:
| Version control inside an IDE helps with noticing these types
| of changes, even if you aren't a programmer
| sidpatil wrote:
| You can try asking ChatGPT to rewrite the original script to
| include the improvements.
| genewitch wrote:
| Chatgpt can output a straight diff, too, that you can use with
| patch.
|
| That's how aider commands the models to reply, for example.
| apwell23 wrote:
| yea its great at toy projects
| gojomo wrote:
| Such "hallucinations" can also be plausible & useful APIs that
| _oughtta_ exist - de facto feature requests.
| dullcrisp wrote:
| That's right, sometimes it's the children who are wrong.
| atomic128 wrote:
| Last week, The Primeagen and Casey Muratori carefully review the
| output of a state-of-the-art LLM code generator.
|
| They provide a task well-represented in the LLM's training data,
| so development should be easy. The task is presented as a
| cumulative series of modifications to a codebase:
|
| https://www.youtube.com/watch?v=NW6PhVdq9R8
|
| This is the actual reality of LLM code generators in practice:
| iterative development converging on useless code, with the LLM
| increasingly unable to make progress.
| sunami-ai wrote:
| I asked o3-mini-high (investor paying for Pro, I personally would
| not) to critique the Developer UX of D3's "join" concept (how
| when you select an empty set then when you update you enter/exit
| lol) and it literally said "I'm sorry. I can't help you with
| that." The only thing missing was calling me Dave.
| t_mann wrote:
| Hallucinations themselves are not even the greatest risk posed by
| LLMs. A much greater risk (in simple terms of probability times
| severity) I'd say is that chat bots can talk humans into harming
| themselves or others. Both of which have already happened, btw
| [0,1]. Still not sure if I'd call that the greatest overall risk,
| but my ideas for what could be even more dangerous I don't even
| want to share here.
|
| [0] https://www.qut.edu.au/news/realfocus/deaths-linked-to-
| chatb...
|
| [1] https://www.theguardian.com/uk-news/2023/jul/06/ai-
| chatbot-e...
| tombert wrote:
| I don't know if the model changed in the last six months, or
| maybe the wow factor has worn off a bit, but it also feels like
| ChatGPT has become a lot more "people-pleasy" than it was
| before.
|
| I'll ask it opinionated questions, and it will just do stuff to
| reaffirm what I said, even when I give contrary opinions in the
| same chat.
|
| I personally find it annoying (I don't really get along with
| human people pleasers either), but I could see someone using it
| as a tool to justify doing bad stuff, including self-harm; it
| doesn't really ever push back on what I say.
| renewiltord wrote:
| It's obvious, isn't it? The average Hacker News user, who has
| converged to the average Internet user, wants exactly that
| experience. LLMs are pretty good tools but perhaps they
| shouldn't be made available to others. People like me can use
| them but others seem to be killed when making contact. I
| think it's fine to restrict access to the elite. We don't let
| just anyone fly a fighter jet. Perhaps the average HN user
| should be protected from LLM interactions.
| tombert wrote:
| I use ChatGPT to generate code a lot, and it's certainly useful,
| but it has given me issues that are not obvious.
|
| For example, I had it generate some C code to be used with ZeroMQ
| a few months ago. The code looked absolutely fine, and it
| _mostly_ worked fine, but it made a mistake with its memory
| allocation stuff that caused it to segfault sometimes, and
| corrupt memory other times.
|
| Fortunately, this was such a small project and I already know how
| to write code, so it wasn't too hard for me to find and fix,
| though I am slightly concerned that some people are copypasting
| large swaths of code from ChatGPT that looks mostly fine but
| hides subtle bugs.
| KoolKat23 wrote:
| And subtle bugs existed pre-2022, how often my apps are updated
| for "minor bug fixes" would mean this is par for the course.
| fzeroracer wrote:
| > I'll finish this rant with a related observation: I keep seeing
| people say "if I have to review every line of code an LLM writes,
| it would have been faster to write it myself!"
|
| > Those people are loudly declaring that they have under-invested
| in the crucial skills of reading, understanding and reviewing
| code written by other people. I suggest getting some more
| practice in. Reviewing code written for you by LLMs is a great
| way to do that.
|
| Not only is this a massive bundle of assumptions but it's also
| just wrong on multiple angles. Maybe if you're only doing basic
| CRUDware you can spend five seconds and give a thumbs up but in
| any complex system you should be spending time deeply reading
| code. Which is naturally going to take longer than using what
| knowledge you already have to throw out a solution.
| nottorp wrote:
| > I asked Claude 3.7 Sonnet "extended thinking mode" to review an
| earlier draft of this post [snip] It was quite helpful,
| especially in providing tips to make that first draft a little
| less confrontational!
|
| So he's also using LLMs to steer his writing style towards the
| lowest common denominator :)
| layer8 wrote:
| > Just because code looks good and runs without errors doesn't
| mean it's actually doing the right thing. No amount of meticulous
| code review--or even comprehensive automated tests--will
| demonstrably prove that code actually does the right thing. You
| have to run it yourself!
|
| I would have stated this a bit differently: No amount of running
| or testing can prove the code correct. You actually have to
| reason through it. Running/testing is merely a sanity/spot check
| of your reasoning.
| dmos62 wrote:
| Well, what if you run a complete test suite?
| layer8 wrote:
| There is no complete test suite, unless your code is purely
| functional and has a small-ish finite input domain.
| suzzer99 wrote:
| And even then, your code could pass all tests but be a
| spaghetti mess that will be impossible to maintain and add
| features to.
| e12e wrote:
| You mean, for example test that your sieve finds all primes,
| and only primes that fit in 4096 bits?
| nnnnico wrote:
| not sure that human reasoning actually beats testing when
| checking for correctness
| ljm wrote:
| The production of such tests presumably requires an element
| of human reasoning.
|
| The requirements have to come from somewhere, after all.
| layer8 wrote:
| Both are necessary, they complement each other.
| not2b wrote:
| If the hallucinated code doesn't compile (or in an interpreted
| language, immediately throws exceptions), then yes, that isn't
| risky because that code won't be used. I'm more concerned about
| code that appears to work for some test cases but solves the
| wrong problem or inadequately solves the problem, and whether we
| have anyone on the team who can maintain that code long-term or
| document it well enough so others can.
| t14n wrote:
| fwiw this problem already exists with my more junior co-
| workers. and also my own code that I write when exhausted!
|
| if you have trusted processes for review and aren't always
| rushing out changes without triple checking your work (plus a
| review from another set of eyes), then I think you catch a lot
| of the subtler bugs that are emitted from an LLM.
| svaha1728 wrote:
| If X, AWS, Meta, and Google would just dump their code into a ML
| training set we could really get on with disrupting things.
| chad1n wrote:
| The idea is correct, a lot of people (including myself sometimes)
| just let an "agent" run and do some stuff and then check later if
| it finished. This is obviously more dangerous than just the LLM
| hallucinating functions, since at least you can catch the latter,
| but the first one depends on the tests of the project or your
| reviewer skills.
|
| The real problem with hallucination is that we started using LLMs
| as search engines, so when it invents a function, you have to go
| and actually search the API on a real search engine.
| nojs wrote:
| > Hallucinated methods are such a tiny roadblock that when people
| complain about them I assume they've spent minimal time learning
| how to effectively use these systems--they dropped them at the
| first hurdle.
|
| If you're writing code in Python against well documented APIs,
| sure. But it's an issue for less popular languages and
| frameworks, when you can't immediately tell if the missing method
| is your fault due to a missing dependency, version issue, etc.
___________________________________________________________________
(page generated 2025-03-02 23:00 UTC)