Post B6ubeyRsmQvomotz6m by elebertus@eigenmagic.net
(DIR) More posts by elebertus@eigenmagic.net
(DIR) Post #B6qlbCCpuWvNClrU6C by jonny@neuromatch.social
1 likes, 1 repeats
RE: https://hails.org/@hailey/116657391001259044all the criticism has been said, all the takes been had. the only metaphor i have been finding consistently useful for understanding what is happening with people and "AI" is addiction, and specifically gambling addiction.RT: https://hails.org/users/hailey/statuses/116657391001259044
(DIR) Post #B6qlbCQJ6RhRsYAF60 by jonny@neuromatch.social
0 likes, 0 repeats
i love gambling. i have used "AI" extensively. it feels the same.
(DIR) Post #B6qlbCbeQGm2RjTImG by jonny@neuromatch.social
2 likes, 0 repeats
So, look. One shot rewriting the whole test suite in another language is probably not great to do, but what happened here is so much worse than you are expecting. https://github.com/RsyncProject/rsync/pull/903/This does not "translate tests into pytest" or a unit testing framework, it writes its own testing framework where tests are whole python scripts that redefine basic test functions in every script. Surely there would be a single way to "run rsync and get the results" - nope, well, there is, but then every test file will randomly redefine its own _run_and_capture function. So like now rsync needs a test suite for its test suite.If instead of telling an LLM to "rewrite the tests in python" you just searched "python testing" you would find the pytest docs. And then you would find examples. And then you could write fixtures to deduplicate all the prior shell script setup and teardown stuff, and so on. But since it was just "rewrite the tests in python" its now worse than before, and the odds of the rewrite actually being a 100% faithful translation are close to 0.
(DIR) Post #B6qlbCqBYEOrAoGuQq by jonny@neuromatch.social
1 likes, 0 repeats
I think the modal situation here is that the people are reading none or very little of what is being generated by the LLM, so the tests have a special role: Tests function as the pull arm on the slot machine, you just generate until tests pass, and that's a jackpot. Obviously that's meaningless when the tests are meaningless, so tests take on a very different meaning and role in slot machine coding. Previously we would write careful test conditions that were based off some real problem or an understanding of what the code under test did, and had a specific thing they were intended to protect against. Tests move slow and are designed to protect us against the things we know can go wrong. When we learn of a new wrong thing, we add a test.LLM tests have the form of tests but don't do the same thing. They often test nothing, and are just expressions of truisms that the probabilistic text space explored while generating. They have strongly worded names but end up actually asserting that basic language features work as expected. Because it is not us writing tests for ourselves, where we only harm ourselves by making them weak, they function instead as a passively obfuscated justification for the code that the LLM generates. The user wants the tests to pass. The LLM provides. The tests are theater: they are the play field for the slot machine. They are mild, surmountable, need to fail a few times to be plausible, but must eventually pass within the expected generation loop window to deliver the payout.
(DIR) Post #B6qlbD7uUKZu3mZ43k by jens@social.finkhaeuser.de
0 likes, 0 repeats
@jonny related:https://finkhaeuser.de/2026-04-10-outsourcing-thought-is-going-great/
(DIR) Post #B6qlbDYqsA83PLAa3M by sesamzoo@mastodon.social
0 likes, 1 repeats
@jens, great article, thank you. Did you pull the lever "just one more time" and if so, did it get even worse?@jonny, thank you for this thread and lots of your other threads on the topic.Both help feeling that I'm not the ghost driver although these days there is lot of contraflow on my lane. Mostly at work where the AI fanboys/believers/addicts are at least way louder than the people trying to understand and keeping their code in maintainable shape.
(DIR) Post #B6qlbDlG823O1oyUOO by jonny@neuromatch.social
1 likes, 0 repeats
Here's an example from some code that was thrust at me this week. The rest of the tests try a bit harder to look like tests, but this one is perplexing. What does it test? The function name suggests its a smoke test. LLMs love to call things smoke tests. That would suggest this would be an early-run test that fails loudly if some basic precondition - like having ffmpeg - fails. Or, I guess we are smoke testing the ensure_ffmpeg function? Anyway who knows. However we first check if ffmpeg or ffprobe are present, which is exactly what ensure_ffmpeg does. If they aren't present, a warning tells us that ffmpeg/ffprobe are required for the video tests, which makes it seem like this should be a parameterizing test that controls which tests are run, which of course it does not do.So the test literally does nothing and cannot possibly fail, but says it does at least two things, because to an LLM something saying it does something is the same thing as it actually doing that thing.
(DIR) Post #B6qlbEaJ4Et2a8rFxY by jonny@neuromatch.social
0 likes, 0 repeats
To a person, the whole purpose of the test is for it to fail when it should. That's an elemental part of writing good tests: they must fail before the patch, or else they provide no protection. We want protection from failure, that is good for us. We need tests to protect us because we can't possibly evaluate all the other parts of a complex system when we try to fix one part of it.LLM slot machines change what tests mean - of course we still want the code to work good, but if we're not evaluating the code or the tests, then what the slot machine turns them into is just a high score and the jackpot condition. 130 new tests added, that means its good. They pass, that means I win.The bugfix loop with LLMs defeats the purpose of automated tests and renders it no better than manual testing: you notice a bug, you yell at the LLM to fix it, you keep looking at the specific thing that's broken until its fixed, good robot, ship it. The changes don't have meaningful tests, and nothing else does either, so the slot machine loop repeats, bug->fix->win. Very velocity. Rocket fuel even.
(DIR) Post #B6qlbFPhz80H9YuJ4y by jonny@neuromatch.social
0 likes, 0 repeats
But its not just as simple as "OK if I read the tests I should be fine" because LLM code is often untestable. It writes code with function and class names that make it seem like a something does something, but they might just be flat wrong. Or there is some invisible fallback condition the LLM encountered while generating code and added to just make tests pass, but has entirely different behavior. If you've watched an LLM generate a project over time, you see it generating its own private language, and ive even seen it reinvent language features like function definitions themselves. Its names form part of an increasingly inaccessible web of meaning that no human can penetrate. Writing tests requires a kind of "information gap" where you can have enough intuition about what something does, but not how it does it, so you can a) know what it should do, b) make a strong assertion about that expectation, c) without mirroring the internal implementation's limits. That's hard! And really only possible when the foundation, (a) is true. Code must have an articulable purpose in order to be testable, that's tautological, that defines what failure is. But since LLM code increasingly detaches from any kind of stable description or expectation, even if the tests look very rigorous, you can't know if they are just tailored to the specific internal details of its function to eke out a pass, because it's hard to know what it should do anyway.So really you have to read the test code, the code under test, and also all the other code that might call the code under test. Aka you have to read everything. And rather than reading something that was written to be read, you're wading through a slop swamp. So you can't. It takes more time than just writing it. The erosion of testing is just an intrinsic part of the loop that you can't escape without breaking the spell of the slot machine, and it is what drives the loop.
(DIR) Post #B6qlbGF6u17ViyxMCO by jonny@neuromatch.social
1 likes, 0 repeats
So rsync rewriting all the tests puts the entire project in play. Now the entire protective surface has been sloshed through a layer of probability, so the loop must accelerate. Followup PRs add more carveouts with lengthy LLM justifications that sound perfectly plausible but amount to an erosion of the protective surface. We go from cumulative improvement to a random walk.
(DIR) Post #B6r9DuRICPOCLF3JUu by icedquinn@blob.cat
1 likes, 0 repeats
@jonny oh yeah those things try to sneak so much shit past you. its baffling the slightly nontrivial things they get completely right and the trivial things they get completely wrong
(DIR) Post #B6ubexrMyBiyxZopCC by elebertus@eigenmagic.net
0 likes, 0 repeats
@jonny i looked at the second picture showing the line changes and laughed out loud.Even the objective points you're making aside, a change that big is not something you just yeet through a single PR lol
(DIR) Post #B6ubey95uHu1qY6yp6 by jonny@neuromatch.social
0 likes, 0 repeats
@elebertusI think bun's rust rewrite is the single largest high profile yeet I have ever seen, if you haven't seen that yet
(DIR) Post #B6ubeyRsmQvomotz6m by elebertus@eigenmagic.net
0 likes, 0 repeats
@jonny I have not. I was reading more of your thread and I have had the exact same shit thrown at me regarding those tests.IME the biggest pitfall for LLMs and existing projects with over zealous people are "write a unit test for this function" which as you know, you need to express the types of behavior you expect. Gotta know how all that works, especially checking if something is in path; if not; check if thing is in path...shit is so bad man
(DIR) Post #B6ubeygPuOYdVthalM by jonny@neuromatch.social
0 likes, 0 repeats
@elebertusIve read so much LLM code at this point, there are still patterns that are present but elude my understanding, but one thing that's clear is that there are foundational flaw categories that are not improved upon by model version and appear in wildly different projects using wildly different models and harnesses. Testing is a big nexus of those flaws. I am not close to what would be a satisfying explanation of the dynamics, but every project suffers fucked testing problems.
(DIR) Post #B6ubeytB8wlY9Tfmee by sun@shitposter.world
0 likes, 0 repeats
@jonny better or worse than projects with no tests?
(DIR) Post #B6ubiA5NyCaHkNtoX2 by sun@shitposter.world
0 likes, 0 repeats
@david_chisnall @jonny my wife is a professional translator and she tells me translators are very afraid of AI because it can in fact replace a lot of their work.
(DIR) Post #B6ugNNd2vEnPRIYNwe by ooignignoktoo@shitposter.world
1 likes, 0 repeats
@sun @david_chisnall @jonny Don't worry if your wife loses her job to AI. Start worrying if she starts carving the words "no fate" in tables.
(DIR) Post #B6unZjQUPA9sCHW7UG by jonny@neuromatch.social
1 likes, 0 repeats
@sun@david_chisnall That's sucks, sorry that's happening to y'all. Of course whether or not something actually performs the same task or does it as well as a professional translator would is relatively orthogonal to whether the bosses will use that thing to increase profit by laying off workers and papering over failures.
(DIR) Post #B6unbplDpBzCQEDVqq by jonny@neuromatch.social
1 likes, 0 repeats
@sunidk hard to say if the wild free for all of untested vibe coding hooliganism is better or worse than the vibe coded stuff that purports to be Enterprise Grade
(DIR) Post #B6ungCPdY0k9plCbKa by sun@shitposter.world
0 likes, 0 repeats
@jonny @david_chisnall that is absolutely true, it's not always better and a lot of people don't care if it's worse. that said, she is worried because a lot of it really isn't bad, it's getting better rapidly. I think it's good to keep an accurate assessment of what it can and cannot do. I think there will always be a place for good professional translators and told her that, but a lot of people are gonna lose their jobs yes, and some translations are gonna end up worse not better.
(DIR) Post #B6up0JMTWMCJ8DsSi8 by jonny@neuromatch.social
1 likes, 0 repeats
@sun @david_chisnall definitely agreed on that, i am a filthy monolingual and so am the last person who can speak on translation, but as david is saying above it definitely can get to the point where it does the trick most of the time, up to a point when the communicative context escapes the space covered by the training data. that probably includes most of "business-relevant communication" that makes up most of the moneymaking end, but also i can imagine easily leaves people in e.g. complex legal situations navigating asylum or immigration proceedings in a really bad place, particularly when the very non-neutral language surface of the LLM starts to rear its head. like at once it is a miracle that i can get "pretty good" translation on demand any time, and not to be underrated, but as you say some things will be worse not better, and unfortunately most of that "worst" is going to fall on people who already have it bad.
(DIR) Post #B6up0JdqTm5m060Kmm by sun@shitposter.world
0 likes, 0 repeats
@jonny @david_chisnall I can speak from personal experience that LLMs in general still f***ing suck for any "legal work". disclosure I'm generally positive on AI (with serious reservations) but hallucinations are out the wazoo every single time I tried to use it to research or translate legal docs (I live in Japan)
(DIR) Post #B6urMpuxMHKiNjsGdE by jonny@neuromatch.social
0 likes, 0 repeats
@sun i figure the next stage and longer tail of the commercial end after the outrageous debt and speculation in the consumer market pops is latching onto workflow data to provide company- or domain-specific finetunes or strapon autoencoders. i'm sort of surprised on a longer term scale that isn't already the case, that the foundation models can be 'good enough', the economics restrictive enough where the ai giants are not licensing weights, the ui problem where people gravitate towards their favorite box, etc. haven't made that more of a thing yet. it's all still selling prefilled context windows at best, and all those businesses are doing terribly. i do wonder if the consumer market has been too poisoned by the chatbot modality to be able to adopt the 'whole life surveillance/whole life product surface' shift that google and microsoft have been trying to push, and these kinds of economically viable domain-specific products will always just look like Hated Work SaaS App.if you stretch the boundaries of what you consider "AI" to include an assemblage of purpose-built models and algorithms, then at the outer limits there you start to get back to "normal computing." the language | program interface is still, as far as i can see, an intractable one in the general sense, where the leap from text generation to tool calling can land you somewhere but it can also yeet you off into space. i would say on latest model anthropic and openapi models i get about a 50/50 chance of whether the thing is capable of spawning a subagent or whether it fails and makes excuses about it (and sometimes even simulates the output of a subagent but in the debug logs you can see it failed the tool call). That's a non-negotiable barrier that doesn't have an easy technological solution in sight that prevents the wildest subdomain takeover scenarios, but in the meantime a lot of stuff sort of works and for most people that seems to be alright.but yeah if one gets past all the sentient god-machine occultism, it's not hard to imagine this generation of tech giants going the same way at the prior (current?) gen, spinning off their magic product into a thousand derivatives that capture a lot of cash, displace a lot of labor, provide some useful services as a byproduct, etc. most of my serious reservations come from the economic and epistemic violence of it all, and i definitely do active research into the failure modes, but of course they do some things, and as you say it's important to stay appraised of the real capabilities both to calibrate criticism and appreciate what can be done.
(DIR) Post #B6urMqDOFk4vIuUzMe by sun@shitposter.world
0 likes, 0 repeats
@jonny can I just say thank you for being reasonable and thoughtful about this on here, straight up really hard to find that right now, it feels like almost everybody are either in a full blown moral panic or completely swallowed the marketing material (or are literally an ai-cultist as you mentioned, jesus christ). So, just, thanks. It was a pleasure to hear you write about it.
(DIR) Post #B6usGX5IZCJmBkGmeG by jonny@neuromatch.social
1 likes, 0 repeats
@sun i think it would be a real shame if the outcome of the alienation tech was to make us all self-alienate and hate each other. i find most binaries exhausting and unenlightening. like even with monster trucks, i can acknowledge how they're unbelievably wasteful and often enmeshed in some pretty hateful shit, while still being able to go like "whoa holy shit" when they are doing wheelies and flips and whatnot.
(DIR) Post #B6uzraB6j744B2bIsC by deutrino@mstdn.io
1 likes, 0 repeats
@sun @jonny and both ends of that bimodal distribution are tiresome beyond belief after hearing it nonstop day in day out for years at this point