(C) Daily Kos This story was originally published by Daily Kos and is unaltered. . . . . . . . . . . The folly of letting A.I. write tests for software still going strong [1] ['Alonso Del Arte', 'Alan Singer', 'Chitown Kev'] Date: 2026-07-23 It happened again last week. A speaker at a tech meet-up told a room full of software developers that they should let A.I. take care of the boring stuff so that they can concentrate on the fun stuff, the stuff that matters. The boring stuff, according to the presenter, includes boilerplate and tests. My facepalm could not have endeared me to the presenter. And it wasn’t even the first time I heard that particular bit of idiocy. I wrote about this back in February. I stand by what I wrote back then, but unfortunately in the Daily Kos change-over to WordPress some of the content of that article has been mangled or deleted and it doesn’t quite make much sense. And maybe I can figure out a more eloquent way to explain this to regular people who know very little about software development beyond the broadest generalities. Or maybe even if I do find that eloquent way to explain this, I still can’t snap people out of this madness. Letting A.I. write tests for software is kind of like letting Bart Simpson grade his own exam. But at least with Bart Simpson you have some idea of what his motivation is for giving himself an A+. And presumably in this scenario you didn’t let Bart write the test questions himself. What’s the motivation for A.I. to tell you all the tests it wrote are passing? None, really, it’s just plagiarism filtered through a pseudorandom number generator with no intelligent intent behind it. In the craze to let A.I. automate everything, we’re automating a lot of the wrong things. Automating the writing of boilerplate might be a good idea. Automating the writing of tests is definitely a very bad idea. Boilerplate is important. Without the appropriate boilerplate, your computer doesn’t know what to do with a file. But, by its very nature, boilerplate tends towards a statistical average. And since A.I. seems to be very good at giving the statistical average, A.I. should be good at writing boilerplate. Boilerplate is anything you can pretty much copy from a template or archetype. We had templates long before this A.I. fad made so many people insane. For example, if you want to create a Spring Boot app in Java, there’s a template for that. For example, a project of mine called Dearborn Cinema. It’s open source, available on GitHub. In what seemed like an auspicious start, the project was both a student project and a project with an actual customer in mind. The actual movie theater this would have been for wasn’t in Dearborn, but after that fell through, I’ve continued working on that project on and off by myself. If it had been strictly a project just for fun, I probably would not have used Spring Boot. Because Spring Boot requires the Maven build tool, the folder hierarchy is more complicated than in an Apache Ant project or a JetBrains IntelliJ IDEA project using the proprietary JetBrains IntelliJ IDEA build tool. The following screenshot shows some of the folder hierarchy. Screenshot of the macOS Finder showing some of the folders and files in the Dearborn Cinema Spring Boot project. Screenshot, macOS If the project is Java-only, why is it necessary to have a java folder in main and test enclosing com? That’s how the Maven build tool requires it. If this project was just for fun, I would not be using Maven, I would be using whatever Eclipse, IntelliJ or NetBeans uses by default. Do take the “Today at 9:16 p.m.” in the screenshot with a grain of salt. As far as Git is concerned, I haven’t worked on this project since May (Git is a version control system). The screenshot is actually atypical because it suggests that there has been work on src\test\java\com\dearborncinema long after the initial commit (the project was started on Windows machines, but Git lets you use forward slashes on Windows and presumably backslashes on macOS). A lot of files get checked into the first commit of a Spring Boot project that are never touched again. Such as, for example, AppTest.java. The AppTest provided by the template, after we boil away the boilerplate and parse out the comments, boils down to a single test that does nothing other than assert true . So you run that, and your computer tells you that all the tests are passing and you think your program is great. The only use of that is making sure that the testing framework is properly connected to the project. AppTest from the template actually includes the comment “Rigorous test ;-)”. I really don’t know if that was supposed to be a self-deprecating joke by the template author or if it was sarcastic, as in “I already know you won’t bother to change this into actually meaningful tests.” I haven’t substantially changed AppTest yet, but I have added other test classes. My last commit to entity.ScreeningTest was two years ago, in which I selectively disabled some of the tests. One of my classmates had insisted on a continuous integration workflow that runs all the tests whenever commits are pushed to the main branch, and sends out an e-mail to all the project participants if any of the tests fail. This is not a problem on a project that only has the “rigorous” test from the template. The silly test passes and no notification gets sent out. On another student project he actually just deleted all the tests I had written, leaving only the silly AppTest , rather than ask me what was going on with the new tests. I don’t remember that classmate’s name, as his name is not on the Dearborn Cinema repository and I’m fairly certain he removed himself from the list of project participants once it became clear this project would not count for a business or at least for a grade in the course. The e-mail from the continuous integration workflow that there are failing tests does annoy me, so on that project I avoid pushes with failing tests. At this point I could probably delete the continuous integration workflow and no one would care. I vaguely recall another teammate asking “Why are you testing the mock movie data maker?” I didn’t have to explain to him things like why it’s important that the program not schedule a movie when the previous one hasn’t finished playing. But testing the mock movie data maker, which would only be used in the test suites and not in anything customer-facing, that seemed to him to be excessively fussy on my part. I get it, testing can be tedious. But due diligence is not meant to be fun. For example, if you’re writing software for an ATM, you should probably write a test that checks that when the ATM dispenses money to a user, that user’s account is reduced by the correct amount. That’s not my idea of fun. If I wanted to have fun, I can think of at least a dozen other things I’d rather be doing. A.I. doesn’t care if your ATM doles out all of its money without recording any of the transactions. A.I. doesn’t care if my movie scheduling program schedules two movies to play at the same in the same auditorium. A.I. doesn’t care about any of the things that could go wrong with the projects you entrust it to work on. Murphy’s law says that whatever can go wrong will go wrong. It doesn’t say when things will go wrong. I want things to go wrong early, in the dev environment, when they’re fresh on my mind, and not later on, in production, when actual customers might be affected. This suggests test-driven development (TDD) as the best way to ensure that your program works correctly when you ship it to your customers. TDD starts with unit tests, which are short and quick to run. A.I. doesn’t understand TDD. A.I. tries to do the whole program in one go, and instead of making a small, incremental change to fix one defect, it tries rewriting everything. Many beginning programmers do test their programs, but only after the whole thing seems to be working correctly. If all the tests are passing, that confirms the programmer’s belief the program is perfect as it is, and he or she is unaware of defects that might be lurking in the program. If even one test is failing, that’s annoying because you’re not sure if the mistake is in the test itself or in the program being tested. With TDD, by contrast, you test your tests by seeing them fail first. Either way, testing after rather than before tends to make testing feel like a pointless chore. Matti Bar-Zeev explains for DEV Community why testing after with A.I. is even worse than testing after without A.I. AI Agentic coding, and in that sense, LLMs in general, have their reality. Since they are stateless, their reality is the accumulated “problem” context they pick along the way. When your code is already implemented, it becomes part of the context, and therefore, part of the reality for the AI Agent. In this case, the Agent will write tests that satisfy this reality, for example, if you implemented code that has a logical flaw within, it will write a test which asserts this logical flaw. Suppose for example that the ATM checks how much cash is available and rejects a withdrawal if it has more cash than the requested amount. The A.I. writes a test confirming that fallacy and next thing you know you have an ATM that just won’t give out any cash ever. You could try emptying out the cash drawer, and then the ATM reduces a customer’s balance without giving that customer any cash. That would not be good. It might interest you to know that as I started writing this article, the A.I. in Microsoft Edge actually suggested to me that I should use the redundonym “ATM machine.” But in a sense, Bing Copilot knows that’s redundant. When I searched for “redundoym” in Bing, the A.I. gave “ATM machine” and “PIN number” as two prominent examples. A.I. does not know what it seems to know. It in fact does not know anything, it does not understand anything. [END] --- [1] Url: https://www.dailykos.com/stories/2026/7/23/800065754/community/the-folly-of-letting-a-i-write-tests-for-software-still-going-strong/ Published and (C) by Daily Kos Content appears here under this condition or license: Site content may be used for any purpose without permission unless otherwise specified. via Magical.Fish Gopher News Feeds: gopher://magical.fish/1/feeds/news/dailykos/