(C) Daily Kos
This story was originally published by Daily Kos and is unaltered.
 . . . . . . . . . .



Automated software testing should remain a human activity [1]

['This Content Is Not Subject To Review Daily Kos Staff Prior To Publication.']

Date: 2026-02-25  

To reassure us A.I. won’t completely replace software developers, some people have taken to saying that A.I. won’t replace software developers, but will instead help them work faster by taking care of the boring stuff: boilerplate and tests. Therefore, learn A.I. or get left behind.

And that I find even more troubling than A.I. completely replacing devs, because it opens the door on something more insidious: the false assurance that the program is being tested when in fact the A.I. could very well be just deceiving you and deceiving itself. There are ways to deceive yourself while unit testing. I wrote a Medium article about that.

But an A.I. can fool you a lot faster than you can fool yourself. In the time it takes you to write one test based on one false premise, an A.I. can write dozens of tests for 100% test coverage that will still fail to catch a problem in production you could have foreseen.

Brian Asparro, chief operating officer of CarbonQuest, stands in a production room where liquid carbon dioxide is converted from a byproduct of a natural gas fired water boiler to a salable industrial product. (AP Photo/John Minchillo)

Having A.I. write boilerplate might seem more reasonable than having A.I. write tests.

When most of you think about boilerplate, you probably think about standardized text in laws or legal contracts that can be copied over from one document to another with few if any changes. For example, an end user license agreement for a software package will invariably include a paragraph declaring that the software is provided “as is.”

Likewise, in software projects, boilerplate is anything that can be copied over from one project of one kind to another of the same kind with few if any changes. For example, if you’re writing an iPhone app, you will invariably need a content view with a preview macro, which goes something like this:

import SwiftUI struct ContentView: View { var body: some View { // TODO: Write actual content } } #Preview { ContentView() }

Aside from the To Do comment, you probably won’t need to change any of that. Everything except that To Do comment is boilerplate. If you delete any of this besides the To Do comment, you’ll probably want to undo that deletion.

Boilerplate was already well taken care of by templates in your favorite integrated development environment (IDE) long before A.I. came along.

In some cases, boilerplate can be both necessary and irrelevant. The classic Java Hello World is a great example of this. Java programmers practically know it by heart, but in this context I should quote it.

class HelloWorld { public static void main(String[] args) { System.out.println("Hello, world!"); } }

Except for

System.out.println("Hello, world!");

pretty much everything is boilerplate. Since this class doesn’t have any instance functions and the implicit constructor is never invoked, it’s kind of irrelevant to put this little program is in a class. And since the argument array is ignored, it’s also irrelevant that the main procedure receives it at all.

Without these elements, however, your Hello World won’t run and you flunk your first Java lesson. Java 21 introduced a quicker way to write Hello World. Doesn’t completely eliminate boilerplate, in my opinion, but is a lot less to memorize.

But you don’t need to memorize that boilerplate. If you need a Java main procedure, you can avail yourself to a keyboard shortcut in Apache NetBeans or JetBrains IntelliJ IDEA: type “psvm” and then press Tab. Might also work in Eclipse. Vim probably has a shortcut for this as well.

So I guess I don’t have a problem with A.I. writing boilerplate. Though I don’t have much sympathy for people who complain about how boring it is to write boilerplate. We already have tools to take care of boilerplate.

A member of the USGS checks the information in the bridge station on Hermi's Bridge checking the water gauge along the Chattahoochee River near Paces Ferry Drive SE after Hurricane Helene passed the area, Friday, Sept 27, 2024, in Atlanta. (AP Photo/Jason Allen)

I do have a big problem with A.I. writing tests.

I understand the title of this article might seem contradictory to some of you. If the software testing should be automated, why can’t we let A.I. write the tests? By “automated,” I mean that at the push of a button, the computer runs a sequence of tests that have been prescribed by a human, when that person decides those tests should be run.

To understand what I mean, it might be helpful to compare so-called “manual” testing. One form of such testing is to simply run the program and see if it works as expected.

For example, let’s say you’re writing a program to help your customers prepare their state and local tax filings. So you run the program using the scenario of a head of household with a spouse and two dependents. Next, you run the program using the scenario of an unmarried individual with no dependents. Maybe you also have a scenario for a widower and pensioner, and a few other scenarios that cover every type of customer your program might have to deal with.

And that’s just fine. But now let’s say that in one of the scenarios, you notice a little problem with the program. So you exit the program, edit the source, recompile and run the program again. First you check the scenario that had the problem and you see that your edits to the source fixed that problem. Next, you run some of your other scenarios but not all of them.

Then you run the risk that one of the scenarios that you did not test this time around has a problem caused by your changes to fix the problem with the other scenario.

That example is obviously of practical importance, but maybe I should also give a simpler example without regard for its practical applications.

Consider a Boolean function that tells if an integer of small absolute value is prime or not. Such functions are already deployed where they’re needed, and a new primality function only has practical value if it’s for large prime numbers that might be used in cryptography. Writing a primality function for small numbers from scratch is worthwhile only as an exercise, or as an example.

Let’s say you write that function and load it into a REPL (or “language shell,” as some might call it). Obviously you can’t test every possible input, or wouldn’t want to even if you theoretically could. You’ll have to settle for testing a few typical cases, and whatever edge cases you can think of.

scala> isPrime(2) res1: true scala> isPrime(97) res2: true scala> isPrime(194) res3: false scala> isPrime(0) res4: false scala> isPrime(-7) res5: false

I don’t like that last one. I want it to say True instead of False. So I change the function, recompile, go back in the REPL and test it with a few negative numbers. Suppose however that I forget to test it with any positive numbers.

With automated tests, I can add tests for the negative number cases and have those run together with all the tests I wrote before. If what I did to make the function return True for −7 makes the function give incorrect results for the positive numbers, the automated test runner will let me know.

Kent Beck, the “inventor” of test-driven development, decided to give A.I. a chance. Well, he noticed that sometimes A.I. just deletes tests rather than figure out why they’re failing.

I can imagine A.I. doing a lot worse than that, however. I can imagine A.I. adding two characters to every failing test in order to make them pass.

@Test def testNegativePrimeIsPrime() { val result = isPrime(-7) // assert(result) }

Can you tell which two characters the A.I. added to make the test pass in this hypothetical scenario?

I might not catch that, especially if it’s one of dozens of tests that the A.I. spat out in a couple of seconds. But at least this is just a little toy example, and not a real program for real customers expecting correct results.

[END]
---
[1] Url: https://www.dailykos.com/stories/2026/2/25/2368905/-Automated-software-testing-should-remain-a-human-activity?pm_campaign=front_page&pm_source=latest_community&pm_medium=web

Published and (C) by Daily Kos
Content appears here under this condition or license: Site content may be used for any purpose without permission unless otherwise specified.

via Magical.Fish Gopher News Feeds:
gopher://magical.fish/1/feeds/news/dailykos/