Subj : Unit vs. integration testing (was: Not debugging?) To : comp.programming,comp.lang.java.programmer,comp.lang.lisp From : rem642b Date : Sat Sep 03 2005 12:48 pm > From: Peter Ammon > I don't believe that unit testing eliminates debugging. But would accept that comprehensive unit testing, to make sure every tool does its job correctly before it's called from a higher-up tool, reduces the needed amount of whole-system debugging by one or two orders of magnitude? > We had a bug where an update to our library disabled some features of a > client program. After some investigating, it was determined that the > client program was doing this: > if (library_version == 3) > instead of this: > if (library_version >= 3) It's obvious to me that the client program was never unit tested properly. Since there was a line of code that checkd the library version, there should have been a unit test whereby different library versions were supplied by a test rig and the client program was checked to make sure it performed correctly in all such cases. Testing boundary cases would have sufficed for this line of code. > Our fix for this bug was to detect the client and report 3 for the > version if the client was the offending program, and otherwise report > the true version. Given that the client program had a bug, it made more sense to fix that bug rather than accomodate it by introducing a matching bug in another module. What happens, per your "our fix", if somebody later adds new code to the client to deal specially with library version 4, and that code doesn't work, because the library is reporting 3 instead of 4? Imagine how long it'll take to track down that bug, then it can't be fixed without re-breaking the version 3 bug? The best immediate fix is to document client as having a bug that makes it not work with any version beyond 3, and roll back the library to version 3 on any system used by that client until such time as somebody can fix the client, meanwhile on any system with properly working clients they get a full working version 4 of the library. > This is the sort of bug that gets caught in integration testing. Only because complete unit testing of the client never occurred. With a TDD-style of testing, although not necessary actual TDD (development), each such line of code that tests a condition would be deliberately disabled in various ways, and tests would be set up to signal malfunctions whenever any line of code other than the correct one is in that position. Perhaps having a contest between two teams would make this practical: One team designs tests to signal malfunction whenever the line of code is wrong. Other team contrives bogus lines of code to try to fool the test. Rule is you can't make a super-complicated perverse line of code such as: if ((library_version == 3) || (library_version == 4) || (library_version > 7)) which would pass all boundary tests for versions 1,2,3,4, and also pass far-beyond test such as 100, but fail at untested 5 or 6. The bogus lines of code have to be approximately the same complexity as the correct line of code. So the predicate could be == != > >= < <= or any other test of a similar nature that might be in the programming language, as well as stupid mistakes such as = or +=, and the constant value can be any integer, and the variable name might be wrong. At some point two different lines of code would be recognized as correct (with < or <= for different adjacent constants), and a policy decision would have to be made to reject one of the two in favor of the other, then the game could continue with that restriction on the bogus-loc-generator team until the tests are sufficient to make the bogus-loc-generator team resign the game. However very early in the game/contest the bogus line of code actually in the program would have been signalled as malfunctionning by the test for version 4. (I admit the test rig for this game, whereby various features are attempted and a malfunction is signalled if some supposedly available feature turns out not to be available, is a bit complex. But maybe the first team would enjoy such a challenge?) > I can't think of any way that unit testing would have helped this > situation. I'd be very interested in hearing how this sort of bug > would be approached in the test-first "no debugging" philosophy. Have my ideas about need to fully unit-test the client, and the contest, helped you think of a way that unit testing the client could have helped? I'm not saying that unit testing is guaranteed to help in this case, because unit testing isn't perfect, and could possibly overlook this particular case. I'm just saying that unit testing would have a good chance of having found this bug already, and the better quality the unit testing is the lower the chance of the bug slipping through undetected. Like I said at the top: I don't claim unit testing eliminates *all* bugs, merely that it eliminates 90-99% of potential bugs already, depending on the quality/exhaustiveness of unit testing, leaving just those very few remaining bugs to occur later. .