The expected functionality of software lazarus / 21-General ---------------------------------------------------------------------- SymbolicFrank · 2022-01-11 08:19 Is it just me, or do most people nowadays expect almost all software to be buggy and show unexpected behavior, and it is up to them to find out how to work around all the issues to have it do what they want? Because it is. Also online services: more than half the time they don't work or require you to call the help desk to find out how to do it. From the last five online resellers I tried to order something, two were broken. And for the last five sites with services, four required calling  the help desk. Or, calling the help desk of a large internet provider because your internet kept malfunctioning and hearing that it was quite hard for them to help me because their first software system was offline and the second kept crashing. Etc. Ok, that testing has become almost non-existent for most of it and that it all has to be delivered today probably has a lot to do with that. SCRUM isn't helping, either. And you have to support a myriad of online APIs that keep changing all the time. Most of your development tools get updates at least once a month, that might break compatibility. And perhaps because I'm a programmer and expect it to work in a certain way. Or perhaps my standards are way too high. Dunno. ---------------------------------------------------------------------- 440bx · 2022-01-11 08:53 It's probably not just you... here are some statistical facts... 1. if it doesn't have bugs, it probably isn't software. 2. Talking to a customer service representative is almost as likely as having a beer with Jesus Christ on his 7th coming.  3. Talking to a _qualified_ customer service representative is, for practical purposes, impossible.  The reason being is, that guy is talking to Jesus and he is not about to let the guy go. 4. Testing occurs sometime late in the night and finishes before morning, either because the guy''s wife or Mom expects him to be there for breakfast. if you really want to help yourself, get some silicon lubricant and porn.  if you need help, get some blue pills from the corner drug store.  Whatever you do, don't pick up the phone for help... if you manage to get attention, it's probably a scammer. HTH. ---------------------------------------------------------------------- MarkMLl · 2022-01-11 09:47 I'd vote for this being moved promptly to "Other" since it is not just a software issue. Quote from: 440bx on January 11, 2022, 09:53:04 am3. Talking to a _qualified_ customer service representative is, for practical purposes, impossible.  The reason being is, that guy is talking to Jesus and he is not about to let the guy go. There /are/ exceptions. Without wanting to be accused of too-blatant advertising, AAISP for Internet and Wizards for telephony in the UK. Regrettably, this is a symptom of the complexity of modern systems: whether technical, commercial or social. People try to build things which are more and more complex, but interpersonal interaction has got to the point where it's virtually impossible to manage a team effectively. One of the symptoms of that is that since computer hardware has improved in both power and affordability by many orders of magnitude over the last 75 years, it's become acceptable to use vastly inefficient software architectures in an attempt to provide what a project owner thinks the customer will tolerate using a workforce uncommited to project success and unmotivated to learn efficient techniques to exploit the available equipment. Compare the Spitfire- design to first flight in four years, and in large-scale service four years after that, with the American F35. Or the software systems behind the American Moon landing in 1969, with those behind America's so-far-unsuccessful attempts to replicate that. Or the fact that these days you can't book an appointment with your local doctor without attempting to speak to an "agent" in a location which I'll describe as "offshore". It's not just software or technology which is broken. It's /everything/. Pax vobiscum. MarkMLl ---------------------------------------------------------------------- marcov · 2022-01-11 10:00 Quote from: 440bx on January 11, 2022, 09:53:04 am4. Testing occurs sometime late in the night and finishes before morning, either because the guy''s wife or Mom expects him to be there for breakfast. Not that that matters, since the manager shipped the product with the pre-alpha anyway. I don't think the complexity is the problem. I think it is more that the lifecycles are too short to do anything worthwhile than just try to pile up some stuff and try to connect them. Government IT is different (and even worse) ---------------------------------------------------------------------- Warfley · 2022-01-11 11:05 I have worked in software testing research, and there are a few interesting things about this. First you have often the problem that testing is the first thing that gets cut when the deadline approaches or the budget runs low. The reason is that it is rather hard to quantitavely measure the performance, you could have hours of testing with no single bug report but you can not tell if it is because the testers are lazy or because they simply did not encounter bugs. There should be at least 30% of the development budget going into testing (which is the minimum, more like 50% to be rigourus), but this isn't happening. Most companies do not even employ a specialized testing team. The best case scenario is that you have 3 independent teams, the development team, the testing team and the QS team that coordinates the two and acts as a bridge. In reality there is one development team that is expected to fulfull all the roles. The reason is that the testing team should not know about developing, because if developers create tests, they tend to test their implementation, not the target, which misses a lot of conceptual problems, basically testing around the bugs. Lastly, the problem is with the kinds of tests. Everyone does unit tests, because unit tests are easy and cheap to do, but just because two functions are correct, does not mean that the combination of those two functions is correct (there are approaches to this, I've worked on a project at uni, where a domain specific language was developed that was desinged such that this property actually holds true). Most bugs occur due to unforseen state changes. This requires integration and user tests. This is hard, because of the exponential nature of the problem. If you have n functions, there are n! possible combinations you would need to test. This is especially true for GUI applications, where the user could basically do anything at any time. While there are  a lot of solutions for automatically testing algorithmic code or batch jobs, AFAIK there is no good solution for systematically testing GUIs. And this is where user tests are needed, basically you need to pay people to use the software in it's completeness, but this is the most expensive part of testing, and the one that might yield the least results, because some bugs are so unlikely that you would need to test it for years before finding it. There are attempts to solve these problems, a new development cycle called test driven development tries to target these problems, especially the missing test team, by switching the development cycle around. First the tests are written (either as code as soon as the interfaces are known, or test specifications that need to be implemented afterwards against the interfaces) before coding starts. This way you don't have the problem that you have cuts on testing, because it is the first, not the last thing happening, second because the tests are written before the development, developers don't write the tests according to their implementation, but according to the goals. The second solution that is gaining more popularity, mostly due to advances in technology, is automated testing. Fuzzing, Model Checking or Symbolic Execution are often employed to automatically generate test suites, and are designed to specifically find all the corner cases that humans might miss. And i'ts basically free, you just take your program put it in a Fuzzing or SymEx engine, let it run over a day or two and get your results But there is one more aspect of this that makes this pretty hard, that is that most software is not a single product anymore, but rather a service. Most bugs are not introduced in original software, but due to updates. I don't know the exact figures, it's been a while since I've studied the research in that area, but I think it was like 70%-80% of all bugs are introduced through software updates. So while the initial product is very stable and well tested, updates are often not. The reason for this is that on updates, usually the new functionality is tested, but not the implications this might have on existing functionality. The assumption being, that what was correct previously, will still be correct afterwards, which might not be true as the new functionality can influence existing functionality. As we live in a time where most programs today are continous services, where you get new versions regularly, this of course means that there will be more and more bugs like this introduced. Something that simply did not happen that much before this business model became the norm. Back when I was working at research there where some novel approaches to this, where the researchers tried to developed a systematic and (partly) automatable approach that checks how new code interacts with existing code and tries to specifically test these interactions, but this was just very novel, very academic research at that time, I don't know what the current state of this is ---------------------------------------------------------------------- 440bx · 2022-01-11 11:20 Quote from: MarkMLl on January 11, 2022, 10:47:05 amI'd vote for this being moved promptly to "Other" since it is not just a software issue. I agree.  Quote from: MarkMLl on January 11, 2022, 10:47:05 amThere /are/ exceptions. Without wanting to be accused of too-blatant advertising, AAISP for Internet and Wizards for telephony in the UK. You're right. Fortunately, there are still exceptions.  In my case, the ONLY way I get to talk to someone who works for my ISP is for me to pretend that I'm calling to purchase additional services, if I don't do that, I don't get past the automated menu no matter what the problem is.  This is true, I am not exaggerating one bit.  Either you pretend you want to buy something or you're out of luck.  It's amazing! (not to mention completely disgraceful.) Quote from: MarkMLl on January 11, 2022, 10:47:05 amRegrettably, this is a symptom of the complexity of modern systems: whether technical, commercial or social. Quite likely there is some of that.  I believe that another significant factor is that everyone seems to be addicted to instant gratification.  We're in McDonalds society (and Europeans are lucky - you get a lot better burgers from McD than here in America - it's amazing that we have what may be the highest technology and the product is the lousiest burger on earth but, I gotta admit it's dirt cheap... probably because what you get is organic dirt between two bun with "secret sauce" and onions to cover up the sorry taste of the little organic between the buns.) I think social media has something to do with the general loss of quality in most things. Quote from: MarkMLl on January 11, 2022, 10:47:05 amOr the fact that these days you can't book an appointment with your local doctor without attempting to speak to an "agent" in a location which I'll describe as "offshore".That and, most of the time, the "agent's" job is to somehow find a way to prevent you to see a doctor (or whatever you're trying to get.) Quote from: MarkMLl on January 11, 2022, 10:47:05 amIt's not just software or technology which is broken. It's /everything/. I have noticed that it has gotten significantly worse in the past 4 to 5 years and I strongly believe it has something to do with a "president" that showed it's "ok" to have nothing that resembles decency, moral or ethics.  Some businesses seem to have taken a page out of that book.  Quote from: marcov on January 11, 2022, 11:00:11 amI don't think the complexity is the problem. I think it is more that the lifecycles are too short to do anything worthwhile than just try to pile up some stuff and try to connect them.I think both are causes but, yes, the software lifecycle is definitely short and I suspect that a driver behind that is the lack of initial design which mean, the product is "designed" while running.  IOW, and we know this, users are the testers now.  "Give us your money, we'll give you our software to test, for a year then, you have to pay again to keep testing".... what a deal !! @Warfley, I agree that testing is often and, unfortunately, a residual activity.  Particularly if the product is for outside consumption (we have users to test the product... whoopee!... and we charge them money to test... that's how we like to test our products) I am firm believer in using tools for every aspect of software development including testing.  A good tool can help enormously but, I also believe that nothing can take the place of a programmer who is meticulous and dedicated. There was a time when code walkthroughs were standard fare in software development.  It seems that has become a relic. At the end, it seems the driving force is to lower the cost and maximize the profit, a goal which is too often accomplished at the expense of quality. be it software or any other product. ---------------------------------------------------------------------- Thaddy · 2022-01-11 11:25 Imho both Agile and Scrum are false paradigms. They are false efficiencies. Not in theory but in practice. Where they go wrong is not in programming but in testing. ---------------------------------------------------------------------- MarkMLl · 2022-01-11 11:30 Quote from: Warfley on January 11, 2022, 12:05:29 pmAnd i'ts basically free, you just take your program put it in a Fuzzing or SymEx engine, let it run over a day or two and get your results But even so: you cannot prove a negative :-/ Considering a bit more of the meat in what was a particularly interesting posting: the asynchronicity inherent in a GUI, where the user can press any button or potentially resize any component in a completely unpredictable way. I wonder whether it would be possible to write an MVC-style program such that /all/ state-change requests went through a single queue, and there was a supervisor watching the sequence which at the very least logged what had happened before things went wrong? /However/: I can't help but feel that that is just reinforcing my point: it's throwing technical and in particular processor resources at a problem which should have been solved /much/ earlier in the design and implementation chain (as should the problems of a former president of the USA, a former lady entrepreneur, and so on). MarkMLl ---------------------------------------------------------------------- Thaddy · 2022-01-11 11:34 Quote from: MarkMLl on January 11, 2022, 12:30:38 pmBut even so: you cannot prove a negative :-/ Yes, Karl Popper versus verification.... You are oh so correct. You can verify into infinity but one falsification disproves it. ---------------------------------------------------------------------- Warfley · 2022-01-11 12:07 Quote from: MarkMLl on January 11, 2022, 12:30:38 pmBut even so: you cannot prove a negative :-/ In general you cannot prove the correctness of a program, because it can have infinitely many execution paths. But you can use a lot of techniques to prove reduce the execution paths to be finite. For example one approach I've worked on was to try and detect when a state was reached that was already encountered, and by this make infinite programs finite. This is incredibly useful to test state machines with a limited state set. So even if it has loops, which would result in infinetly many paths (e.g. A -> B -> A -> B ...) you can detect when you are in the same state and not follow the chain any further. Quote from: MarkMLl on January 11, 2022, 12:30:38 pmConsidering a bit more of the meat in what was a particularly interesting posting: the asynchronicity inherent in a GUI, where the user can press any button or potentially resize any component in a completely unpredictable way. I wonder whether it would be possible to write an MVC-style program such that /all/ state-change requests went through a single queue, and there was a supervisor watching the sequence which at the very least logged what had happened before things went wrong? I also worked on something similar, not GUI events, but signals, but generally speaking it's the same problem, as signals are executed on a specific thread. There was the idea that we compute for every possible instruction (in case of a GUI application, for every event), which data it touches as read or write, and then compute all the queues with all combinations of "colliding" events. So for example if you have an event that touches object A and one that only touches object B, and these objects do not interfere with each other, it doesn't matter in which order you execute them, so you can test them independently from one another. But if both touch the same object, for example one writes a value and the other one reads that value, it does matter in which order, so you need to always test them in both possible orders. That said, this was all active research at the time I worked on it, and I don't know what the current state of this is. Whats also quite interesting is the usage of constraint languages like OCL. In OCL you can write for each object invariants, and for each function pre and postconditions. An example for a store class an invariant would be something like: "Store.Items.ForAll.Stock >= 0", and a precondition for a function BuyItem could be something like "Store.Items[ItemID].Stock > 0" and the post condition could be "Store.Items[ItemID].Stock := Store@pre.Items[ItemID].Stock - 1". This describes the class on an abstract level, can be used as documentation, but also used to generate test cases without knowing the actual implementation. Also such a specification can be verified for it's consistency (e.g. test if there is any combination for functions where the post conditions might violate an invariant, or if there is any combination that results in a deadlock, where there is no single function that could be executed because no precondition is true). The problem is that this requires effort, and is something mostly is only academic or done by only large companies. The biggest software testing company is microsoft, which does a lot of research in this area. But all their solutions are proprietary and cost a lot of money. So what is mostly seen in companies is mostly fuzzing, because this is rather easy. The more advanced stuff requires either people who are specialized in this sort of stuff and can make use of the academic works, or pay a huge amount of money to Microsoft to use their solutions. So while in theory a lot of this stuff is really great, and at the institute where I worked, we had a lot of success and found a lot of bugs in open source software, like the (e.g. the gnu core utils was one of our main testing target) it's not something you will see very often "in the wild" so to say. But it's always complementary, it can help find really corner case problems, but a good testing cycle requires more. One thing is for example, automated approaches can not detect visual bugs, or user experience problems. Also it requires a good testing strategy. All the tools don't help if you don't use them correctly ---------------------------------------------------------------------- Warfley · 2022-01-11 12:20 Quote from: Thaddy on January 11, 2022, 12:25:48 pmImho both Agile and Scrum are false paradigms. They are false efficiencies. Not in theory but in practice. Where they go wrong is not in programming but in testing. Agile development is great, but it's a shame that no team lead or project manager has actually read the agile manifesto, and don't replace the centralised structures with agile ones, but just ommit a lot of the planning structures and hope that it works out. It's like saying, if we use more roundabouts and reduce the speed limit we can remove traffic lights and reduce traffic congestion, and all the city planners understand is: Ok just remove traffic lights and all will work out ---------------------------------------------------------------------- Thaddy · 2022-01-11 12:22 Quote from: Warfley on January 11, 2022, 01:07:53 pmIn general you cannot prove the correctness of a program, because it can have infinitely many execution paths. That is not true. One can often prove correctness by falsification. And that can be done by sheer logic and on paper. (Dijkstra comes to mind,  he rarely used hardware https://en.wikipedia.org/wiki/Edsger_W._Dijkstra ) Although programmers rely on logic, they often lack the skills to apply it properly. For completeness: https://en.wikipedia.org/wiki/Falsifiability This has been a leading guide throughout my professional carreer, not only as a developer. ---------------------------------------------------------------------- Thaddy · 2022-01-11 12:23 Agile is great, in theory, as I mentioned. ---------------------------------------------------------------------- Warfley · 2022-01-11 12:37 Quote from: Thaddy on January 11, 2022, 01:22:11 pmThat is not true. One can often prove correctness by falsification. And that can be done by sheer logic and on paper. (Dijkstra comes to mind,  he rarely used hardware https://en.wikipedia.org/wiki/Edsger_W._Dijkstra ) Although programmers rely on logic, they often lack the skills to apply it properly. Yes you can do it on paper because your brain does a lot of optimizations "automatically". These are the optimizations I have been talking about (e.g. recognizing when you are in the same state), which I have worked on. For example you can proof correctness using Hoare logic and while the verification of program with this is rather easy on paper, it requires finding the loop variants and invariants, which is often easy for humans if we look at them, but not so easy to do automatically. And when we are talking about projects with 10 kloc you cannot simply do all of the verification by hand, this does not scale, so you want automated approaches that can do the reasoning for you ---------------------------------------------------------------------- Thaddy · 2022-01-11 12:49 Well, mostly agree, but note that currently a lot of work done in the field applying AI (e.g. MIT). I suspect it won't be long we will see that applied to software development and testing. ---------------------------------------------------------------------- MarkMLl · 2022-01-11 13:07 Quote from: Thaddy on January 11, 2022, 01:22:11 pmThat is not true. One can often prove correctness by falsification. And that can be done by sheer logic and on paper. (Dijkstra comes to mind,  he rarely used hardware https://en.wikipedia.org/wiki/Edsger_W._Dijkstra ) That turns out to be a myth, which isn't supported by the primary references: in one of the EWD papers he specifically discusses what should be on a programmer's desk in a computing laboratory. https://www.eevblog.com/forum/programming/how-to-build-programming- logic/msg3908525/#msg3908525 But he was definitely a strong proponent of "think before you do". In most disciplines testing is something you do to demonstrate that design and implementation is correct, not to investigate whether it is correct. MarkMLl ---------------------------------------------------------------------- SymbolicFrank · 2022-01-11 14:52 If things keep growing worse at the same speed they did for the last five years or so, it will become impossible to deliver any software that survives the first user trying random stuff in less than a few years. My own frustrations are primarily the lack of design and interfaces. The best location to see those problems are REST services. And yes, actual testers instead of misusing the customers for that would be a great start. While Microsoft might be the largest producer of testing equipment and methods, IIRC they first outsourced their own testing to India and later scrapped it altogether. I think the only things that are still tested are, if it confirms to the demands of the managers and if the lowest levels of the kernel keep working. And they still have a very expensive subscription and payment service, if you really need something fixed. As an example, it is impossible to print on A4 paper if you have a keyboard configured for the "EN-US" language when installing the printer. Only Letter is allowed in that case. That bug is from Windows 10, build 1903 and still not fixed. Although they put it in their "outstanding issues" document a few years ago... People seen to accept all that. On the other hand, many steps in configuring and starting your computer and software now require a valid Microsoft certificate and often a working internet connection and Microsoft server. That's probably the largest single point of failure at the moment. ---------------------------------------------------------------------- Thaddy · 2022-01-11 15:04 EWD 279 op. cit.: "1) A program should be regarded as the design of a large class of possible computations: these computations —the "making" of which is left to the machine— are the final product about which the programmer must make his assertions. 2) Correctness proofs are essential. Programming first and then testing is putting the cart before the horse, program testing can be used to show the presence of bugs, but never to show their absence. Program composition and demonstration of correctness are two activities that should be merged. (To quote A.G. Fraser: "I just want to make the point that reliability really is a design issue, in the sense that unless you are conscious of the need for reliability throughout the design, you might as well give up.") 3) The requirement that the correctness proofs can be given without undue amounts of intellectual labour calls for conscious exploitation of our powers of abstraction and a programming tool in the usage of which our abstractions can be suitably reflected. 4) A program should not be regarded as an object all by itself but as a member of a family of related programs, i.e. either alternative programs for the same task or similar programs for similar tasks; the function of the levels of abstraction is to indicate to what extent the different members of the faculty can be mapped on each other. We have to recognize the similarity between "the changed decision" and "the postponed decision". 5) The potentially large number of members of the program family considered requires that the correctness proofs are concerned with the family rather than with an individual member. This requirement gives a rather clear picture of the logical function of modularity; the programming tool has to allow (or even to invite) the corresponding "textual encapsulation"." Notably from 2 I rest my plea. ---------------------------------------------------------------------- SymbolicFrank · 2022-01-11 15:40 Well, as a programmer, you are encouraged strongly to become a system administrator, to build your programs by downloading and attaching building blocks together. LEGO and scripting. That seems to be much faster than writing it all yourself. But of course, only if you change the requirements to the capabilities of those blocks / libraries and throw correctness (or even understanding how it works and what it does, exactly) out of the window. And when enough of those blocks are updated, it stops working altogether. Update only when really needed and make sure you collect and archive all the tools and components you use to produce your software. So you can actually build the bugfixes and improvements a year from now. And with FPC / Laz: you get no guarantees, but you can test and fix it yourself, if needed. Which is more than you get with most components and libraries. ---------------------------------------------------------------------- Warfley · 2022-01-11 16:39 Quote from: SymbolicFrank on January 11, 2022, 03:52:19 pmWhile Microsoft might be the largest producer of testing equipment and methods, IIRC they first outsourced their own testing to India and later scrapped it altogether. I think the only things that are still tested are, if it confirms to the demands of the managers and if the lowest levels of the kernel keep working. And they still have a very expensive subscription and payment service, if you really need something fixed. The main area where microsoft uses most of their automated testing is when incorporating third party code, e.g. drivers. The main reason why bluescreens where so prevalent in Windows XP and earlier was because of bugs in drivers, so Microsoft started a program where they supplied the drivers through windows directly (either as part of the installation or via the windows update), but for a third party to get their drivers into that system these drivers must pass an online verification process, which basically does automated correctness proves. And this system is really good, bluescreens rarly happen (I had like 1 since windows 10 came out) and it makes Microsoft a fair bit of money. ---------------------------------------------------------------------- MarkMLl · 2022-01-11 16:55 Quote from: Warfley on January 11, 2022, 05:39:08 pmThe main reason why bluescreens where so prevalent in Windows XP and earlier was because of bugs in drivers, And the main reason that bluescreens were both common and fatal was that MS had moved far too much untrusted code into ring zero in the interest of performance. To misquote Benjamin Franklin, "He who sacrifices robustness and security for performance deserves none of them." And you never heard of /him/ spending his time watching cat videos. MarkMLl ---------------------------------------------------------------------- SymbolicFrank · 2022-01-11 17:01 @Warfley: good point. Although, for drivers that are part of Windows Update, lots of new computers were bricked when Windows 10 was still new. Happened twice with computers I had to install. And the same thing is happening with Windows 11. It even goes further than just Windows itself, because your UEFI BIOS also needs to be signed by Microsoft. (Or at least, files on the virtual drive inside). And with a new OS, changes are needed and updates are installed. That's probably why they now are part of the Windows Update program as well. Which is interesting, because there's a whole certificate chain to make it all work. You need one to write to that BIOS when it is in Secure Boot mode. Which is required by Windows. And which makes it easier than ever to brick them. Which already happened on a small scale some months ago. @MarkMLI: definitely.