[HN Gopher] The presumption that computers are working correctly
___________________________________________________________________
The presumption that computers are working correctly
Author : ColinWright
Score : 257 points
Date : 2023-09-30 09:34 UTC (1 days ago)
(HTM) web link (emptycity.substack.com)
(TXT) w3m dump (emptycity.substack.com)
| denton-scratch wrote:
| A big part of the problem with these prosecutions is that the
| prosecuting authority is the Post Office itself. Until this case
| came along, I had no idea that the Post Office had these powers.
|
| So effectively, the Post Office was acting as both complainant
| and prosecuting authority.
|
| "The Post Office Investigation Branch (IB) is the oldest
| recognised criminal investigations force in the world. For more
| than 335 years it has worked to detect offences against the post
| and prosecute the perpetrators of these crimes."
|
| https://www.postalmuseum.org/blog/the-post-office-investigat...
| masfuerte wrote:
| Anyone can bring a prosecution in England. In theory it
| provides a way to hold the powerful to account but it's very
| expensive so almost all private prosecutions are actually by
| corporations.
| denton-scratch wrote:
| It's true that anyone can bring a prosecution. It's also true
| that the Crown Prosecution Service can take over a private
| prosecution, and then drop the case.
|
| Obviously, the CPS should have done this in the Post Office
| case. During the relevant period, the CPS was underresourced
| and overloaded (it still is now).
| vjk800 wrote:
| The presumption becomes problematic mostly when the operator of
| the computer is the same party who is being persecuted. 1) It is
| difficult or impossible for any outsider to inspect the workings
| of the computer, 2) the operator has every incentive to fake the
| outputs of the computer or hide the errors.
|
| Related issue, but from the opposite perspective: some years ago
| I remember the copyright trolls using their own logs of the
| bittorrent swarm as an evidence in courts. Faking of this data
| would have been both very easy and pretty much impossible to
| detect by an outsider. Also they had a massive financial
| incentive to do so. The fact that this "evidence" was considered
| in the courts seemed outrageous to me.
| agentgumshoe wrote:
| Ah the Prenda Law saga, such memories
| [deleted]
| dboreham wrote:
| The Windows task bar auto-hide feature has been broken
| (spontaneously decides to quit hiding, but doesn't adjust the
| desktop viewport accordingly, so the bottom of any app window is
| obscured by the errant taskbar), since its first beta in 1994.
| chrisjj wrote:
| https://archive.ph/wip/AJl8i for permanance.
| ozim wrote:
| I can see why they changed it. I expect in last 3 decades amount
| of computer evidence grew up to amounts where there is no
| practical way to prove computer was operating correctly.
|
| If I present a digital photo as evidence it would be dismissed by
| defense lawyers to easily. Then basically any digital evidence
| would be dismissed the same way.
|
| I can see how it was easy to manage in the 80's and not anymore
| in 00's.
| chaps wrote:
| Similar happens with shotspotter in the context of early stages
| of criminal court proceedings.
|
| https://law.stanford.edu/publications/soundthinkings-black-b...
| stevesimmons wrote:
| David Allen Green is a great commentator on law in the UK. During
| the Brexit period, he was a contributing editor to the Financial
| Times, with similarly insightful analyses of how Boris Johnson et
| al were misusing the government's legal powers.
|
| Anyway, David also posted this to his non-gated "The Law and
| Policy Blog" at [1].
|
| [1] https://davidallengreen.com/2023/09/computer-says-guilty-
| an-...
|
| There are more comments there, including this one from me
| (written for his blog's legal/non-technical audience, not for the
| HN crowd!):
|
| "There has been a big shift in the practice of computer
| programming in the last 25 years. In the "old days", software was
| delivered on floppy disks or tape, and pretty much had to be
| correct first time.
|
| Nowadays, everything is on the web, "agile" means the
| functionality gets delivered incrementally in 2-week sprints. It
| is regarded as ok to leave out the difficult cases or ship bugs,
| provided they don't affect too many users too much of the time.
| In other words, the software is never really finished.
|
| With that in mind, it becomes simple to rebut the presumption of
| correctness: if the company's tech organisation describes itself
| as "agile", using "scrum" etc, its software - by design -
| contains bugs and omissions, both known and unknown.
|
| Fortunately, modern development practices means evidence for this
| will be in the dev tickets in GitHub, Azure DevOps or whatever
| other systems their dev teams use."
| _dain_ wrote:
| I had a visceral negative reaction to the idea that computers
| should be presumed to be working correctly, but I was having
| trouble articulating in precise words what was wrong with it. I
| mean, obviously we get work done with use computers every day
| and they seem to work alright. A commenter at that link does a
| good job:
|
| _> To treat the operation of an extensive software system as
| being a mechanical device is as gross a category error as
| treating an airliner as being merely a collection of nuts and
| bolts. In either case we can satisfy ourselves that the
| elementary parts are performing within specification without
| having gained any confidence that the complete assemblage will
| function properly under all conditions to which it will be
| reasonably exposed; indeed we won't even be confident that we
| understand what it was exposed to. Even showing that it has
| passed a collection of tests isn't adequate: one needs to see
| what unfixed issues are being registered against it and how
| they are being resolved (fixed? Swept under the carpet to keep
| the numbers below since arbitrary threshold?) The opposing side
| in case hinging upon accuracy of computer records should have
| the right to access the issue database, and absence of a well-
| run system or a large number of issues in it (or evidence that
| issues have been "scrubbed") should be taken as a strong
| argument that the computer records are unreliable._
|
| _> Much could be learned by the software industry from the
| culture of aviation safety, where even minor incidents are
| followed up by detailed investigations intentionally run in a
| non-judgemental way and the findings distilled into public
| reports and recommendations. Similarly, one of the best legal
| developments in recent years has been the increasing tendency
| to publish detailed judgements._
| bostik wrote:
| Thank you, I have been looking for a resource like this, one
| that would come with an RSS feed. Now I have one less thing to
| hunt around for.
| layer8 wrote:
| This will also become interesting in the context of AI, because
| it's unclear what it means for AI to "work correctly".
| SoftTalker wrote:
| My usual presumption is that the computers are not working
| correctly, as I've seen way too many cases where they in fact are
| not working correctly. I don't even believe Google Maps when I'm
| driving, I always sanity check the route and double check street
| names/exit numbers when it tells me to turn. I've been burned too
| many times.
| thesalsabear wrote:
| While this is a question of what the court should presume, should
| we ask what does a software developer presume or doesn't:
|
| A software developer would presume proper working of software if
| it is being used widely. I would presume JavaSDK or pandas
| library to be working properly, inspite of the fact that pandas
| that 3.5K bugs file on GitHub issues. But if I am thinking about
| integration a less known library or using a less known software I
| am less confident about its working.
| nine_k wrote:
| Given enough eyeballs, all bugs are shallow. As such, they are
| complained about by thousands of people, for years.
|
| Less ironically, one can expect widely used features of widely
| used software to have been tested in realistic conditions so
| whatever bugs there might be are at least known. But it takes
| an insane amount of testing to cover the phase space even of a
| moderately complex piece of software: look how large is the
| test suite of SQLite.
|
| Apparently the only working approach to less buggy software is
| to compose software from small, self-contained,isolated, well-
| understood parts, so that you can reason about the whole more
| easily. This is the philosophy behind the original Unix. This
| is the philosophy behind functional programming. This is partly
| the philosophy behind actor systems (even though reasoning
| about async interactions is harder). I expect to see more and
| more of these in practice.
| makeitdouble wrote:
| Yes. Bugs are there, but either they're shallow, well known
| and worked around, or have low impact for the majority of
| people using them.
|
| The epitome of that was utf-8 support for a long time: there
| were a huge number of bugs, but alaphabet based languages
| would be mostly ok, and most devs would know it was a mess
| and work around that as much as they can.
|
| To me one should expect any software to be broken in numerous
| ways, and take the time to check what happens around the area
| that are critical to them.
|
| > compose software from small, self-contained,isolated, well-
| understood parts, so that you can reason about the whole more
| easily.
|
| That looks awfully close to the micro-service approach as
| well. The approach can be good, but it still gives me pause.
| TeMPOraL wrote:
| > _That looks awfully close to the micro-service approach
| as well. The approach can be good, but it still gives me
| pause._
|
| Microservices aside, GP's statement seems to imply
| composition is like addition wrt. complexity.
| Unfortunately, in reality it's multiplication. Complexity
| multiplies with each moving part. There's hardly a better
| example of that to what we inherited from "original Unix" -
| chaining small, focused tools in pipes turns into a write-
| only mess after around the fourth pipe.
|
| Software complexity isn't simply a divide-and-conquer
| problem, because we pay a penalty for each individual
| piece, and each piece of glue that connects it with other
| pieces. That's the whole reason we rely on _abstraction_
| and _generalization_ so much - the ability to take a bunch
| of "small, self-contained, isolated, well-understood
| parts", put them in a box, and stick a higher-level name to
| it. Recursively.
| phire wrote:
| As a programmer, it's my experience that the computer itself
| usually works correctly. It's almost never a hardware bug.
|
| But software.... software gets things wrong all the time and the
| computer hardware just blindly does what the software told it to.
| bmitc wrote:
| > As a programmer, it's my experience that the computer itself
| usually works correctly. It's almost never a hardware bug.
|
| By hardware do you mean CPU?
|
| Also, these days, "hardware" is still something that's either
| got a microcontroller, FPGA, or some other thing that is eother
| running or was designed with software. The area is quite muddy,
| especially if you deal with hardware systems of a collection of
| various computers and hardware components.
|
| And I'm not so sure things aren't CPU or IC bugs that get
| bubbled up. The amount of Heisenbugs and various other
| transient bugs make me question that.
|
| In my experience with instrument control systems, the bug could
| be anywhere from a cable to the UI or anything in between.
| These systems had to be developed in strict layers to help
| isolate bugs and also workaround them.
| cf141q5325 wrote:
| You are describing the exact same experience people who arent
| programmers have with software. It usually working and being
| almost always problems with the user.
|
| It also leads to the same horrible misjudgements as in the
| article mentioned above. If you dont know what to look for you
| will come to erroneous convictions by ruling out the error
| cases you do know.
| whoknowsidont wrote:
| >it's my experience that the computer itself usually works
| correctly.
|
| Oh you sweet summer child. Hardware has so many little "quirks'
| or hacks to make it work correctly and it often doesn't! It's
| often expected that really low-level, "touching the metal"
| software makes up for these little bumps that happen quite
| often.
| murderfs wrote:
| As a programmer that works on drivers, it's my experience that
| hardware almost never works correctly. "We'll fix it in
| software" might as well be the official motto of the hardware
| industry.
| m-ee wrote:
| As a firmware engineer I'm horrified at the presumption that
| hardware is always operating correctly. Many times I've been
| told "your new firmware is broken" and the real problem is
| something as mundane as an improper capacitor or resistor
| value.
|
| Also just look at the errata sheet for any microcontroller.
| Hardware bugs are so prevalent we have a special term and
| semi standardized document to describe them.
| PartiallyTyped wrote:
| A good friend of mine is a hypervisor/kernel engineer, and
| I can swear he has said the same thing, almost word for
| word.
| amelius wrote:
| That's funny, because this just came in:
|
| https://apnews.com/article/iphone-15-overheating-apple-
| softw...
| runeks wrote:
| What makes you think this isn't just a software bug?
| amelius wrote:
| Because "too hot to handle" tells me that this should
| have really been prevented at the hardware level, and
| this is also common practice.
| andromeduck wrote:
| Power control is all configuration. Hardware merely
| provider the signals and mechanisms for SW control.
| amelius wrote:
| Within limits, yes.
| harry8 wrote:
| What you say is completely true in my experience. Comically
| so.
|
| However when using a computer application and noticing
| unexpected and erroneous behaviour, what proportion is the
| time can the bug be traced back to hardware vs software?
|
| I'm going with about 1 to 1000. I think that's conservative.
| Do you see it differently?
| gwd wrote:
| But thinking about a court case, what should be the
| presumption? Should it be up to the prosecution to prove that
| the RAID controller / network card didn't corrupt the data or
| that the GPU did the math correctly? Or should it be up to
| the defense to prove that it didn't? And how would each side
| demonstrate their case to a reasonable level of certainty?
|
| As others in this thread have said, for bits of the system
| that have clearly defined parameters and get extensive
| testing (by like, millions of people), I think you can begin
| with the assumption that those bits are correct: you can
| assume the CPU is adding numbers correctly, that the compiler
| compiled the code correctly, that the spreadsheet executed
| the formula it was given correctly, and it's up to the
| defense to prove otherwise. Bugs in the CPU / compiler /
| spreadsheet software would get caught pretty quickly.
|
| But for bits where the right answer is _not_ clear, and / or
| _doesn 't_ get extensive testing -- the formulas inside the
| spreadsheet, custom-written software -- the assumption should
| be that it does _not_ work, and it should be up to the
| prosecution to demonstrate that the software is working
| correctly.
| cf141q5325 wrote:
| You are treating hardware like software by arguing with the
| experience we have made so far with a given model.
| Environmental conditions as well as it being different
| physical objects is a huge problem for hardware. The data
| about error cases is also simply not recorded. You get at
| best statistical reliability with small sounding chances
| for errors that are still happening all the time.
|
| You can also not make any statement about reality by
| arguing with feasibility.
|
| edit: I am unfortunately not qualified to tell you how
| feasible ways to deal with reality in this case could look.
| Which doesnt mean that i cant identify some error cases.
| You also require different levels of competence for
| designing a rocket and recognizing a non functioning one.
|
| edit1: In regards to the usage of statistics as a basis one
| should also remember confidence and these cases
| https://www.science.org/content/article/unlucky-numbers-
| figh...
| auggierose wrote:
| Mathematical verification of both hardware and software
| should be mandatory, at least as long you rely on it for
| the law. Simple as that. For example, how can you trust a
| photo, when tons of computational photography has been
| applied to it, and you don't know the exact amount the
| algorithms can change the photo?
| PartiallyTyped wrote:
| That's easy to say, until you've tried to write a
| provably correct piece of software or tried to do model
| checking.
|
| I have tried to build models for the schemas that
| databases have, not their implementation, just the
| schema, and it is difficult AF. Using z3 for this might
| as well be trying to start a fire with a pair of rocks.
| auggierose wrote:
| What exactly have you tried to do there? Isn't a schema
| already a model? What properties did you want your model
| to have?
|
| Yes, in general it is difficult, because then software
| becomes mathematics. And that's on top of the software
| doing something useful and what the user wants.
|
| But I think we are now getting the tools to get there.
| You will still need to become a mathematician, though.
| PartiallyTyped wrote:
| > What exactly have you tried to do there? Isn't a schema
| already a model? What properties did you want your model
| to have?
|
| Yes, a schema is a data-model, and a database can tell
| you whether the row you are trying to insert satisfies
| the constraints, but it can't tell you whether an
| instantiation of the database exists, nor can it tell you
| how to construct it.
|
| There are 2 issues here, first proving that an
| instantiation exists; and second proving that there's a
| sequence of insertions that satisfy the constraints.
| Notice that some tables can have non sat constraints.
|
| This becomes significantly harder when you are dealing
| with recursive definitions, cycles in the dependencies,
| foreign keys that refer different tables across the same
| constrained columns, type compatibility, and on and on
|
| In other words, the space of all databases described by a
| schema may not even be constructible :upside_down:
| auggierose wrote:
| Ok, I understand. So first you want to see your schema as
| a property on a database (which is just a bunch of tables
| with data, I guess), and see if there is actually any
| such database, or if the property is always false.
| Second, assuming that there are any such databases: Given
| certain commands which operate on databases, is there any
| sequence of commands which transform the empty database
| into a database such that the schema property holds for
| it? Actually, you probably would want the property to be
| true for every intermediate step as well, right?
|
| Sounds interesting! Although in practice, solving this is
| probably not that important, because if you cannot come
| up with some examples of constructing databases for your
| schema and application, then the schema is the wrong one
| pretty much by definition.
| PartiallyTyped wrote:
| You needn't think of databases as tables and data, you
| may think of them as trees of structs, ie an object
| relational model and show satisfiability of the model by
| constructing a tree of dependencies.
|
| You can do it incrementally for parts of the model, but
| that doesn't guarantee that long range dependencies are
| satisfiable.
|
| Yes, you then need to show that it is incrementally
| constructible, which is a different can of worms... the
| tree view above isn't actually fully correct because you
| duplicate entries mapping to the same object. Because of
| this duplication, the tree may refer to an object that
| can only be inserted later.
|
| Oh and the encoding must be able to accommodate arbitrary
| length arrays and arbitrary objects ... which makes usage
| of SMT solvers difficult to say the least.
|
| While I agree that for the average company this isn't
| useful, and the average engineer won't shoot their feet
| on purpose, what I am doing needs to accommodate all
| those scenarios because I have no guarantees of sanity.
| auggierose wrote:
| Now I am interested in what exactly your use case is :-)
| Maybe you would want to model this first in a general
| purpose interactive prover such as Isabelle, and proceed
| from there. But in the end you seem to be looking for a
| push-button algorithm for proving the consistency of a
| particular logic, and I can tell you, that's probably not
| gonna end well.
| PartiallyTyped wrote:
| I can't say much more unfortunately, not yet at least.
|
| I have semi-sketched the idea in Dafny instead of
| Isabelle. Direct encoding of arrays makes it difficult,
| but I am thinking iterative deepening might just work. I
| am working on a Z3 approach for this, and I am crossing
| my fingers.
|
| If Z3 doesn't work, I will try emitting Dafny code
| instead and hooking up to the compiler.
| andrewaylett wrote:
| The rule is either "on the balance of probabilities" or
| "beyond a reasonable doubt". Neither of which was well-
| applied in the Horizon cases. "Mathematically proven" is
| almost always the wrong standard for legal cases, as it
| doesn't speak to the quality of the proof.
|
| The relevant standard ought to have been "beyond a
| reasonable doubt", but obviously even those who _did_
| know better managed to convince the court that doubting
| the computer was unreasonable. Apparently without
| manually checking the output against reality, even when
| the defendent brought their own numbers?
|
| I strongly suspect that the racial discrimination that's
| being revealed by the press played its part in that -- no
| need to worry about the computer, or even about reality,
| if one's mind is set anyway.
| xyzzy123 wrote:
| That's not really economic at present, hopefully it will
| be one day. There are a very small number of verified
| programs, no mainstream operating systems and no major
| application suites. We just don't have the technology for
| it yet.
|
| It does not seem very practical to throw out all digital
| evidence (eg photos and videos) because there is some
| chance of misprocessing. Who even records in analog
| today? What corresponding standard would you demand for
| analog evidence? Even in analog there were film issues,
| processing issues, noise, smudges, tampering and
| uncertainty, people have been dealing with these sorts of
| confounders for a long time.
|
| We don't generally require "mathematical certainty" for
| other types of evidence. Personally I think GP was
| proposing a reasonably balanced commonsense perspective.
| auggierose wrote:
| Common sense does not get you very far in a world where
| any video or audio can be fake, and you cannot really
| tell which one is and which one isn't. Common sense is
| officially done for. Bring on the math.
| gwd wrote:
| Come now; imagine a world where nobody could ever be
| convicted without mathematical verification of all
| components of evidence. Nobody could be convicted of
| anything, since even if we had 10 people and three 4k
| video cameras all see someone pull out a gun and shoot
| somebody, we can't have mathematical certainty even of
| the three video cameras, much less the unreliable wetware
| memories of 10 people. We might as well give up having
| trials entirely.
| auggierose wrote:
| I rather rely on the memories of 10 people than on 4K
| cameras which can be hacked. I don't mind relying on a 4K
| camera for something like tennis, to decide if a ball is
| out or not. But if it is just up to software if I go to
| prison for 10 years or not, I very much DO mind.
| ericjmorey wrote:
| Are you aware of just how unreliable eyewitness testimony
| is?
| helsinkiandrew wrote:
| > As a programmer, it's my experience that the computer itself
| usually works correctly. It's almost never a hardware bug.
|
| Wow there are some patronising people responding to your
| comment.
|
| I think this is mostly true for 95% of software and the
| experience of software developers. But CPUs are full of bugs
| that only manifest when certain code is run - perhaps the order
| of instructions or when in certain states. Often these are
| hidden/fixed in software - compilers/drivers/microcode etc.
| drt5b7j wrote:
| It's appears to you this way because you're an insider in
| software. If you were in hardware, you would believe the
| opposite.
| oldgradstudent wrote:
| No.
|
| Hardware people are fully aware of their shortcomings, but
| they also are fully aware that software is even worse.
|
| Far far worse.
| TedDoesntTalk wrote:
| It does happen to Hardware. Here are two older examples:
|
| "The Intel Pentium series of CPUs had two well-known bugs
| discovered after it was brought to market, the FDIV bug
| affecting floating point division which resulted in a recall in
| 1994, and the F00F bug discovered in 1997 which causes the
| processor to stop operating until rebooted."
|
| https://en.m.wikipedia.org/wiki/Hardware_bug
| [deleted]
| thaumasiotes wrote:
| > and the F00F bug discovered in 1997 which causes the
| processor to stop operating until rebooted
|
| You reminded me of the Things I Won't Work With piece that
| observed in passing that the behavior of dioxygen difluoride
| was surprisingly well described by its chemical formula. (
| https://www.science.org/content/blog-post/things-i-won-t-
| wor... )
| k0k0 wrote:
| Your experience is quite limited. Too limited to have a
| meaningful opinion on this subject frankly.
| cf141q5325 wrote:
| > Your experience is quite limited. Too limited to have a
| meaningful opinion on this subject frankly.
|
| I am having quite the big problem with this getting
| downvoted. The poster is spot on here and this needs to be
| communicated. Not doing so is irresponsible. Overconfidence
| in this regard is extremely dangerous and there is no nicer
| way to say this.
|
| You can not reasonably assume that you are able to judge
| something works reliably without understanding how the error
| cases look. Especially if these are capsuled from you and or
| you dont know what to look out for.
|
| This is at the very core of why something like the post
| scandal was possible. People repeating the same mistake in
| this very thread is just really bad. The only thing worse
| would be not telling them this.
| rambambram wrote:
| Enlighten us then.
| cf141q5325 wrote:
| A very common problem is bugs caused by cosmic radiation if
| you want an example. But bugs in silicon are also not rare
| and really difficult to catch. You really got to stick your
| neck out and hope you can actually proof that through stuff
| like debugging with electron microscopes. Especially if it
| becomes an issue of responsibility between different
| departments. The monetary stakes make this all the more
| difficult.
|
| Hardware bugs are simply capsuled better so people dont
| recognize them as such. Its the same mechanism at play as
| for people who assume software normally functions. They are
| also just unaware of the inner working.
|
| Stuff is just really complicated and people should manage
| their expectations. I get that going from assuming to have
| a solid base to no such thing existing is difficult to
| handle but there is no sensible way around that. Not doing
| so gets you stuff like the article.
| k0k0 wrote:
| No one else but you can provide you with experience.
| Hardware has both bugs (defects) and it fails, sometimes in
| ways not gracefully. An experienced multidisciplinary
| developer is certain to encounter them at some point.
| Depending on the discipline this may be a very common
| occurrence or it may be rare. Devs working at either
| extremes of scale are more likely to encounter these
| problems. Defects may go unnoticed due to software
| workarounds of all things
|
| Here is just one example to get you started:
| https://nakedsecurity.sophos.com/2011/08/10/bh-2011-bit-
| squa...
|
| https://edc.intel.com/content/www/us/en/secure/design/confi
| d...
| tivert wrote:
| > As a programmer, it's my experience that the computer itself
| usually works correctly. It's almost never a hardware bug.
|
| > But software.... software gets things wrong all the time and
| the computer hardware just blindly does what the software told
| it to.
|
| That's not a distinction that's relevant to this article. By
| "computer" it means a system composed of hardware _and_
| software (and which is almost always what "computer" is used
| to mean):
|
| > The presumption that computers are presumed to be operating
| correctly, unless there is evidence to the contrary is what
| lawyers call "a presumption of evidence".
|
| > This means that a court can be satisfied that a relevant fact
| can be established just by computer records, unless there is
| evidence that the computer is not working properly.
|
| > And so when the computer record shows, for instance, a
| financial shortfall by postmaster or postmistress, the court
| will accept that as evidence of an actual shortfall - unless
| the defendant can show that the computer was not operating
| correctly.
|
| > In short, when the computer record is the essence of a
| prosecution case: _computer says guilty_.
| [deleted]
| User23 wrote:
| What do you mean by works correctly? By what standard do you
| say the software gets things wrong? After all it's an
| indisputable fact that computers only ever do exactly what you
| tell them to.
| albert_e wrote:
| > After all it's an indisputable fact that computers only
| ever do exactly what you tell them to.
|
| too broad a generalization i think.
|
| who told these computers to do what they did?
|
| https://www.bbc.com/future/article/20221011-how-space-
| weathe....
| jfoutz wrote:
| How does that work? I'm not a lawyer. I'm a lazy programmer.
|
| So, does the defense get to subpoena the logs, and depose a
| programmer to explain why some random request returned a 502?
|
| Does the defense get, like, red team access?
|
| Although I'm a lazy programmer, I do think a lot about failure
| modes. If my freedom is on the line, I can probably think up a
| few (orders of magnitude) test cases. Happy to provide a shell
| script or python or golang binary to check. Heck, if it comes
| down to it I'll provide a stack of fortran cards. Do I get to see
| the source? That "few" is probably going to move a decimal place
| or 5.
|
| I get volume is not reasonable in the eyes of the court. Scale is
| how you(I) find problems, but whatever. Gimmie source, gimme
| access, I doubt I could beat Knuth, but with my freedom on the
| line I could give him a run for his money.
|
| How do CVE's count? If I find evidence of log4j do I auto win?
|
| This makes me think of the Jepsen tire fire talk. The system
| probably mostly right. But there's no way it's correct. What's a
| "fair" level of access to prove that?
| Guvante wrote:
| There already exists rules for certainty in law. After all just
| because you were seen by fourteen witnesses found with the
| murder weapon and admitted to the crime doesn't mean you are
| 100% guilty.
|
| It could have been your identical twin doing those things as a
| silly example.
|
| That doesn't mean the prosecution loses.
|
| The goal is to meet that certainty which is by definition below
| 100% but high enough that society accepts it.
| KaiserPro wrote:
| its divide and conquer and abuse of power.
|
| The post office found that money was "missing". It went after
| the sub-postmasters. The sub-postmasters didn't know that it
| was a systemic issue, they were told to plead guilty to reduce
| their sentence.
|
| Unless you have a credible lawyer, and have a clue that
| something in the system might be wrong, then you're not going
| to know.
| rwmj wrote:
| The real problem is this isn't a "right" that the postmasters
| could exercise. You're accused of stealing money. You're not
| paid very much in the first place and have only limited legal
| aid, and may even be in prison (on remand) already. There's no
| "defense team", you only have a solicitor assigned to you at
| the police station. You're also not a programmer. How are you
| going to argue that the database which shows you stole money
| wasn't working correctly?
| ric2b wrote:
| And the only evidence the prosecution needs to show is some
| log messages from the system saying the math doesn't add up?
| Nothing else?
| rwmj wrote:
| Yes. Try reading the article?
| qingcharles wrote:
| In the US at least it is hard to bring source code into the
| trial, even though you technically have a right.
|
| I remember one case where the FBI installed malware via a
| browser exploit on thousands of computers to get their IPs and
| one defendant tried to get the source code but the government
| managed to keep it secret.
| Terr_ wrote:
| That sounds like one of the "Playpen" cases, for which the
| EFF has a writeup on the hacking tool. [0]
|
| Perhaps also relevant would be _United States v. Lafon Ellis_
| [1], where the defendant sought access to the source-code of
| a DNA fuzzy-matching service.
|
| [0] https://www.eff.org/pages/playpen-cases-frequently-asked-
| que...
|
| [1] https://www.eff.org/deeplinks/2021/02/federal-court-
| agrees-p...
| Roark66 wrote:
| >Perhaps also relevant would be United States v. Lafon
| Ellis [1], where the defendant sought access to the source-
| code of a DNA fuzzy-matching service.
|
| I'm glad at least in this instance the defendant got access
| to the source code. He was being helped in pursuing this by
| EFF so I assume there were volunteers that helped with the
| analysis. But what if your case is lower profile? What if
| you can't afford to hire 10 programmers for 3 months to
| write test cases?
|
| Also I wonder how such a request would go down in a UK
| court.
| sjy wrote:
| > So, does the defense get to subpoena the logs, and depose a
| programmer to explain why some random request returned a 502?
|
| In theory yes, but you can get some idea of how it worked in
| practice from the transcript of the trial of Seema Misra [1].
| This was one of the few cases where the postmaster pleaded not
| guilty and expert evidence was presented to the jury. The
| defence complained throughout the trial that their expert did
| not have adequate access to the underlying computer records.
| New documents were identified during the course of the trial,
| and the defence applied unsuccessfully for the trial to be
| stayed at the close of the prosecution case.
|
| [1]: https://journals.sas.ac.uk/deeslr/article/view/2217/2151
| didntcheck wrote:
| The last time I looked into the Post Office scandal, the
| impression I got was that "we just had too much faith in the IT
| system!" was a convenient story, but the high-ups knew full well
| about the problems, and full well that they were persecuting
| innocent people, but did it anyway to cover up their own
| failings. If so, people need to see substantial jail terms for
| this. Perverting the course of justice is no misdemeanor. There's
| even allegations that Paula Vennells knowingly lied to Parliament
| wahnfrieden wrote:
| Why say punishment via rules can solve this, in other words,
| putting people in cages at the end of the day (or in other
| cases, taxing a permitted behavior), instead of advocate for
| structurally removing that power from them?
| Jochim wrote:
| Punishment is an intermediate solution.
|
| Holding executives responsible for their misdeeds is the bare
| minimum in the meantime.
| dazzawazza wrote:
| There is substantial evidence that the people in power in the
| Post Office Board dismissed the evidence from the sub post
| masters because they were, more often than not of Asian descent
| (Indian, Bangladeshi, Pakistani,...). So the Post Office Board
| had a mental block to joining the dots because they were
| racist.
|
| When they were eventually confronted with the overwhelming
| evidence of hundreds of false convictions they doubled down
| because they lack any sense of morality.
|
| So while it is a sad technical tale it is also an indictment of
| British Board room culture.
|
| The whole sorry affair has been covered by Private Eye magazine
| for over ten years.
| switch007 wrote:
| Private Eye is an absolute gem. It's quite quirky and jokey
| but it's the real deal if you want to know how this country
| is really run.
| envsubst wrote:
| > they doubled down because they lack any sense of morality.
|
| I don't believe you know what their internal mental states
| are.
| iudqnolq wrote:
| We can and do reasonably infer mental states from actions.
|
| If someone knows they are sending innocent people to
| prison, benefits from doing so, and continues doing so we
| can reasonably infer they lack morality.
| dan-robertson wrote:
| Do you think an increase in punishment the executives expected
| would have led to the problems for subpostmasters being
| resolved faster or slower?
| fanf2 wrote:
| It doesn't matter either way, they need to be punished for
| the way they treated the people who were under their power.
| actionfromafar wrote:
| Will punishment cause time-travel, you mean? Probably not.
| dan-robertson wrote:
| Consider it as either a counter factual or a question about
| similar events in future.
| Spooky23 wrote:
| Your question makes a good point. Punishment is only
| effective at the bottom of the funnel.
|
| A well designed system, whether a computer or an organization
| should always have controls that guide things to where they
| should be. That means carrots and sticks.
|
| In a place like a post office, that means separation of
| duties, audit, and clear lines of accountability. Things fail
| when nobody asks awkward questions or looks.
|
| I do think that leadership responsible for a debacle like
| this must be punished. Their incompetence destroyed lives of
| innocent people.
| zzbn00 wrote:
| In principle there was already some separation of duty as
| solictors in England owe a duty to the Court regardless of
| who pays them. However it seems not to have been enough in
| this case. Hopefully will be looked into.
| etothepii wrote:
| I think you mean barristers.
|
| The problem here was that as many sub-postmasters were co
| owned by a husband and wife with children. There was an
| enormous incentive for one of the sub post masters to
| plead guilty and say, "my wife had nothing to do with
| it." Doubly so as the sentences would often be on the
| cusp of the "suspendible range", a guilty plea combined
| with contrition is much more likely to get a judge to
| suspend your sentence if he can.
|
| Of course, this "plea-bargaining" is standard practice in
| the states but we aren't used to it in the UK.
|
| The issue is that no barrister would allow their client
| to plead guilty without giving them a little canned
| speech about not pleading guilty if they didn't do it and
| having them, "sign their brief".
| didntcheck wrote:
| Am I inferring correctly that you are suggesting I may cause
| a perverse incentive, wherein they would be more motivated to
| cover it up due to the potential of punishment? I don't think
| that applies, as the punishment I'm suggesting is _for_ the
| cover-up and knowing prosecution of innocent people, so the
| incentive would be to _not_ do that. Not to mention they
| already seemed fully motivated to continue saving face anyway
|
| Having rich people face actual material consequences to their
| life, rather than "punishments" like losing a job or honor,
| would hopefully reduce corporate malpractice, yes, as they
| now know that they are not invincible (obligatory note that I
| am talking about criminal responsibility, not suggesting we
| do away with LLCs or something)
| dan-robertson wrote:
| I think that might be an effect, but I'm not sure. If
| something like this were to happen again, the thing I would
| most want would be whatever rules would most speedily lead
| to any prosecutions against subpostmasters being sopped and
| any convictions being overturned (with just compensation
| for the wrongful convictions).
|
| I think to achieve that it has to be better in expectancy
| for someone who knows about the problem (and widely
| believed by that person too) to act sooner to resolve the
| problem than to try to cover things up, but I don't really
| know what rules would help. For example, maybe an
| alternative thing that would work would be some kind of
| whistleblower rules like in finance[1].
|
| In the hypothetical where I am stood with my hand on the
| lever choosing between stopping the scandal sooner and
| letting executives off entirely or having more wrongful
| convictions for longer but later throwing the executives in
| jail, I will choose the former every time, even though it
| feels unjust.
|
| [1] In finance the whistleblower rules are designed to
| incentivise people to not be accomplices to crimes done by
| their superiors/coworkers. The incentives must be high
| because, no matter what the rules say about it,
| whistleblowers will expect to not work in finance again
| after blowing the whistle. The rules work because those
| large-scale financial crimes generally are hard to without
| one's competent colleagues finding out.
| momirlan wrote:
| Computers, as in hardware, work as designed. Are there bugs in
| the software run on them ? Then the title should be about "bugs"
| or logic errors, not "computers"
| zubairq wrote:
| After a lifetime of fixing bugs I'm often surprised that computer
| systems work at all. I had something similar to the failed post
| office system affect me personally, and everyone just rejects
| that the computer system could be at fault, even when presented
| with evidence, as it too too much work for them to try to
| understand.
| bboygravity wrote:
| I would add some nuiance and state that a lot of popular
| software is broken to the point that it's main function (the
| reason that the software exists) is not working.
|
| Some examples: -- The websites of various airlines (TAP,
| Vueling, KLM) have often had bugs in them (in the past 2 years
| when I used them) to the point that they don't allow you to
| book tickets or log-in with your account. I would say they
| where broken more often than not. -- Instagram and Google have
| been deactivating people's accounts (automatically) for no
| apparent reason. I've had this happen with Instagram 3x in the
| space of 2 months after which I just gave up and stopped using
| it. -- Skype, no explanation needed for those who use it. --
| Teams fails in many amazing ways no matter how often they
| "update" it. I've heard stories about the mute button not
| actually muting, Teams suddenly reversing the order of the
| letters in your sentence as you're typing in a chat, it just
| shuts down in the middle of a call, unresponsive, etc etc etc.
| The amount of productivity these bugs in this software burn
| worldwide must be significant and the real cost to companies in
| the form of wasted man-hours is likely higher than whatever
| they pay in licensing fees to Microsoft. -- Telegram can't
| handle an external monitor (the scaling gets all messed up
| causing you to have to restart Telegram to fix it). -- iCloud
| takes between seconds and an hour or so to sync files even when
| all devices are on a good connection.
| hulitu wrote:
| >Teams suddenly reversing the order of the letters in your
| sentence as you're typing in a chat
|
| I think they had a button for this on the right click menu.
|
| TBH, Teams is the quintesence of today's SW development state
| of the art.
| bboygravity wrote:
| To add some meta irony to my previous post: as you can see in
| my previous post the formatting got all mangled. The new-
| lines (enters) that I typed got ignored upon posting. I
| cannot think of any good reason for that "functionality" from
| a user perspective?
|
| Also I can't edit that post anymore to fix it.
|
| And to think that HN is an example of relatively good
| software that seems un-bloated, super stable, fast and seems
| to always work :P
| fineIllregister wrote:
| > I cannot think of any good reason for that
| "functionality" from a user perspective?
|
| Markdown ignores single return because people used it to
| manually wrap their plain text but HTML can wrap
| automatically on the fly.
|
| Add two spaces before the return to make a line break
| `<br>`. Two returns in a row start a new paragraph `<p>`.
| fanf2 wrote:
| Hacker News does not use Markdown. Double space at end of
| line does not work, but double newline is a paragraph
| separator.
| d_tr wrote:
| The rapid growth of the software industry is both the reason
| for the unsalvageable mess and for companies being able to
| spend obscene amounts of money to somehow keep shit together.
| At least that's how I feel about it.
| almatabata wrote:
| Sadly this often happens because often times the machine works
| 99% of the time. So when the thing starts failing in 1% of the
| cases people will dismiss it because it worked in the previous
| 99% of the time. People have trained themselves to trust the
| results so questioning it does not even enter their minds.
|
| It does remind me though of the book Systemantics:
| https://www.amazon.com/Systemantics-Systems-Work-Especially-...
|
| One of the core thesis in the book is :
|
| The system itself does not actually do what it says it is doing
| (The Operational Fallacy).
|
| This people and engineers as well often forget. Just because
| the system says it does X does not mean it actually does it.
| People would do well to cultivate a healthy skepticism towards
| those complex IT systems. They might not fail most of the time
| but they inevitably will.
| HPsquared wrote:
| Occam's razor can be double-edged sometimes.
| SanderNL wrote:
| It is too much work. It's way too complex.
| bmitc wrote:
| > I'm often surprised that computer systems work at all.
|
| I also remain contunually amazed that anything works. It
| boggles my mind, because I have seen how software gets built.
| How can it possibly all work?
|
| I think my answer to this conundrum and seeming paradox is that
| it doesn't all work. We have trained ourselves and been trained
| to accept things that aren't working as something else.
|
| And like you say, people treat the computer system either as
| all knowing and infallible or as a scapegoat.
| vasco wrote:
| Any sufficiently complex system operates constantly in
| failure mode
| uoaei wrote:
| This is an incomplete view of complex systems. It is more
| accurate to say they operate constantly in _recovery_ mode.
| simonh wrote:
| Which of course also applies to human cognitive processes.
|
| We've had millions of years of evolution to prune out
| problems, but in many cases the problems were papered over
| by a kludge that introduced other problems, which were then
| kludged, etc. No wonder the complex systems we build are
| such a mess. They're a reflection of ourselves.
| taneq wrote:
| They're not yet a reflection of ourselves, only of our
| thought processes. They're flawed, as are we, but we've
| been ruthlessly battle-tested in the only arena that
| matters.
| bmacho wrote:
| > The Fundamental Failure-Mode Theorem (F.F.T.): complex
| systems usually operate in a failure mode.
|
| https://en.wikipedia.org/wiki/Systemantics
|
| now this is depressing. I hate everything and I live in a
| constant 'everything is shit' state of mind, but this book
| hates things way more than me.
| K0balt wrote:
| To me, modern computers, in particular, are an enigma, and I
| am an electronic engineer specializing in digital systems.
|
| Imagine something with 200 billion individual parts with
| individual functions (and that's just the cpu) that is
| presumed to work perfectly flawlessly billions of times a
| second for a decade.
|
| By contrast, the crankshaft on your car goes through a bout 1
| billion revolutions before it is worn out. Your heart might
| beat 4 billion times before you die, if you are really lucky.
|
| The precision and reliability we take for granted from modern
| computing hardware is dumbfounding.
| zubairq wrote:
| That makes sense, yes we have trained ourselves to see things
| that don't work as working , like frogs in boiling water!
| psychoslave wrote:
| While the metaphore is great, it's actually stand on a
| myth.
|
| That's all the more interesting in such a context to
| evocate it. We often take for granted what feels like aptly
| grounded easy to grab information. I think that is where
| LLM feels the most as a terrible threat to human societies,
| because they can at large scale provide customized very
| convincing utter bullshits.
|
| https://en.m.wikipedia.org/wiki/Boiling_frog
| bostik wrote:
| People outsource their critical thinking all the time, but
| with computers the failures are more egregious. There is less
| recourse to correct things, and information propagating
| through systems can make the error more persistent. Also:
| people take the verdict from a computer as a given even when
| reality and computer disagree.
|
| I've used this story a couple of times before on HN, and if
| it hadn't happened to mother's coworker, I would have thought
| it a preposterous joke. Mom's colleague was buying something
| during her lunch break, and had her card rejected. Got the
| same result in another store, so obviously thought her card
| had just broken.
|
| Went to her local bank branch to ask what's wrong. The clerk
| took the card, looked up from her screen and against the
| evidence in front of eyes declared: "You are dead." Once that
| kind of flag starts to make its way through our computerised
| systems, it is _very_ difficult to reverse. It took several
| weeks to get the error corrected, and in the meanwhile other
| systems that trusted the bank as a source of truth had picked
| up the information and started to apply it to their accounts.
|
| Eventually it came to light that a full namesake's estate had
| had their accounts finally closed, and the clerk doing that
| had done only a name search when applying the "account
| terminated, owner deceased" flag to the accounts. Bank never
| took responsibility for their process failure. They fought
| for a good time to even have to accept that there had been an
| error in the first place.
|
| Which incidentally is not entirely unlike the situation in
| this perversion of justice.
| citizenpaul wrote:
| I had a similar experience as your moms colleague.
|
| I transferred 20k from savings to checking. During the
| process I think what happened is the web banking page froze
| and allowed me to hold the enter key on confirm that
| repeated several times. I ended up with 200k in my account
| that came as a negative 180k balance to my saving account
| sent to my checking account. (values rounded for story
| simplicity)
|
| This meant that not only did their online portal not have
| controls to make sure that internal money transfers are
| confirmed before processing. They also had no controls to
| make sure that when an internal transfer is done that the
| money is there to be transferred.
|
| I waited a couple days expecting a call. Nope. At the end
| of week I call them. Get transferred around a bunch. The
| whole time everyone is saying everything is good nothing is
| wrong, there is no problem. I 100% certain every single one
| of them was lying through their teeth while they go the FBI
| on the phone because of the automatic assumption that the
| computers are right and I am automatically a criminal.
| There is no way all those people saw a -200k balance in a
| saving account and thought it was fine. (BTW that means it
| is standard bank policy to lie to a customer when something
| is wrong)
|
| So what happened. They immediately zeroed my accounts and
| froze all my bank activity for three months while they
| "investigated" Then they only returned 15K of the original
| 20k that I moved so they effectively stole 5k from me. I
| spend another 6 months trying to get that money back and
| did not succeed.
|
| I would have been screwed if I had not been born poor and
| was familiar with the banking system corruption. I have my
| income,spending,saving and investing split among different
| banks and banks accounts in case of something like this
| ever happened.
|
| The bank initials are WF.
| jonjacky wrote:
| Here is a posting[1] from a thread about a somewhat
| related experience with a bank more than 30 years ago,
| from the RISKS Digest, 1988:
|
| "Someone, in this case a customer though in other cases
| it has often been an employee, complains that something
| is badly amiss with some system, receives no redress, and
| finds that their reports are ignored. ..."
|
| 1. http://catless.ncl.ac.uk/Risks/7.09.html#subj1
| ploxiln wrote:
| +1 to the advice to have different accounts with
| different banks. I've done this for a bit over 10 years
| now: checking account at one bank, savings at a different
| bank, credit card at yet another different bank, yet
| another credit card at yet another different bank, joint
| checking account with spouse at yet another different
| bank ...
|
| My spouse did run into some nonsense some years ago due
| to her checking and savings accounts being at the same
| bank and automatically linked and mixed-up. Banks are
| always offering you super convenient awesome features
| with "protection" and "connection" and just PLEASE NO -
| that's how one mistake, that was that bank's in the first
| place, takes down all your available funds. You'll
| probably get most back, eventually, but do you really
| want to live with that for some weeks/months ...
| RetroTechie wrote:
| _" Which incidentally is not entirely unlike the situation
| in this perversion of justice."_
|
| So true. I went through the Wikipedia article, and couldn't
| escape this feeling:
|
| 1) Some errors were made (faulty IT system).
|
| 2) Handling of said errors expands _enormously_ into
| various reports, IT vendor denying their system is faulty,
| legal proceedings, criminal convictions "tainted" &
| overturned, etc.
|
| To the point where original errors almost _seem_ irrelevant
| within the bigger frame of subsequent events.
|
| Imho _all_ the mistakes were made in the processes
| concerning 2), and 1) is a tiny side-show at most. IT
| systems can fail, period. What matters is how that 's
| handled.
| PartiallyTyped wrote:
| > How can it possibly all work?
|
| What must not to fail has very rigorous standards, e.g. Nasa,
| aerospace, and medical devices.
|
| Everything is built like an onion of failsafes.
| tpmoney wrote:
| Even the stuff that's built to rigorous standards is mostly
| built with a serials of failsafes because the same issues
| of being able to trust the thing apply. But it's fine
| because like you note, failsafes limit the effect of any
| individual failure.
|
| Beyond that too, I think it's just important for people to
| realize that most things just don't matter that much.
| Everything we do eventually has to reach human scale and at
| human scale, there's so much room for slop and correction
| that most failures don't matter. Missing a paycheck because
| the payroll computer dropped your row is awful, but society
| can be built around allowing people to recover from that.
| Late payment forgiveness, forbearance, procedures around
| cutting a new check early.
|
| I think the biggest danger about the "presumption that
| computers are working correctly" is in it's ability to
| erode our understand of the fact that "to err is human".
| It's too easy if you assume the machines are perfect to
| assume that any errors are therefore malicious, and so it's
| easy to go from a system where you have "oops my tax
| software messed up, no big deal I'll file an amended
| return" to "the only way for a mistake like this to happen
| is to intentionally put in bad data, so you must be guilty
| of tax fraud". Or even worse "the computer won't let me do
| it". I understand you want computers to have check in place
| that prevent the overriding of procedure and you don't want
| to give your low level employees the ability to override
| procedure at will. I also think that EVERY system should be
| built such that it is possible with a certain combination
| of people on site at the point of contact to override ANY
| check in the system to make the state of the system match
| what it should be. My specific example in this case was a
| problem with a prescription at the pharmacy. The script had
| been partially filled because they had stocking problems,
| but instead of going into the "partial" state, it was
| closed as filled completely. The tech knew what had
| happened and knew we still were owed the rest of the
| script. The pharmacist knew what had happened and agreed we
| were owed the rest of the script. The store manager agreed.
| And yet no one could change the state of the system. The
| only option was to get a new script from the doctor for a
| new fill entirely. That's the danger of assuming everything
| is working correctly all the time.
| envsubst wrote:
| Having worked in the areas you are describing, it's still
| duct tape and rubber bands, they just are better prepared
| for redundancy and error recovery.
| hgomersall wrote:
| With medical devices you have to act on the assumption that
| the software will fail and that mitigations should be in
| place to deal with the consequences of failure.
| vladms wrote:
| Most people have a tendency to take the easy path and think
| stuff is perfect rather than accept everything works
| "approximately", although we see again and again proofs of the
| contrary.
|
| But as an engineer I find this natural and efficient! You
| should prioritize and some bugs are less important than others.
| If anything more effort should be put in education, health and
| fundamental research rather than fixing all bugs in all
| software.
| genr8 wrote:
| a 1984 law
| di4na wrote:
| Nope, a 1999 one. Sadly.
| suzzer99 wrote:
| This is the first I've heard of this scandal. I'm curious what
| the major problems were with the software that caused the
| discrepancies. Dropped connections? Wikipedia didn't go into much
| detail with that.
|
| Ok I found this: https://www.benthamsgaze.org/2021/07/15/what-
| went-wrong-with...
| di4na wrote:
| It is one of the biggest scandal of our time related to
| software. I ... want to recommend learning more about it but
| only do so with a strong stomach.
|
| Numerous people that rabbithole into this case end up running
| to a nearby bucket. It is... not pretty.
| cf141q5325 wrote:
| Its a major problem with many of such issues. Some stuff is
| just really horrible, often even without any clear approach
| of how to fix it. It then gets even more problematic if
| people react to this by retreating into easy stories that act
| as mental fig leaves. Especially once these leads to clearly
| dysfunctional attempts to solve the issue, making the
| situation even worse for those effected.
| di4na wrote:
| I mean sure, but it is also ok to delegate handling these
| to people with the proper psychological support.
| cf141q5325 wrote:
| Determining who to trust with this (on what topics) seems
| to be an incredibly hard problem with no obvious
| solution. And unfortunately a lot of people either seem
| to delegate to those that make nice sounding promises or
| ignore the issues all together. Even the statement you
| just made could always turn out to be a "noble lie" aimed
| preserving personal sanity at horrible cost for others.
| After all, having psychological support, or being able to
| deal with this, says nothing about what kind of solution
| they would produce. In all likelihood those people to
| delegate to will turn out to be in large parts depressed,
| detached or psychopaths. So i dont see a way around
| verification.
|
| Having agency in a society ( through democracy, markets,
| shaping narratives in discussions and many other ways)
| might just require adults who deal with the horrible
| aspects society produces in a responsible manner. Which
| includes not designing solutions for a false peace of
| mind. After all, you yourself (as in every individual)
| are the one responsible for the consequences you produce.
|
| Personally i hope this might be a matter of outer
| alignment, as in a problem of not agreeing on shared
| goals that you can apply to problem solving in a
| coordinated manner. Which has to include the ability to
| verify process and outcome (which unfortunately in itself
| are highly safety critical) and the unwillingness to
| accept false but pleasant conclusions with high costs for
| others.
| zelos wrote:
| I agree, I find it hard to read about because it makes me so
| angry. Four people driven to suicide, hundreds of lives
| ruined. It took them so long to hold all the inquiries and
| decide on compensation that dozens have died before being
| acquitted or receiving compensation.
| di4na wrote:
| And that is only the one we know of.
|
| There are thousands of cases that are hidden in the bowels
| of the post office court system
|
| Oh and the compensation are uh... let say hard to get the
| money for.
| andrewaylett wrote:
| And I've yet to hear mention of the people who weren't
| _prosecuted_ because they paid the excess out of their
| own pocket. From the stories I have read, it seems that
| many who were prosecuted had tried that but ran out of
| money. Many more must have successfully dealt with the
| "shortfall" and are surely due their own restitution now?
|
| Every transaction the system touched should be inspected,
| because I'm sure there were plenty of sub-postmasters who
| took the "shortfall" at face value and paid up without
| even thinking about questioning the system.
| lozenge wrote:
| I think the issue wasn't the bug but the lack of care? Pretty
| much any large system develops bugs during the development but
| they get noticed and fixed. These bugs got noticed and not
| fixed.
| di4na wrote:
| Worse. They knew of the bugs and prosecuted postmaster for
| the discrepancy instead of accepting them as bugs.
| Muromec wrote:
| Tldr, distributed ledger provided by a third party with
| unauditable access went out of sync and branch managers were
| thrown under the bus for imaginary fraud because computer says
| so.
| taneq wrote:
| Having seen this in action, it's fascinating how fast humans flip
| from "this weird new computer is automatically wrong" to "
| _computer_ is _wrong_? Haha nice try."
| quickthrower2 wrote:
| The title is maybe the opposite of clickbait. The article talks
| about the Post Office scandal. That is a big topic to take on!
| civilized wrote:
| Shouldn't the defense attorney consider the possibility that
| computers are not working correctly? Is there... not a defense
| attorney?
| topspin wrote:
| "Code is law." -Lawrence Lessig
|
| The UK apparently made that explicit. Doubtless others have as
| well: if we could forego the flag waving and whataboutery, That
| Would Be Great (tm).
| User23 wrote:
| Lawrence Lessig's greatest claim to fame is handing Disney and
| friends unlimited copyright before the Supreme Court.
|
| I'd hold up Judge Alsup if I wanted to give an example of a
| tech saavy legal practitioner.
| morelisp wrote:
| While it's increasingly clear that Lessig's legacy will be as
| (to borrow a phrase) the internet's own failson, he is far
| more famous today for CC than Eldred v. Ashcroft.
| kazinator wrote:
| Every prosecutor in possession computer evidence of a crime
| having to prove the software correct is a nonstarter.
|
| Assuming it's right, and proving it buggy, is the right
| principle.
| zubairq wrote:
| I'm quite interested to learn and understand the line of
| reasoning for this comment. Care to elaborate?
| jpc0 wrote:
| If bugs were easy to find and solve there wouldn't be an
| entire highly paid industry to do it.
|
| Add to that malicious actors actively trying to work against
| systems (bank/post office/medical system etc) and it becomes
| pretty hard to prove that both the software and the data
| referenced by the software is in fact correct.
|
| Now think about the last time a critical piece of data was
| recorded on paper with a pen/analogue typewriter and both
| parties got a physical copy of it.
|
| Last time you made a cash withdrawal did you have to go to a
| teller at the bank, fill in paperwork, have two copies of
| that paperwork exist and then that paperwork was digitised or
| was the entire process digital beginning to end?
|
| Think about how you, if you were an expert witness, would
| prove that an transaction made on a banking website was not a
| computer fault.
|
| Do you show a code audit which meets some standards?
|
| Do you have the bank's CISO explain all of the security
| measures?
|
| INAL but I'm reasonably sure the defendant only has to prove
| that a system with the same security and audit trail has in
| fact been faulty and your evidence goes out the door or at
| least the validity of it decreases.
| zubairq wrote:
| Thanks for the detailed answer. I think however that the
| real reason that there is a big industry to find and solve
| bugs is because of decision made when building and
| designing the software in the first place. For example if
| software was built in a robust way with care taken to
| refactor code along the way then maintenance and bug fixing
| would probably be a lot easier and may not need an entire
| industry to support it
| jpc0 wrote:
| I would hope so but I don't think that is the case, there
| is a portion of bugs, including security vulnerabilities
| that are actually hardware related but has a workaround
| in software.
|
| That and no matter how hard you try there will always be
| logic errors in programming that will need to be fixed.
| zubairq wrote:
| Yes that is actually a very good point, I stand corrected
| kazinator wrote:
| If programs were easy to prove correct, there wouldn't be a
| testing and debugging industry.
|
| Bugs don't have to be root caused and solved in order to
| call into question the use of some computer situation as
| circumstantial evidence.
|
| The claimed buggy, externally-visible behavior just has to
| be reproduced. If there are third-party reports of it, that
| would help.
| dboreham wrote:
| > If programs were easy to prove correct
|
| This is just another turtle layer because such proof is
| only vs a "specification", which turns out to also be a
| program, written in another language.
| jpc0 wrote:
| I would argue programs are impossible to prove correct, I
| would speculate it is related to the halting problem.
|
| That doesn't mean that in general the subsections of it
| related to any specific court case isn't correct.
|
| Unfortunately disproving something is much much harder
| than proving something.
|
| Proving that there was a bug in a program is hard but
| proving that there isn't is nearly impossible and that is
| likely why the law is the way it is.
| reuben364 wrote:
| Taking this to absurdity you can create a new obfuscated
| program for each case that presumes guilt based on whatever
| evidence is could be in principle be decided with a computer,
| maximizing the burden on the defendants.
|
| Maybe another principle would be that a program is assumed to
| function as proven in the past. The further a given program is
| shown prior to be correct, the more the burden of proof shifts
| to the defendant to show otherwise.
| checkyoursudo wrote:
| This is contrary to the principle of presumed innocence. This
| is presumed guilt. The burden should always be on the
| prosecution to prove guilt, and if that requires the
| prosecution to prove the correctness of hardware, models,
| software, or data, then that is what should be required. And if
| the prosecution cannot prove the legitimacy of evidence, then
| than evidence should be disallowed.
|
| The standard does not have to be beyond all doubt. The standard
| is fine at beyond reasonable doubt. If the prosecution can
| replicate consistent evidence for and against the defendant
| using the same computer systems with simulated data, then that
| is probably enough to clear reasonable doubt.
|
| But the burden of proof should absolutely always be on the
| party wishing to use "evidence" to take someone else's
| property, freedom, or life.
|
| Assuming it's right is assuming guilt. That is the wrong
| principle.
| kazinator wrote:
| > _This is contrary to the principle of presumed innocence._
|
| No it isn't. It just means that some situation involving
| information from computers is "circumstantial evidence".
|
| If programs have to be proved correct, that basically means
| data from computers could never be introduced as
| circumstantial evidence.
|
| For instance, guns are normally assumed not to go off unless
| someone pulls the trigger. If a member of the Crips guns down
| a Bloods member across the street, he could claim that his
| gun accidentally went off the moment he was inadvertently
| pointing a gun in the direction of the enemy gang member. If
| he could prove that true, then it could be ruled an
| accidental death. If the defense lawyer does not make that
| argument, it will be assumed that there was nothing wrong
| with the gun, and thus someone pulled the trigger.
| jonjacky wrote:
| This sort of thing has been going on since ... well, forever. I
| was reminded of this post [1] by Phil Agre in the RISKS Digest
| from 1988:
|
| "I've noticed that responses to attempts to complain about
| uncorrected technical problems tend to take certain recurring
| forms ...
|
| What has happened? Someone, in this case a customer though in
| other cases it has often been an employee, complains that
| something is badly amiss with some system, receives no redress,
| and finds that their reports are ignored. So they make noise
| about it. The response?
|
| 1. Misplaced accusations of uncooperativeness. ...
|
| 2. Condescending lectures about how `incredibly complicated' the
| systems in question are ... "
|
| etc. See the post itself for much more:
|
| 1. http://catless.ncl.ac.uk/Risks/7.09.html#subj1
___________________________________________________________________
(page generated 2023-10-01 23:01 UTC)