[HN Gopher] Proposed amendment to legal presumption about the re...
___________________________________________________________________
Proposed amendment to legal presumption about the reliability of
computers
Author : chrisjj
Score : 164 points
Date : 2024-12-02 10:50 UTC (12 hours ago)
(HTM) web link (www.postofficescandal.uk)
(TXT) w3m dump (www.postofficescandal.uk)
| Macha wrote:
| For those not familiar with recent UK scandals, there has been
| one ongoing for a few years where there was a bug in a system
| designed by Fujitsu that was calculating the finances for post
| offices. This resulted in the system quoting wrong balances for
| post offices, and when that didn't match the actual cash in hand,
| resulted in prosecutions of the postmasters responsible for those
| offices which in turn led to convictions, dismissals, fines,
| imprisonments, and suicides over a 15 year period as the computer
| system was presumed correct.
|
| Anyway, it was later proven the computer system was incorrect but
| the government there dragged their heels on exoneration and
| compensation.
|
| https://en.wikipedia.org/wiki/British_Post_Office_scandal
| Latty wrote:
| A strong recommendation for the excellent "Mr Bates vs The Post
| Office" to anyone who hasn't followed it, a short (four part)
| drama about it which is really worth a watch, and a good way to
| get a rough feel for what went on, and also managed to finally
| catalyse a real response by the government.
| n4r9 wrote:
| Alan Bates sounds like the epitome of the British "stiff
| upper lip". Stubbornly standing against overwhelming
| adversity for years, for the good of his fellow society and
| his own integrity.
| bobnamob wrote:
| > where there was a bug in a system designed by Fujitsu
|
| To clarify, there was not _one_ bug, but hundreds.
|
| https://en.wikipedia.org/wiki/British_Post_Office_scandal#Pr...
|
| From my read of the information publicly available, there was a
| fundamental lack of distributed/transactional system
| understanding from the developers at Fujitsu.
|
| This[1] article recently posted here[2] reminded me of the Post
| Office Scandal.
|
| The lack of respect for established CS theory (transactions and
| distributed systems) and established accounting practices
| (double entry book keeping or even the idea of a ledger) is
| mind boggling.
|
| [1] https://news.alvaroduran.com/p/engineers-do-not-get-to-
| make-...
|
| [2] https://news.ycombinator.com/item?id=42269227
| movpasd wrote:
| It seems to me that it's a fundamental failure of software
| engineering culture.
|
| The number one rule for engineering domain applications is to
| understand the domain.
|
| I would blame the perennial neophilia and lack of (or
| inadequate adoption of/respect for) standardized texts in the
| industry. Though, to be fair, a lot of this does come down to
| the rapid changes in the technology.
|
| Iterative development is necessary for software, of course,
| but this should be understood as a necessity due to the
| medium, not as an excuse for skipping research and design. A
| lot of these domains (especially something as critical as
| accounting) should be solved problems.
| arethuza wrote:
| From what I can see "software engineering" for the most
| part isn't really "engineering" in the sense that a PE or
| CEng would recognise?
| pjmlp wrote:
| It is partially on the countries where one cannot just
| slap engineering on their job title as they feel like.
|
| Where signing off contracts does have some implications
| beyond "it works on my computer".
| BartjeD wrote:
| Engineering is a professional activity, a legal title is
| just a moat, it doesn't mean that it is Engineering, just
| a protected profession of some sort.
|
| In al seriousness, Engineering is about verifying systems
| to make sure their lifespans and failure modes are known,
| up front.
|
| This has a legal dimension and a practical one. Legally
| you can make people liable for unreliable systems. But
| you can also be liable for failure to maintain properly,
| or failure to warn about impending calamity. Because it's
| all documented and verified.
|
| Practical you can live worry free in earthquake and
| flooding proof buildings, trusting in the diligence of
| Engineers, and maintenance workers, because they and
| others have liability imposed on them.
|
| For software this is only the case in a few sectors. For
| buildings in all cases. Not comparable.
| pjmlp wrote:
| When society depends on software is more than comparable.
|
| Any life can be absurdly destroyed via malware, security
| exploits, accounting gone wrong, a database deleted in
| production,....
|
| People also don't put up with faulty products, why should
| computing be an exception, shitty ship now fix later
| culture?
|
| And above all, calling oneself "engineer" out of a
| bootcamp, has nothing to do with Engineering.
| cedws wrote:
| This is not a story of a failure in software engineering.
| Shit happens, no system is 100% reliable. The failure lies
| in how management handled it (pretend there's no issue,
| don't launch an internal investigation, blame others).
|
| No doubt management also oversaw the development of the
| system and rushed it to production.
|
| A healthy culture should accept failure as inevitable and
| learn from it when it occurs. It should also listen to the
| people who know best: the engineers who built the thing.
| You know, like the aerospace industry.
|
| The damage that morons in suits do in pursuit of their
| bonus cannot be overstated.
| nuc1e0n wrote:
| There's also the issue where public software engineering
| related projects are too broad in scope and poorly
| defined. Rather than making small projects that are
| useful and then expanding upon them the trend is to
| specify massive nationwide databases and then just throw
| money at them, often using 'Big bang' deployments. These
| are so called because you deploy them and then there's a
| large explosion ;)
| bobnamob wrote:
| I'm not sure I entirely agree in this circumstance.
|
| Normally I'd give the developers the benefit of the
| doubt. But the sheer number of issues, and how
| fundamental some of them ~were~ are[1] leave me little
| room for sympathy.
|
| https://en.wikipedia.org/wiki/British_Post_Office_scandal
| #Pr...
|
| Transaction idempotency is such a basic property for a
| financial system that I struggle to believe that Horizon
| was tested in any meaningful way.
|
| "the engineers who built the thing" (Gareth Jenkins) are
| also under investigation for perjury.
|
| [1] Horizon is still in use, in its buggy state, with
| replacement scheduled for 2030....
| cedws wrote:
| >Normally I'd give the developers the benefit of the
| doubt. But the sheer number of issues, and how
| fundamental some of them ~were~ are[1] leave me little
| room for sympathy.
|
| The developers were just doing their job. It's
| management's responsibility to construct a functioning
| system of checks and balances and understand the
| limitations of their systems, both of which they failed
| to do. If it weren't for their hubris the fundamental
| issues with Horizon could have come to light much
| earlier.
|
| Let's also not forget that the reason executives are
| compensated well is for them to take accountability in
| situations like this.
| danaris wrote:
| It _can_ be more than one thing at a time.
|
| But I would also say that that kind of toxic management
| is absolutely a _part_ of "software engineering
| culture". How many horror stories do people on here have
| of managers who care nothing about the quality of the
| product, only meeting the deadline so they can get their
| bonus?
|
| "Software engineering culture" is way, way more than just
| "how write good code." It includes how we work, how we
| manage/are managed, how we advocate for ourselves, or
| fail to do so, and much more.
|
| It certainly includes the very common resistance to
| unions among programmers, and _assuming_ this was caused
| by management pushing a known-bad product out the door, a
| strong union would have (at least potentially) been able
| to stand up to such demands.
| marcosdumay wrote:
| > The failure lies in how management handled it
|
| The thing literally went into a criminal court as
| evidence, and was "presumed correct" in a way that
| overloaded any technical or reasonable discordance.
|
| The largest failure here was from the judges and lawyers.
| The software failure isn't even relevant.
| pera wrote:
| Software engineering culture is still in a very primitive
| phase: just take a look at the comments of the category
| theory submission currently in the front-page
| https://news.ycombinator.com/item?id=42291141
|
| Try suggesting to use a tool like TLA+ to validate some
| complex design and the most likely scenario is that people
| will laugh at you, even if it's a critical component for
| the business.
|
| Most decisions in the industry are based on weak anecdotes
| and unfounded opinions of underserved "authorities".
| djtango wrote:
| IMO it's not really a lack of respect as it is software
| vendors and managers not really being incentivised to do the
| right thing. Therefore even if the engineers know the right
| thing to do, they are unable to solicit the buy in from
| stakeholders to invest the development into doing the right
| thing because there's no recourse for doing the wrong thing
| but it's more expensive.
|
| And even if you do the right thing, if a competitor comes in
| and offers to do the "job" quicker and cheaper because
| they're doing the wrong thing. You may have the moral
| highground but that doesn't pay your employees' salaries.
|
| What we need is a better framework for punishing bad
| software, in an ideal world without a bunch of red tape and
| reducing the burden on non experts identifying what is or
| isn't bad behaviour.
| exe34 wrote:
| > What we need is a better framework for punishing bad
| software
|
| So like engineering licensing and insurance?
| djtango wrote:
| I'm pretty ignorant on the ins and outs and trade offs
| associated with accreditation but yes that sounds like
| one solution.
|
| It doesn't strike me as unreasonable, software engineers
| are now able to cause billions in damages and/or cause
| loss of life via primary or secondary effects.
|
| Sure, one of software's greatest strengths was that
| anyone could learn it and it'll be sad to lose that but
| that's not really a reason for structurally enabling
| things like the Post Office scandal.
|
| You probably don't need accreditation to serve cat pics
| but things dealing with money and life/death should. But
| IANAL so hopefully someone else can bring better insight
| to this area.
| rswail wrote:
| > it'll be sad to lose that
|
| People will still be able to learn how to program and
| actually program. But if they take money or deal with
| people's private information, then they'll need to comply
| with the standards that will be regulated.
|
| I think we need to start thinking of an individual's
| _persona_ which includes all the information about them
| that is online or in government (non secret) files.
|
| That needs to be considered when designing regarding
| regulations for software that interacts with someone's
| _persona_.
|
| So anyone can set up the equivalent of an art stand in
| the park (serving cat pics), but if they start either
| _selling_ cat pics or _selling_ advertising that relies
| on collecting information about the _personas_ on the
| site, then that needs to be regulated.
| RHSeeger wrote:
| > People will still be able to learn how to program and
| actually program. But if they take money or deal with
| people's private information, then they'll need to comply
| with the standards that will be regulated.
|
| You can even have non-licensed people doing work, and a
| licensed individual signing off on the end result. They
| would need to review the designs and work to make sure
| they agree with how the work was done, because it's their
| license on the line if something goes wrong because of
| bad craftsmanship.
| oersted wrote:
| It doesn't need to be too dramatic. It's already a
| general rule never to write your own cryptographic or
| financial code from scratch, you reach out for
| established libraries or services written by experts.
|
| The law may require to use the ones that are certified if
| you are doing something sensitive. It doesn't need to be
| universal necessarily, but it should apply to public
| tenders for critical infrastructure at least.
| maxwell wrote:
| You want some government mandating your dependencies...?
| HeatrayEnjoyer wrote:
| Yes. Legally mandated supply requirements have been an
| element in every other engineering sector for decades or
| centuries. If you're building an airplane you can't
| purchase from any random aluminum supplier, there's an
| authorized list.
| maxwell wrote:
| That would seem to result in industry consolidation and
| declining innovation.
|
| Software isn't made of physical materials anyway, it's
| speech. How about instead we impose open source
| requirements to enable public verification of critical
| systems?
| gorft wrote:
| Innovation like calculating numbers incorrectly and
| getting people sent to jail? Or maybe just another
| hundred slow, rent-seeking React web apps that'll
| disappear when the series B money dries up, if we're
| lucky. We must have different ideas of what innovation
| is.
| oersted wrote:
| At least when you are building key infrastructure for the
| government, directly or indirectly through third parties.
| I don't think it's unreasonable.
|
| And it is not as absolute as you make it sound. Only
| dependencies for specific critical functions may be
| regulated. And they don't have to literally force a
| whitelist of dependencies on you, just whichever has been
| certified as appropriate for that purpose.
| svieira wrote:
| This is how you get FIPS 140 [1], which for those not in
| the know is a US Federal standard that mandates
| encryption which is _less_ secure the the current state
| of the art and has been for decades. (Yes, there's a new
| version which was approved _5 years ago_ and which is
| still rolling out [2]).
|
| [1]: https://en.wikipedia.org/wiki/FIPS_140
|
| [2]:
| https://csrc.nist.gov/Projects/fips-140-3-transition-
| effort
| maxwell wrote:
| Who maintains the mandated dependencies? Who performs
| certification?
|
| What prevents regulatory capture?
| exe34 wrote:
| how does it work for aircrafts? how does it work for
| anti-money laundering policies?
| lucianbr wrote:
| The way it works for aircraft is that now it's getting
| prohibitively expensive to design new planes. But new
| planes are still needed, so Boeing found a solution, and
| made the 737-MAX which is in theory an upgrade that does
| not require re-certification, but is in fact different
| enough that the differences lead to hundreds of deaths.
|
| Perhaps without the certifications lots more people would
| have died. I'm just an armchair analyst. Just food for
| thought.
| gorft wrote:
| Uh, the MAX was a disaster because Boeing is run by MBAs
| instead of engineers. Regulation is why air travel is
| remarkably safe. Those rules are written in blood,
| something everyone here conveniently seems to forget.
| gorft wrote:
| Famously, civil engineering has never solved this
| problem, either, leading to the sad state of affairs in
| which we find ourselves today, wherein no bridges have
| been built anywhere.
| danaris wrote:
| This isn't (or doesn't have to be) about laws specifying
| how _all software_ has to be made.
|
| What they need to specify is the standards for software
| that _certain types of organizations_ can use. Like
| government agencies, government contractors, medical
| organizations, construction and engineering firms, and
| probably some other kinds of large private businesses,
| depending on their industry.
|
| Basically, if the software _your organization uses_ can
| cause the level of destruction that Horizon did, it needs
| to have specific certifications, or you can 't use it.
|
| In order for such software to _be_ certified, it needs to
| meet certain clearly-defined standards of quality,
| potentially including having all the technical leads of
| some level (or just all the developers, depending on
| various factors) be licensed, and have their licenses on
| the line of something like this scandal occurs.
|
| It's not a panacea, and it would definitely be an
| absolute _bear_ to get the terms of all of it defined
| both clearly and in a way that is likely to actually
| produce a quality product, but IMO it is likely to be
| worth it in the long haul.
| exe34 wrote:
| think of it as accreditation profiles. the highest is for
| life threatening cases like flight/medical, the lowest
| might be government expenses.
| danaris wrote:
| While I personally _love_ that idea, I suspect it 's too
| complex to fly in actual attempts to get something like
| this in place.
| exe34 wrote:
| the same could be said for aerospatial regulations, and
| we still manage to do them. civil constructions are
| mindbogglingly complicated and we still manage.
|
| all it takes is enough people to die, and/or for rich
| people to lose enough money and it'll become the rule.
| maxwell wrote:
| Transparency and competition seem to work better than
| certifications and credentials.
|
| Just mandate open source if using public money.
| gorft wrote:
| How's it working for those postal workers who committed
| suicide?
| agentultra wrote:
| I think it needs to be on the individual-liability level.
| Blessing implementations from corporations is an
| environment for monopolies to grow and corruption to set
| in.
| raxxorraxor wrote:
| The Post Office scandal at its core was a political/legal
| problem, not an engineering one.
|
| Software needs to be transparent, a human expert worker
| needs to be able to verify the result a software
| generates within reasonable time.
|
| There is no expertise for software that you could
| delegate to aside from an entity using some form of
| integration test.
| tremon wrote:
| > Software needs to be transparent, a human expert worker
| needs to be able to verify the result a software
| generates within reasonable time.
|
| This is becoming more and more important as the AI
| industry is pushing non-deterministic computing further
| and further. There will be court cases where the full
| automated decision chain will be called into question,
| and a default assumption of "well, we don't have any logs
| so we should assume the software did the right thing"
| terrifies me.
| Terr_ wrote:
| The magic word that always pops up to me is "liability."
|
| Even speaking from an IC/non-managerial place, far too
| many technical problems in businesses are really just
| symptom-alleviation (or worse, performative
| look-I'm-doing-something theater) for a deeper problem
| which involves misaligned incentives for humans.
| oersted wrote:
| Sure, it doesn't need to up-end the whole field, but it
| is rather obvious that critical infrastructure should
| require significant compliance, particularly if it's
| being purchased in a public tender process where the
| cheapest supplier that fits the criteria is hired.
| Certifications like SoC 2 are already a big thing.
| yobbo wrote:
| > managers not really being incentivised to do the right
| thing
|
| It's more likely that managers are just incompetent. Not
| being able to distinguish between infrastructure and
| "decoration" means all their decisions are at best
| superficially motivated.
|
| The punishment we have for this is unemployability and
| bankruptcy. It will happen by itself if you let it.
|
| A framework for "punishing bad software" sounds to me to be
| hiding the ambition to "protect bad managers from
| consequences of their decisions".
| RHSeeger wrote:
| > The punishment we have for this is unemployability and
| bankruptcy. It will happen by itself if you let it.
|
| In my career, I have never seen a manager see
| consequences for making the decision that values "money
| now" vs "avoiding things going wrong later". I do not
| believe that it will happen by itself.
| yobbo wrote:
| Well yes, agree this is how it looks now. But my
| perspective is that there has to be layers of mechanisms
| protecting them, rather than the natural order. (These
| mechanisms are tricky to pin-point but seem to be on the
| themes of mutual loyalty, avoiding embarrassment, keeping
| face, and so on.)
|
| Imposing some formal framework is more likely to insulate
| managers even further while making software developers
| accept legal responsibility beyond their pay.
| tacticus wrote:
| > The punishment we have for this is unemployability and
| bankruptcy. It will happen by itself if you let it.
|
| yeah, nah.
|
| "No one ever got fired for buying IBM"
|
| after decades of incompetence the multinational
| contracting firms are still going fine. IBM still runs
| fine and they're focus is now even more in the insulting
| side of the business.
| belter wrote:
| The many bugs are still there. Fujitsu should be renamed
| Bugitsu....
|
| "Majority of Subpostmasters still getting unexplained Horizon
| discrepancies" -
| https://www.postofficescandal.uk/post/majority-of-
| subpostmas...
| pdpi wrote:
| There were "bugs", yes. There were also fundamental design
| flaws.
|
| One of the things the Post Office did was sell travel money,
| but the whole system was never really designed for ForEx
| operations, so it didn't keep track of exchange rates over
| time. The result is that reconciliation used the exchange rate
| at time of reconciliation instead of at the time of trade. So,
| if the foreign currency had gone up in value, it would show up
| as GBP missing.
| rswail wrote:
| So they didn't record the rate in the transaction?!?
|
| Wow, that's... dumb.
| pdpi wrote:
| IIRC they did record the date, yes. What they _didn't_
| store was historical exchange rates.
| db48x wrote:
| A lot of the problems with Horizon are incredibly dumb.
| Mistakes that anyone with two brain cells to rub together
| would recognize.
| philipwhiuk wrote:
| As a developer working on trading platforms, this is
| horrific.
|
| Given the Post Office had a reputation for really good
| exchange rates, this one design flaw might be responsible for
| a significant portion of the problem.
| ksec wrote:
| I am already consider cynical by many, but reading about the
| scandal constantly reminds me how naive I am. Just how low can
| government and politicians get.
| exe34 wrote:
| > Just how low can government and politicians get.
|
| As low as it takes for a profit!
| bowsamic wrote:
| I actively avoided the news and outrage about this for my own
| sanity. It was just too shocking. The worst part is that I'd
| heard about it and got mad about it many years ago, and it
| wasn't until a TV documentary came out recently that anything
| happened.
| bobnamob wrote:
| Echoing this, I've been passively following the inquiry but
| haven't watched the miniseries.
|
| I think it would leave me furious for weeks and push me
| into a depression rut.
|
| I'm trying to take the whole debacle as long term
| inspiration to be excellent at what I do - it's tough to
| stay positive when so many people involved (Fujitsu, PO
| upper management, the original prosecution) seem entirely
| morally bereft, with little chance of consequence.
| xenadu02 wrote:
| I guess some people never learn from trying to cover their
| mistakes as children. I've explained this to my kids: often
| the initial mistake is minor and warrants nothing more than
| "please don't do that again". It is the lying and hiding that
| blow it up into a big issue. The odds of being found out are
| very high. There is little reward and lots of risk.
|
| When you have such a widespread and ongoing problem it
| becomes clear that such a large proportion of postmasters
| can't be criminals. Computer bugs are well known. Eventually
| it is going to come out that the computer is wrong. Why
| double-down? The earlier you admit a mistake and apologize
| the lower the impact and the less anyone cares. They had a
| built-in scapegoat that everyone understands and accepts: the
| vendor's software had bugs! We have daily meetings to yell at
| them to fix the bugs we promise and we will fix the problems
| ASAP.
|
| Or do the sneaky thing and fix the issues, stop prosecuting
| postmasters, and ignore the ones you prosecuted by mistake.
| Cynical, cruel, and immoral... but contains the damage.
|
| Instead the UK Postoffice seems to have just let the problems
| continue while simultaneously allowing prosecutions to go
| ahead knowing they were faulty. Literally the worst of all
| worlds: ongoing accumulation of liability, now with provable
| malice!
| lucianbr wrote:
| Were there actual bad consequences for the upper management
| and politicians involved? I think the risk assessment you
| teach your children does not apply to british politicians.
|
| I heard some convictions of innocent people have been
| overturned, or are in the process of. But new convictions
| of the actually guilty... no news about any such thing.
| Maybe I'm just uninformed.
| ta1243 wrote:
| If the computer had said one postmaster had been wrong, that's
| one thing.
|
| The real scandal here is that there were hundreds, and those at
| the top knew this, but instead doubled down.
| graemep wrote:
| If it had only sent one person to jail we would never have
| know about it.
|
| I am pretty sure more dependable systems send the odd person
| to jail - there have been many cases where someone was lucky
| not to be convicted.
| arethuza wrote:
| The bit that I find incredible is that each one of the
| victims was told that it was just them and it hadn't happened
| to anyone else....
| matteason wrote:
| Also, mind-blowingly (to me at least) - Horizon is still in
| use, Fujitsu are still being given hundreds of millions of
| pounds to support it, and its replacement isn't due to go live
| until 2030
|
| https://www.computerweekly.com/news/366587174/Fujitsu-set-fo...
| jonplackett wrote:
| Dragging heels is a bit of an understatement.
|
| Both the gov and Fujitsu and the post office absolutely knew it
| was a bug and intentionally hid the fact while the post masters
| lives were ruined.
|
| However bad the bug was, the cover up was _much_ worse.
| ndsipa_pomu wrote:
| To be fair, the government were lied to by the Post Office
| people about whether or not there were bugs. They also lied
| about whether Fujitsu had the ability to amend the ledgers
| even though Fujitsu had a team working to correct some of the
| inaccuracies.
| thehappyfellow wrote:
| Buggy software is important aspect of the story but to me it's
| not the main one.
|
| The main one is that the post office management/officials at
| some point became aware of the bugs and that they were ruining
| lives of innocent people _and they knowingly kept lying to save
| their asses_.
|
| The inquiry page has all of this and more:
| https://www.postofficehorizoninquiry.org.uk/about-inquiry
|
| I've read through it, it's long but it's a good (while
| terrifying) read.
| thehappyfellow wrote:
| I'm sorry I don't have a summary but the judgment is also
| enlightening: https://www.judiciary.uk/wp-
| content/uploads/2019/12/bates-v-...
|
| And wiki has rest it the judgments: https://en.m.wikipedia.or
| g/wiki/Bates_%26_Others_v_Post_Offi...
|
| I have a lot of respect for the judge after reading this.
| Here's a quote describing Post Office evidence:
| "bare assertions and denials that ignore what has actually
| occurred... [amounting] to the 21st century equivalent of
| maintaining that the earth is flat".
| nonrandomstring wrote:
| > the computer system was incorrect
|
| There's a little more to it. Most of the comments here are
| focusing on "correctness". And yes, the amendments to section
| 69 do something towards tempering its ridiculous and dangerous
| "presumptions".
|
| But the (UK Post-Office) story is that the Horizon system had
| back-doors in it. Fujitsu denied this. "Corrections" were made
| to systems without operator knowledge - to fix actual errors
| caused by a terrible database sync script full of race-hazards,
| faulty locks and duplicated state.
|
| The cover-up began life as engineers trying to hide up
| technical mistakes. It escalated to senior executives trying to
| cover up financial and political mistakes. It ended with the
| Crown colluding in covering up judicial mistakes. It is an
| exemplar of hubris, pride and egotsim resting on a refusal to
| give up a religious belief in technology. Were it not for the
| courage of a few (including judges and MPs) they would have
| gotten away with it (if it weren't for those meddling kids)
|
| The case stands as an important landmark that you cannot "hide
| behind" technology as a means for abuse and injustice.
|
| The amendments are welcome but insufficient. They open up a
| good opportunity for cybersecurity people to work with lawyers
| now.
|
| There are two outstanding problems:
|
| Proprietary code. If you cannot examine the system then the
| right to challenge it is meaningless. This requires changes to
| investigatory powers/discovery if anyone wants to use
| "technical correctness" as a base for argument.
|
| Malicious function. While the discussion revolves around
| correctness it is incomplete. Many systems (perhaps not the
| Horizon system) are not faulty, they work perfectly well to
| deceive, manipulate and swindle.
|
| I'd still push for a complete reversal of presumption [0].
| Where software is part of a legal dispute it should "take the
| stand" as its own witness, in that formal proofs of correctness
| (a very VERY high bar in software engineering) need to be
| brought in front of the court. Otherwise the reasonable
| presumption is that an error or hidden malicious coding "cannot
| be ruled out".
|
| [0] https://cybershow.uk/episodes.php?id=23
| https://cybershow.uk/episodes.php?id=24
| EasyMark wrote:
| It doesn't make sense to change an entire system over once
| incident. That smells of overreaction to me. It was an awful
| situation caused by humans, and the humans involved should
| probably be punished, that doesn't mean one should break a
| system that generally works very well and promotes self
| regulation and continuous improvement rather than shackling it
| with regulations.
| switch007 wrote:
| Also for those not familiar: the government owns the Post
| Office, despite what many try to suggest to the contrary
|
| Yes there is operational separation but it's not like they're
| wholly unrelated and the government totally guilt free
| greatgib wrote:
| That will be wonderful with AI: Sir, I have this
| conversation with chatgpt where the assistant tells me that you
| are guilty. Based on the fact that chatgpt is able to correctly
| count raspberries we can now legally consider that it is reliable
| and so that you are guilty! Game over.
| Etheryte wrote:
| Can count raspberries, but not how many times the letter "r"
| appears in raspberries.
| lionkor wrote:
| It can once you point it out in the training data, hence we
| cannot know if it understands anything unless we can
| __prove__ that it wasn't in the training set (which we can't)
| zarzavat wrote:
| "Pray, Mr Babbage, if I ask an LLM a question that isn't in
| its training set, will it give the right answer?"
| Terr_ wrote:
| There's a little ambiguity in the original quote which I
| think requires a lot more attention with LLMs in the mix:
| Whether the "rightness" of an answer also depends on the
| correctness of the process used to reach it.
|
| I'd like to think that Babbage would have had the
| stricter interpretation.
| int_19h wrote:
| There are infinitely many combinations of things. It's not
| hard to come up with a bunch such that the probability of
| any one of them being in the training set is infinitesimal.
| Terr_ wrote:
| It will once it's trained on all the posts people made
| talking about the raspberry problem, and then the illusion of
| progress will be renewed. :p
| hmottestad wrote:
| Reminds me of the stories going around of teachers asking
| ChatGPT if a student's essay was written by ChatGPT or not.
| Devasta wrote:
| If there is one thing that should be crystal clear to everyone
| its that for some classes of work software should be like real
| engineering, requiring signoff by the senior engineer with
| personal liability and fines and jail for negligence.
|
| If the engineers dealing with the Citicorp center had dealt with
| the problem like software engineers, the fix would have been to
| update documentation in confluence to not expose the building to
| high winds and that would have been the end of it.
| jack_riminton wrote:
| Whilst I agree in principle, would any Senior Engineer want to
| work on such a system that had their personal liability
| attached? I'd want 10's of millions in annual comp just for the
| risk
| nottorp wrote:
| The real problem is who could afford a system that a
| competent senior engineer would take personal liability
| for...
| sfn42 wrote:
| The real problem is our entire industry is a giant
| clusterfuck. Do a 2-week bootcamp, congratulations now
| you're a software engineer.
|
| Every other discipline has education requirements, codified
| standards for how to do things etc.
| nottorp wrote:
| > codified standards for how to do things
|
| I don't know you but i bet that if you and me were locked
| up in a room together for a month we wouldn't be able to
| 100% agree on "codified standards for how to do things"
| :)
|
| Industry isn't mature enough for that and it's perhaps
| doubtful that it will ever be. See the halting problem.
| bobnamob wrote:
| ye gods, can we come up with a "law" to describe
| appealing to the halting problem?
|
| Just because there are unanswered questions that doesn't
| mean we can't have bare minimum codified standards.
|
| Furthermore, standards aren't invalidated just because
| practitioners disagree with them. Plenty of <insert
| engineer type>s disagree with the standards body of their
| respective field, they still follow the standards out of
| fear of prosecution or simply as a path of least
| resistance and when those standards are found to be
| defective, they (generally) evolve.
| nottorp wrote:
| > we can't have bare minimum codified standards
|
| So, functional, imperative or OOP? :)
|
| > Just because there are unanswered questions
|
| The halting problem is undecidable. Not undecided. I.e.
| it has been solved and the answer is "you can't".
| int_19h wrote:
| We don't need to solve the halting problem. We just need
| to come up with a sensible set of practices that, if
| followed, make the risks small enough to be considered
| acceptable. Then we can point at that list and say, "this
| is what the reasonable expectation of due diligence in
| software engineering is" - and legally enforce that.
| ratorx wrote:
| > 100% agree
|
| I don't think it's necessary to agree completely. You
| could start by codifying a minimal set of things that the
| majority of people agree on (user data sanitisation,
| authentication handling etc) and then build on it over
| time.
|
| The standards could also help codify more meta things,
| like vulnerability policies, reporting and outages. This
| would be helpful to form a dataset which you can use to
| properly codify best practices later.
|
| The main problem is that this increases the bar for doing
| software development, but you can get around this by
| distinguishing serious software industries from others
| (software revenue over a certain size, industries like
| fintech, user data handling etc)
| int_19h wrote:
| The industry will be 80 years old soon. If it's not
| "mature enough", it's only because of the pervading
| culture.
|
| For comparison, electrical engineers started introducing
| things like national standards for plugs by 1915.
| pjmlp wrote:
| Thankfully not every country out there has such liberties
| with "enginnering" titles, but yeah that is a problem.
| int_19h wrote:
| It's not the title that's the problem. It's the part
| where people and go write software that gets deployed at
| scale in an environment where bugs can cause very real
| and significant damage (monetary or otherwise).
| pjmlp wrote:
| The title is part of the problem, because it reveals the
| culture, slapping cool titles without upping oneself to
| what those titles actually mean.
|
| As for the rest, anything that brings computing to level
| of the rest of other professionals, has my signature.
|
| A Software Engineering professor of mine used to say,
| many applications are akin to buying shoes that randomly
| explode when tying shoelaces, whereas a minor defect on
| real shoes gets a full refund.
| arethuza wrote:
| Professionally qualified engineers in other fields seem to
| manage OK?
| nottorp wrote:
| They work with certified components on predictable systems
| though.
|
| You realize that you'd need someone at MS to take liability
| for Windows before you can sign off anything running on
| Windows?
|
| Or someone at Google if you do a web app that only runs on
| Chrome, not to mention other browsers.
| Devasta wrote:
| Well yeah, but the fact that no one takes responsibility
| for anything and just smears layers of crap on top of
| each other is the problem. We have build enormous houses
| of cards on foundations of quicksand and its causing very
| real harm, but no one cares because the only thing
| they'll face consequences for is drops in story points on
| their sprint or whatever else and nothing for failing to
| do things that actually matter.
| Symbiote wrote:
| Some systems must have these certifications.
|
| What OS renders the monitoring screens for air traffic
| control systems, or railways signalling? Those both have
| rigorous software engineering behind them -- railway
| signalling is the original of engineered, safety-critical
| logic systems, starting with mechanical interlocks in
| 1843. (The signalman physically couldn't move certain
| levers into bad configurations.)
| lores wrote:
| And on top of all-certified components, it also requires
| the chief engineer to have veto power over the system. If
| business or client asks for features that are potentially
| insecure (and that's going to be a lot of features), they
| have to accept being told 'no'. I'm not seeing that happen
| easily in the software industry.
| oarsinsync wrote:
| This is chicken-and-egg though, surely. If engineers are
| not personally liable for the garbage they produce, the
| business can replace the engineer with someone more
| amenable to their requirements (either because the
| engineer lacks experience to understand why its bad, or
| because the engineer has fewer scruples, or anywhere in
| between) with relative ease.
|
| If all engineers are held personally liable for their
| code, when a business has faced a documented rejection,
| they'll struggle to hire someone else to take on that
| risk.
| lores wrote:
| You're right, and it would radically change the industry.
| No more wild experiments, but a very measured and
| ponderous rate of change. It's not necessarily a bad
| thing, I'm at the age where I curse aloud when I see yet
| another framework doing nearly all the same things but in
| a different way, but it's certainly a big change.
| arethuza wrote:
| You'd still be allowed "wild experiments" - just not
| where they could harm people.
| lores wrote:
| "Harming people" includes money loss, though. Any hacking
| that results in identity theft would end up at the feet
| of the certifying engineer.
| int_19h wrote:
| Good, then maybe it'll be a forcing function for
| companies to stop collecting personal data for their
| users just because they can, and we can go back to the
| days when the only metadata associated with an online
| account is its salted and hashed password.
| marcosdumay wrote:
| > they have to accept being told 'no'
|
| They can always hire another engineer. They only accept
| it, some times, because they won't find a certified
| engineer that says "yes", and because they doing it
| themselves is a crime.
|
| That equates to a huge amount of government intervention
| on the lives of everybody. And even then, fails way more
| often than expected.
|
| Now, we are talking about a case where criminal justice
| failed to uphold the defendants rights to a fair trial.
| Most probably because of corruption. Do we really want to
| bring that huge amount of government intervention into
| this context?
| delfinom wrote:
| Yes? Plenty of non-software engineers do it daily.
|
| You start off by having a Professional Liability insurance
| policy, your company will generally pay for it unless you are
| a consultant/contractor in which case you bring your own
| policy. Depending on size of operation, your employer may
| even indemnify you in the employment contract specifically
| for even negligence lol.
|
| You then do your job correctly. The laws only go after you
| for liability if you were negligent, i.e. you skipped
| protocol and policies, you skipped best practices and
| couldn't justify it, etc. If you weren't negligent and just
| made an error, great, your insurance covers you. Insurance
| can also cover negligence too depending on policy, lol
|
| https://www.nspe.org/resources/professional-
| liability/liabil...
| jack_riminton wrote:
| Yes and they do non-software engineering
|
| Are we going to have international protocols and policies
| on the best language to use, how to do SQL queries and CSS?
| no
| gpderetta wrote:
| Professional liability insurance and legal insurance as a
| minimum.
| teamonkey wrote:
| I'm fairly sure Fujitsu do have liability insurance in some
| form.
|
| In a situation like this an insurer is strongly motivated
| to prove that the company is not at fault, because it
| doesn't want to pay the bond. The company is also strongly
| motivated, even though insured, to prove that the company
| is not at fault, because it doesn't want to have its future
| insurance rates affected or be sued by the insurer for
| breaches of terms.
|
| Either way, it doesn't help the people affected. Not unless
| they have personal insurance against workplace computer
| system errors, in which case their insurance provider is
| also not motivated to pay out, or to battle a corporation
| as large as Fujitsu unless there's chance of a class-action
| suit.
| pjmlp wrote:
| Yes, that is exactly part of being a Professional Software
| Engineer entails, and why there are universities assessed by
| Enginnering Order, and professional exams.
|
| Lets stop glueing "engineering" to any job title where
| someone knows how to write a bunch of code lines.
| arethuza wrote:
| The important bit is that if you screw up badly enough then
| your professional qualification is removed and you _can 't_
| do that type of work any more.
|
| [NB I am frequently reminded of this point by my wife who
| is a solicitor].
| pjmlp wrote:
| I am aware of that, because I happen to be from a country
| with an Engineering Order.
|
| Another important part is that one might be liable when
| signing contracts as the responsible Engineer in a
| project delivery.
| blackbear_ wrote:
| Why the engineers? Why not putting the liability on the product
| owner, or the project manager?
| UK-AL wrote:
| Because its a technical guarantee. How is a product manager
| going to able to personally approve code he can't even read?
| lores wrote:
| I'd argue there are vanishingly few engineers who
| understand all the consequences of even relatively simple
| code in terms of security and reliability. Every time there
| is a security breach at a bank or FAANG, some very smart
| and experienced engineers with the backing of the business
| didn't understand something. It's downhill from there for
| most everyone else.
| int_19h wrote:
| I'd argue that if there was a demand for software
| correctness - motivated by, say, legislation applying
| some baseline standards on such in sensitive applications
| like money processing - we'd have a lot more smart and
| experienced engineers focusing on security in particular.
|
| The way things are now is because we as an industry have
| decided that "move fast and break things" is acceptable,
| and our culture reflects that. So we need to change the
| culture.
| t43562 wrote:
| Because we cannot have that! Absolutely not. PMs need to be
| able to apply excruciating pressure.
|
| Also we couldn't possibly put the burden on the company that
| makes improbably low bids - no! No we must put the blame on
| the peons where it belongs.
| verisimi wrote:
| > personal liability and fines and jail for negligence
|
| Politicians too. And journalists.
| another-dave wrote:
| I think there should be definitely higher standards around
| things in the industry in generally (especially anything that
| touches health, money etc).
|
| But the real culpability here are the upper management who said
| "we don't believe you" (at the most generous interpretation)
| when the postmasters said that the system was buggy.
|
| > Although many subpostmasters had reported problems with the
| new software, and Fujitsu was aware that Horizon contained
| software bugs as early as 1999, the Post Office insisted that
| Horizon was robust and failed to disclose knowledge of the
| faults in the system during criminal and civil cases
| bigfatkitten wrote:
| Some of the 'engineers' responsible, such as Gareth Jenkins of
| Fujitsu happily left a paper trail showing their perjury and
| attempts to pervert the course of justice throughout the legal
| proceedings.
|
| In a just world, he and his co-conspirators would go to jail
| for what they did, but I don't see it ever happening.
| themk wrote:
| As someone who used to work in electrical engineering but now
| does software, I almost refuse to use the word engineer for
| what I now do.
| MrMcCall wrote:
| Yes, indeed. We are still in the "craft" stage of the process
| of software development. Engineering is altogether something
| else than what we're doing.
|
| Strangely enough, 30ya, my friend getting his EE Masters was
| mostly taking programming courses.
| t43562 wrote:
| I call myself a programmer. It's not engineering or maths or
| anything else. It's managing complexity and I contend that
| it's very low cost complexity or we wouldn't be able to
| afford all the software we use.
| bluecalm wrote:
| I don't think it's fair to blame software engineering. Bugs
| happen, especially where there are no incentives to not have
| bugs. First of all justice system is to blame. Sentencing
| someone on assumption that some complicated software worked
| correctly is criminally stupid and should result on all people
| involved being barred from the profession.
|
| Another thing is that software should be treated as just a tool
| to help to fulfill legal/accounting requirements. If the
| software is wrong then the required documents are going to be
| wrong and that's supposedly auditable. This way there are
| incentives to produce/finance correct software because what is
| going to be judge and relied upon is not software itself but
| what it produces (the documents).
|
| Calls to make software engineers responsible will just result
| in fewer competent people willing to do the work. The justice
| system is incompetent enough already. Can you imagine lawyers
| discussing if your off by 1 bug was "criminal negligence" or
| just a normal mistake that happens? If you going to jail
| depends on what they decide you will just not take the job and
| no one sane/competent is going to take it either. The end
| result is going to be over payed morons writing software and
| then sometimes going to jail for it - not an improvement over
| current state of things.
| Devasta wrote:
| Doctors, Surgeons, Lawyers are able to manage it fine,
| software engineering could do it as well.
|
| If you could show a constellation of unit and integration
| tests, well defined schemas and interfaces, for both your
| code and your dependencies, and a responsible engineering
| culture then the chances of going to jail are going to be
| next to nil.
|
| People recoil at the idea only because they see that very
| very little implemented today would be work that anyone would
| stand by.
| t43562 wrote:
| Good practises still result in bugs. How is the bug handled
| after it is found? That is a management responsibility.
| Devasta wrote:
| "Once the rockets are up, who cares where they come down?
| Thats not my department." says Wernher Von Braun...
| t43562 wrote:
| I think the people who paid Von Braun take 99.99% of the
| responsibility for what happened - they decided to kill
| people and chose one of many possible ways to do it.
|
| They could just as easily have poured their money into
| something else. As for von Braun himself, he got employed
| and looked after for the rest of his life despite being
| part of the decision (I think) to use slave labour to
| build his rockets - and that really was a management
| decision.
| MrMcCall wrote:
| Well, the money guys aren't going to let that kind of outlay
| dip into their country club membership fee allotment, now are
| they?
|
| Your proposal is a brilliant and necessary idea, but we don't
| run the world, my friend. The people that run it only care
| about money, and brilliant ideas cost serious money, not to
| mention committment and patience to follow through.
|
| Changing any entrenched status quo is a real slog, for sure.
| That's why our precious Earth is heating up, daily, to give
| just one example.
| jack_riminton wrote:
| The fact that this misnomer of infallible computer systems was
| ever enshrined in law is pretty damning of the whole UK legal
| system and the relationship between technical people and law.
|
| Every person who has ever programmed a computer or worked in any
| complex system knows they can't be relied upon 100%.
|
| Not least because it seems to go against the core concept of
| "innocent until proven guilty" that the whole legal system is
| meant to rest upon.
| impossiblefork wrote:
| I find it really strange to have presumptions in general.
|
| If we're in a court and there hasn't been a decision yet, we're
| there because we're dealing with some kind of complicated edge
| case where one person has a strong argument for one thing and
| another for another.
|
| If one then decides to bring some problem up-- whether with how
| evidence is being judged or anything else, there can be no
| justification for ignoring him. This is why I like free
| evidence evaluation in Swedish courts and the absence of rigid
| precedent. Every question must then actually be dealt with. We
| do have this kind of idiotic rulemaking in other parts of our
| legal system though, so we're not fully free from it.
| ndsipa_pomu wrote:
| The problem is allocating resources if you want to deal with
| every challenge to evidence provided by a computerised
| system. I think it was initially a problem with people
| challenging speed camera results.
| beardyw wrote:
| I imagine it stems from something like: the clock was correct
| at 1pm and at 3pm but was it correct at 2pm?
| Oarch wrote:
| Maybe it was a presumption that was legally convenient... until
| it wasn't and reality struck?
| jack_riminton wrote:
| Yeah exactly, until the miscarriage of justice was too big to
| ignore
| oarsinsync wrote:
| > The fact that this misnomer of infallible computer systems
| was ever enshrined in law
|
| Is this actually a fact, or a fact taken to it's logical
| conclusion to presume a new "fact"?
|
| The article cites "mechanical systems" as being infallible, and
| reading that language, it reads to me as some archaic
| legislation that never got updated for computer software.
| Instead, precedents got set over time by enterprising lawyers,
| but setting a precedent when it's convenient is not the same
| thing as writing a law.
|
| When I see mechanical systems, I think of something like an
| abacus. I've never used one, but I suspect the abacus itself is
| infallible. It's open, it's transparent, it is easily
| auditable, and the same inputs will always produce the same
| outputs. There is no black box translation occurring, like
| occurs with computer software.
| exe34 wrote:
| > setting a precedent when it's convenient is not the same
| thing as writing a law
|
| In the UK, they're pretty much the same thing. You need a new
| case or a statute law to overturn a precedent.
| graemep wrote:
| I think historically the law was introduced to make it harder
| for people to contest things like speed camera based speeding
| tickets.
| disgruntledphd2 wrote:
| > The article cites "mechanical systems" as being infallible,
| and reading that language, it reads to me as some archaic
| legislation that never got updated for computer software.
|
| Apparently the law was introduced along with speed cameras,
| as they were continually being challenged in court.
| M2Ys4U wrote:
| >The article cites "mechanical systems" as being infallible,
| and reading that language, it reads to me as some archaic
| legislation that never got updated for computer software.
|
| It's worse than that.
|
| The law was fixed in 1984[0] and then the fix was
| intentionally reversed in 1999.[1]
|
| [0] https://www.legislation.gov.uk/ukpga/1984/60/section/69/1
| 991...
|
| [1] https://www.legislation.gov.uk/ukpga/1999/23/section/60
| michaelt wrote:
| I know that computers aren't infallible.
|
| But I also know that traditionally, tills and bank accounts are
| pretty reliable.
|
| Sure, the store might charge you the wrong amount if the price
| label on the shelf and the PC don't match up, because by law
| the label on the shelf is the source of truth. But other than
| that? If the till says my purchases add up to PS23.45, and
| after making the purchase my bank account has a balance of
| PS345.67? I don't validate the arithmetic.
|
| How much money is in my bank account? Pretty much the amount of
| money the bank's computer says is in my bank account, modulo
| any funnyness like pending transactions and cheques that fail
| to clear. The bank doesn't keep a shoebox of cash in their
| vault with my name on it for us to reconcile against.
| mattlondon wrote:
| I suspect that this originated from the era well before
| computers were a common part of life and intended to relate to
| purely mechanical things, and the statutes were just applied to
| computers when they started to appear in court cases.
|
| So e.g. a mechanical time clock or mechanical scales etc were
| probably the sort of thing that was the target of the original
| acts. The assumption is they are working correctly if they
| appear fine. This makes sense for basic mechanical things, and
| there is no point arguing that actually the scales that weighed
| how much the truck weighed were wrong/defective only that one
| time Defendant X used it and never again afterwards, and it was
| not in fact due to Defendant X being negligent that the bridge
| collapsed due to an overweight vehicle etc
|
| As we know, computers are a different level of complexity.
| Being wrong randomly for one off things is very possible.
| rswail wrote:
| Sorry? There have been departments of Weights and Measures
| for literaly millenia.
|
| They have required certification of those sorts of items with
| regular inspections and anti-tampering seals.
|
| So yes, you can challenge evidence from mechanical scales, if
| they haven't been properly inspected and certified at the
| required intervals under the regulations.
| mattlondon wrote:
| Exactly that is my point - so for the example of some
| mechanical scales, if they have all been properly inspected
| and certified, then you can expect them to be reliable and
| accurate.
|
| So in a case, if there is evidence against someone that
| relies on the scales doing their job properly, and those
| scales have been inspected and certified, then you probably
| do not have a valid argument to say "ah yeah but the scales
| the defendant used might have not been working properly
| when they used them!". I.e. the accepted assumption is that
| the scales work correctly.
|
| Up until recently the same sort of assumption was given to
| computer systems from what I can tell. This is how we got
| into the Post Office scandal situation where people
| implicitly trusted that the computer was doing the right
| thing.
| Hilift wrote:
| This is particularly true of accounting systems. Anyone who has
| worked in a similar environment with thousands of users on a
| distributed ledger.
| Dalewyn wrote:
| >Every person who has ever programmed a computer or worked in
| any complex system knows they can't be relied upon 100%.
|
| I know I can rest assured the Excel spreadsheet for my monthly
| and annual budget is perfectly accurate and reliable.
|
| I know the computers powering the stuff my life depends on are
| perfectly not accurate and reliable.
|
| Put another way: The microwave oven or coffee maker in my
| kitchen? Yeah, the 'pooters in them are working perfectly. The
| mainframes jackhammering away at the Automated Clearing House?
| My money will get through the banking system perfectly
| eventually some day. The jetliner or my car I'm about to get
| in? Dude, that thing better have dozens of computers acting in
| redundancy because that shit ain't working.
|
| I wonder if there's a law stating that the reliability of a
| computer is inverse to the value of the workload.
| bostik wrote:
| > _I know I can rest assured the Excel spreadsheet for my
| monthly and annual budget is perfectly accurate and
| reliable._
|
| Consider yourself lucky that your use cases are all on the
| happy path.
|
| There are entire categories of bugs and inconsistencies where
| Excel's behaviour is known to be wrong, but which can't be
| fixed because the rest of the ecosystem depends on those same
| errors to manifest in the same ways.
|
| For example - formulas with cycles have an upper bound as to
| how many times they are allowed to cycle. If you happen to
| hit the ceiling before your values converge, you will be left
| with the values calculated on the last iteration.
| rich_sasha wrote:
| I think that take is slightly missing the point.
|
| Notwithstanding the horror that is the Horizon fallout, the
| legal rule is much narrower than what is bandied around. It
| merely says you can't just say "computer got it wrong" and
| expect the other side to prove otherwise. Or in other words,
| you need evidence of incorrectness if you're going to claim it.
|
| Now with the Horizon scandal, there was very clearly plenty of
| evidence. The extraordinary number of mismatched books. Some
| cases of physical records not matching electronic (these were
| the few guys that got away). The issue was that the Post Office
| investigators lied about the evidence, buried it with
| intimidation, legal threats and NDAs.
|
| The law may or may not be bad _as it stands_ but AFAIU this is
| like blaming a "computer bug" for the Boeing 737 Max crashes.
| It wasn't - it was human willful errors executed by imperfect
| code.
| MichaelDickens wrote:
| > It merely says you can't just say "computer got it wrong"
| and expect the other side to prove otherwise.
|
| According to the standard legal practice of innocent-until-
| proven-guilty, you can, in fact, say that and expect the
| other side to prove otherwise. So this ruling violates one of
| the most fundamental principles of criminal law.
| rich_sasha wrote:
| Right - it unquestionably _is_ a custom rule. But it 's not
| as broad as it is made out, not is it the _fundamental_
| problem here. The Post Office knew the system is faulty,
| hid that evidence and lied in court under oath about it.
|
| As to lack of evidence... It's not so simple. A popular UK
| insurance hack is when people overtake you on a motorway
| and slam the brakes, to make a claim on your insurance.
| Unless you present evidence this was done to you, like
| dashcam footage, you are presumed to be responsible. You
| don't need the scammer to prove they didn't crash into you.
| I'm sure the US has similar mechanisms, where fault is
| presumed, like if a car hits a pedestrian.
|
| This is similar. Computer systems are presumed to be
| correct unless evidence is presented. Maybe it's a bad
| rule, but it's not the horrendous dystopian catastrophe it
| is declared to be everywhere.
| xxpor wrote:
| Without accepting this specific rule, that's not how any
| rules of evidence work in any (common law) court. There
| needs to be a point where things are presumed true to
| maintain a working court system. Imagine if e.g. you had to
| prove the _concept_ of DNA testing every time it was used.
|
| In the US, see for example the debate between the Frye
| standard vs. the Daubert standard.
| gmerc wrote:
| Critical in the age of generative AI reliability.
| MrMcCall wrote:
| The only thing reliable about them (and the humans that create
| them) is that they easily fool a bunch of fools.
|
| Of course, it does appear that they are incredibly reliable at
| being somewhat reliable, but that's not the kind of reliable
| that I'm looking for, especially when the environmental cost of
| generating them is so expensive, and doubly so as global
| heating ramps up.
| kjrfghslkdjfl wrote:
| "Presumption about reliability of computers"
|
| Anyone else get the sense that these were laws written without
| any input whatsoever from people who actually have working
| experience in complex systems?
|
| I'm convinced that no moral software engineer would ever suggest
| a conviction on the presumption that the system is correct. At
| minimum you'd have to investigate.
| Robin_Message wrote:
| The presumption was about mechanical instruments, from a
| simpler time. As computers expanded from being purely
| mechanical, it never got revised, which is why it is being
| revised now (three or four decades too late, I'll grant you,
| but legislation is a slow process.)
| int_19h wrote:
| Except it _was_ revised once already to point out that
| computers aren 't reliable. And then a decade later, at the
| peak of the dot-com boom, that revision was reverted after a
| review by committee.
| Kye wrote:
| Computers always do exactly as they're told. Unfortunately, there
| are so many people telling them to do so many things at so many
| layers of abstraction that the operator can't be sure of anything
| anymore.
|
| edit: The thing is, had I prefaced it with "ASSUMING A PROPERLY
| FUNCTIONING COMPUTER," someone would still roll in to pick it
| apart. You can't please pedants and make a worthwhile point at
| the same time.
|
| Do circuits get weird? Can a stray cosmic ray flip a bit even in
| a system with ECC RAM?
|
| Sure.
|
| Does this meaningfully affect a bit about the hazards of
| abstractions on top of abstractions?
|
| No.
|
| The point is that abstractions amplify computer problems whether
| they're human error or ghosts in the machine.
| timthorn wrote:
| They don't - random bit flips are expected over time. ECC
| memory is a thing because of this, but cosmic rays can strike
| the processors too.
| Kye wrote:
| I know
|
| I was doing a bit
| bowsamic wrote:
| I don't think jokes make sense when they are just entirely
| wrong though. Instead it just comes across as you being
| ignorant or perpetuating misinformation, rather than a joke
| Kye wrote:
| So far only two of about ten people took it in an overly
| literal way rather than understanding and going with the
| core of it so I think I'll keep aligning my bits toward
| them. Have a nice day.
| bowsamic wrote:
| You'll align your bits towards falsity and ineptitude?
|
| It is a big deal that computers don't always do what
| they're told in this case. STOP perpetuating lies
| Kye wrote:
| There are, broadly speaking, three genres of Misinformed
| About Computers:
|
| The deeply interested, but informed by charlatans. I
| cannot affect their beliefs.
|
| The immune to information.
|
| The person who believes computers are scary, arcane
| objects. I cannot misinform them more than they already
| are.
|
| Meanwhile, everyone else understood the point is that
| abstractions amplify computer problems whether they're
| human error or ghosts in the machine. Your model of
| misinformation spread needs work.
| notimetorelax wrote:
| They don't, computers generate faulty results once in a while,
| even perfectly functioning computers. And on top of that
| hardware may degrade generating more errors.
| Kye wrote:
| Okay
| jl6 wrote:
| Seems to me that computers _do_ generally perform their tasks
| reliably. (I say generally, not infallibly; cosmic rays and power
| failures and such). The issue is really about whether their human
| programmers have successfully programmed the computer to do the
| intended task, and about whether their human operators are
| successfully interpreting computer outputs. So this is surely
| about what constitutes an expert witness, and how reasonable it
| is to rely on the testimony of an expert witness, where computers
| are concerned.
| Ekaros wrote:
| On can really consider how rare hardware issues leading to
| wrong computing are for your widely used computer chips. In
| general it seems they are very reliably correct and fixed in
| reasonably quick order.
|
| Same really cannot be said about software. But on computational
| level it is actually amazing when you think about it.
| jbb67 wrote:
| I understand why the presumption was there.
|
| If it's not then every single case of tax evasion people would
| claim that they had paid and the computer must have lost the
| payment. Every single traffic camera offense would be contested.
| Every time you didn't have a train ticket people would claim the
| computer lost it. And so on.
|
| And there would be months and months of delays if every time it
| had to be proved beyond a doubt that the equipment was working
| fully in _that_ case out of all proportion to the risk of there
| actually being a problem.
|
| However _clearly_ there does need to be a way to challenge the
| presumption too. Of course just assuming equipment is working is
| entirely unfair.
|
| After a brief look at the article it looks a reasonable weakening
| of the presumption which allows the court to consider and reject
| a challenge when they think fit according to the court rules (to
| be established) Pretty much like any other form of evidence.
| thinkingemote wrote:
| For some background. The legal presumption dates from 1997.
|
| "Previously, section 69 of the Police and Criminal Evidence Act
| 1984 required anyone introducing computer-generated evidence to
| show the system was operating correctly and not being used
| improperly.
|
| The change followed the Law Commission's 1997 review of the law
| on hearsay evidence."[1]
|
| from https://www.lawgazette.co.uk/law/it-experts-call-for-
| review-...
|
| [1] review https://cloud-
| platform-e218f50a4812967ba1215eaecede923f.s3.a... (page 197)
|
| ---
|
| Another interesting briefing note:
|
| "The legal rule that computers are presumed to be operating
| correctly - unforeseen and unjust consequences":
| https://www.benthamsgaze.org/2022/06/30/the-legal-rule-that-...
|
| They suggest, when requested, that a party gets:
|
| - records of known errors and bugs in the system, their effect,
| and the actions taken in response
|
| - description of information security and other relevant
| standards and processes followed
|
| - reports of audits performed on the system and how it is
| managed,
|
| - evidence showing that reports of errors are managed properly
| and that changes to the system are properly controlled
|
| - evidence confirming that the search for documents was performed
| adequately, and was done so by a person with appropriate
| authority and knowledge, and
|
| - assurance that reasonable steps have been taken to establish
| that the evidence presented has not been tampered with.
| Neil44 wrote:
| When I worked for a drug company everything had to be documented
| and proceduralised. Because if you ask the FDA if you can sell
| your pills to the public they will ask how you know they're safe.
| And if you keep asking enough why's you get down to whether you
| can trust the output of computers. So every computer's build
| process and software stack has to be to documented and approved
| procedure so that you have a wad of paper to show them, when they
| ask that 'why'. It seems reasonable that if you're asking to send
| someone to jail based on the output of a computer you should be
| able to ask for reasonable proof that the output is correct.
| Hilift wrote:
| Next up: What did Birmingham do with the PS216 million for the
| 2020 Oracle Financials upgrade that will not be completed until
| 2026? For a city that has a 46% child poverty rate.
|
| Sometimes the technology isn't the problem, it's the people.
| onion2k wrote:
| _Sometimes the technology isn 't the problem, it's the people._
|
| It's _always_ a people problem.
| linsomniac wrote:
| This reminds me of Kurt Vonnegut's book Player Piano. Imagine
| LLMs/AGI but implemented with vacuum tubes. It includes the line
| (from memory): We're sure of his sentencing, because we replaced
| all the vacuum tubes and ran the trial again and got the same
| result."
|
| https://en.wikipedia.org/wiki/Player_Piano_(novel)
| deskr wrote:
| This scandal is probably the biggest miscarriage of justice in
| Europe. The fundamental issue was perhaps "reliability of
| computers".
|
| But the biggest scandal is that a lot of people knew what was
| happening and were either silenced or told others to be silent.
| w14 wrote:
| Indeed, but of equal stature in my opinion is that they people
| on the receiving end of the miscarriage of justice are
| continuing to be abused by the so-called compensation process.
|
| During the inquiry, the statement was made, can't remember by
| whom, that value for taxpayer money had to be respected.
|
| I think its past that point, and I would like to know why the
| instigators of the miscarriage aren't at risk of losing their
| personal assets to (at least partly) cover the cost. Maybe we
| have to wait for the inquiry report for that, but in the
| meantime it is pretty clear that 'value for taxpayer money'
| means, at least to some degree, kicking the can down the road
| until as many of the claimants are dead as possible.
| gadders wrote:
| Does this apply to voting machines?
| M2Ys4U wrote:
| The UK doesn't use any form of electronic voting. I don't think
| anywhere even uses electronic _counting_ machines any more
| either.
| djoldman wrote:
| The linked piece by Stephen Mason sets out all the interesting
| context for the law portion (not the postal service fiasco).
|
| https://read.uolpress.co.uk/read/3a219939-14d3-45c9-9fe0-9b4...
| dang wrote:
| Links related to the UK post office scandal. Others?
|
| _Fujitsu bugs that sent innocent people to prison were known
| "from the start"_ - https://news.ycombinator.com/item?id=39059307
| - Jan 2024 (270 comments)
|
| _Fujitsu CEO Deposition - Post Office Horizon IT Inquiry_ -
| https://news.ycombinator.com/item?id=39059302 - Jan 2024 (1
| comment)
|
| _Fixing Horizon bugs would have been too costly, Post Office
| inquiry told_ - https://news.ycombinator.com/item?id=39039712 -
| Jan 2024 (59 comments)
|
| _Fujitsu says it will pay compensation in UK Post Office
| scandal_ - https://news.ycombinator.com/item?id=39023695 - Jan
| 2024 (26 comments)
|
| _How a software glitch at the UK Post Office ruined lives_ -
| https://news.ycombinator.com/item?id=39010070 - Jan 2024 (326
| comments)
|
| _Post Office Horizon scandal explained: Everything you need to
| know_ - https://news.ycombinator.com/item?id=38983144 - Jan 2024
| (8 comments)
|
| _A TV Show Forced Britain 's Devastating Post Office Scandal
| into the Light_ - https://news.ycombinator.com/item?id=38951802 -
| Jan 2024 (168 comments)
|
| _British Post Office Scandal_ -
| https://news.ycombinator.com/item?id=38937705 - Jan 2024 (149
| comments)
|
| _How the Post Office 's Horizon system failed: a technical
| breakdown_ - https://news.ycombinator.com/item?id=38931792 - Jan
| 2024 (4 comments)
|
| _Ex Post Office CEO hands back award after IT failures lead to
| false convictions_ -
| https://news.ycombinator.com/item?id=38930011 - Jan 2024 (127
| comments)
|
| _Post Office Horizon Enquiry - Fujitsu Report on Eposs PinICL
| Task Force (1998)_ -
| https://news.ycombinator.com/item?id=38926582 - Jan 2024 (1
| comment)
|
| _Fujitsu bosses knew about Post Office Horizon IT flaws, says
| insider (2021)_ - https://news.ycombinator.com/item?id=38890468 -
| Jan 2024 (8 comments)
|
| _Mr Bates vs. the Post Office_ -
| https://news.ycombinator.com/item?id=38869011 - Jan 2024 (3
| comments)
|
| _What went wrong with Horizon: learning from the Post Office
| Trial_ - https://news.ycombinator.com/item?id=38867712 - Jan 2024
| (19 comments)
|
| _UK Post Office: 700 Horizon software scandal victims to receive
| PS600k each_ - https://news.ycombinator.com/item?id=37561428 -
| Sept 2023 (40 comments)
|
| _After 20 years, the Post Office scandal cover-up is happening
| in plain sight_ - https://news.ycombinator.com/item?id=36778486 -
| July 2023 (1 comment)
|
| _The UK post office database scandal - "can 't see the bug =
| user is a thief"_ - https://news.ycombinator.com/item?id=35837576
| - May 2023 (2 comments)
|
| _Hundreds of lives ruined by faulty UK Post Office computer
| system_ - https://news.ycombinator.com/item?id=35792896 - May
| 2023 (4 comments)
|
| _Ex UK Post Office staff tell inquiry of stress of IT scandal_ -
| https://news.ycombinator.com/item?id=30394685 - Feb 2022 (2
| comments)
|
| _Post Office scandal: Public inquiry to examine wrongful
| convictions_ - https://news.ycombinator.com/item?id=30329668 -
| Feb 2022 (149 comments)
|
| _Post Office scandal: 'I want someone else to be charged and
| jailed like I was'_ -
| https://news.ycombinator.com/item?id=30329510 - Feb 2022 (2
| comments)
|
| _Bad software sent postal workers to jail_ -
| https://news.ycombinator.com/item?id=26973583 - April 2021 (1
| comment)
|
| _Convicted Post Office workers have names cleared_ -
| https://news.ycombinator.com/item?id=26924882 - April 2021 (187
| comments)
|
| _UK court clears post office staff convicted due to 'corrupt
| data'_ - https://news.ycombinator.com/item?id=26913037 - April
| 2021 (284 comments)
|
| _UK legal system assumes that computers don 't have bugs_ -
| https://news.ycombinator.com/item?id=25518936 - Dec 2020 (24
| comments)
|
| _Post Office scandal: Postmasters celebrate victory against
| convictions_ - https://news.ycombinator.com/item?id=24661321 -
| Oct 2020 (2 comments)
|
| _Bankruptcy, jail, ruined lives: inside the Post Office scandal_
| - https://news.ycombinator.com/item?id=24440476 - Sept 2020 (1
| comment)
|
| _Postmasters were prosecuted using unreliable evidence_ -
| https://news.ycombinator.com/item?id=23454606 - June 2020 (2
| comments)
|
| _Faults in Post Office accounting system led to workers being
| convicted of theft_ -
| https://news.ycombinator.com/item?id=21795219 - Dec 2019 (104
| comments)
|
| _Post Office hires accountants to review sub-postmasters '
| computer claims_ - https://news.ycombinator.com/item?id=4143107 -
| June 2012 (1 comment)
| lifeisstillgood wrote:
| For a long time I was pushing a campaign that every piece of
| software paid for with public funds should be made FOSS (unless
| national security etc).
|
| I struggle to work out why a post office point of sale should be
| vital to Britains security and we should have been able to see
| the code.
|
| On top of which I believe that making such code open means there
| will become a eco-system of ISPs who will be able to support,
| integrate and improve the software and provide local government
| users (ie postmasters) with worthwhile consultancy Under these
| circumstances it's hard to see how this would have gone uncovered
| for so long.
|
| (Or rather, not uncovered, but unbelieved. The great tragedy of
| this affair is that us was known and reported on for years - but
| nothing happened. You know those films where the hero manages to
| get the proof to the newspaper / tv station and the film ends as
| the bad guys are bundled into police vans - yeah not so much.
| bitwize wrote:
| If they don't call this the Computer Says No Act, I'd say there
| was a missed opportunity.
___________________________________________________________________
(page generated 2024-12-02 23:01 UTC)