[HN Gopher] The legal rule that computers are presumed to be ope...
___________________________________________________________________
The legal rule that computers are presumed to be operating
correctly (2022) [pdf]
Author : zinekeller
Score : 101 points
Date : 2024-04-16 14:43 UTC (8 hours ago)
(HTM) web link (www.benthamsgaze.org)
(TXT) w3m dump (www.benthamsgaze.org)
| qup wrote:
| They must run a bug free system. Probably TempleOS
| kouru225 wrote:
| "Your honor this God song proves that my client is innocent."
| MaxBarraclough wrote:
| SeL4 would be a better choice, if you count it as an OS.
| gavinhoward wrote:
| I have seen a lot of dumb legal takes, usually by non-lawyers and
| non-judges.
|
| But this is so very dumb.
| icyberbullyu wrote:
| This is just an assumption that generally, computers are doing
| what they're supposed to. It doesn't mean that you can't
| challenge it with evidence. With the amount of evidence being
| submitted being increasingly digital, I'm sure the intention
| behind this is to stop people from jamming up the courts by
| calling every single digital artefact into question.
| euroderf wrote:
| Your point is valid. But it does sound like a barrier to
| entry. In the sense of, increasing fixed costs for access to
| the legal system on a fair basis
| minimalized wrote:
| The article makes a strong case otherwise. Proving a system
| built and maintained by a large, well heeled entity is
| unlikely to be possible for all except other large, well
| heeled entities.
| zardo wrote:
| How are you supposed to get the evidence in a situation like
| this?
|
| It's not like they're going to give you the keys to the
| server room.
| Brian_K_White wrote:
| But every single digital artifact is questionable. It's
| inconvenient but it's also true.
| cbsmith wrote:
| > This is just an assumption that generally, computers are
| doing what they're supposed to.
|
| It's more than that. You don't have to instruct a jury that
| they have to accept that a person does what they're supposed
| to do unless proven otherwise... unless that person wrote
| code for a computer.
| mmcdermott wrote:
| The linked document doesn't feel unreasonable to me. The
| concrete example given in the document is the Post Office
| Horizon Scandal where shortfalls in accounts were identified
| based on computer records. All the working presumption
| correctly would mean is that the defense can't be an idle
| "well, it's a computer it had bugs". If the computerized
| records are provided as evidence, the defense has to give a
| reason to believe that bugs account for those shortfalls.
|
| The alternative would be to say that only a system that has
| been proven correct could be used as evidence which would force
| pretty much all accounting back to paper.
| Infernal wrote:
| > The alternative would be to say that only a system that has
| been proven correct could be used as evidence which would
| force pretty much all accounting back to paper.
|
| Assuming that the paper accounting could be proven correct,
| when the computerized accounting could not, your proposed
| alternative would seem to be an upgrade.
| bee_rider wrote:
| Paper accounting seems significantly more error prone, at
| least for a given level of convenience, right? I mean the
| computer might be worried about comic rays flipping bits,
| but ink on paper can be smudged by fingers, and they don't
| even have to be traveling at an appreciable percentage of
| the speed of light.
|
| You can make paper backups of course, but you can also make
| electronic ones, the electronic ones can be shipped around
| the world basically for free.
| cbsmith wrote:
| The legal system already has checks and balances to
| account for errors in paper accounting. They don't
| require an assumption that the paper accounting is
| correct, but rather rely on the jury to make judgements,
| because juries are familiar with the limitations of paper
| accounting.
|
| We're still a long way away from pulling that off with
| computers, but the idea that someone could have made a
| mistake shouldn't be that hard to grasp.
| exe34 wrote:
| Would that be a bad thing? If you want the convenience you
| either pay for something that is proven correct or you take
| the inefficiency of paper (under your implicit assumption
| that it's somehow invulnerable to error, burning down, etc),
| or you accept that you take the convenience but will not be
| able to use it in court as evidence. If you suspect somebody
| is stealing, you could always take further measures like
| switching to magical paper that can't go wrong, or set
| regular inspections, or new procedures, etc.
|
| Don't pretend that fucking over innocent lives in the pursuit
| of profits is a law of nature.
| sidewndr46 wrote:
| Isn't the Horizon scandal the opposite? Everyone knew the
| software was faulty the whole time and no one cared about the
| consequences.
| jcrawfordor wrote:
| That's exactly what this proposal addresses. Consider the
| paper they cite, in part, for the proposal:
|
| "It is a matter of surprise that important documentary
| records, such as the Fujitsu Known Error Log (KEL), were
| disclosed only in response to a direction from the court
| and in the face of opposition by the Post Office."
| (https://journals.sas.ac.uk/deeslr/article/view/5240/5083)
|
| The proposal in this paper would require prosecution to
| disclose extant documents related to the reliability of the
| system. In the Horizon case, such documents existed, but
| were never disclosed, as current practices did not require
| the prosecution to do so.
| sidewndr46 wrote:
| OK, the original comment makes more sense now.
|
| Are prosecutors in the UK required to disclose both
| incriminating and exculpatory evidence to the defendant?
| chrisjj wrote:
| Yes.
| olliej wrote:
| I believe so, but they have a notorious track record of
| not doing so.
|
| But from other articles it sounds like these prosecutions
| involved some kind of parallel legal system just for the
| mail?
|
| I think though that the issue that this saga hit is that
| the law was such that the computer "evidence" was
| presumptively correct so they tried to pretend any "bugs"
| they were aware of didn't change that fact and so the
| existence of bugs was not "exculpatory" - obviously BS,
| but it seems like that was the core behaviour/belief of
| the post office. That put the victims in the position of
| having to disprove the accusation but the only "evidence"
| in the case was the presumptively true report from the
| buggy system.
|
| Ie the only evidence presented was a system that was
| buggy, but the victim could not get evidence the system
| was broken without first proving it was broken.
|
| Yay!
| salawat wrote:
| >The alternative would be to say that only a system that has
| been proven correct could be used as evidence which would
| force pretty much all accounting back to paper.
|
| No. It'd actually just force authorship and disclosure of
| test cases.
| dragonwriter wrote:
| > The alternative would be to say that only a system that has
| been proven correct could be used as evidence which would
| force pretty much all accounting back to paper.
|
| No, there are many alternatives. Including, most critically,
| _the one laid out in the paper under discussion_. Which is -
| _spoilers_ - not "only a system that has been proven correct
| could be used as evidence".
| kd5bjo wrote:
| I can understand how they got here: Most computer-provided
| evidence will be something like a printout of an email thread
| or a saved Excel sheet. _Proving_ that the hard drive, printer,
| CPU, drivers, application software, etc. are all untampered and
| working properly is a huge amount of work that will very rarely
| end up finding a problem.
|
| The original rule-makers probably never considered the idea of
| a giant, bespoke enterprise software system; it's just a
| particularly terrible edge case that got caught up in what
| looks like a reasonable efficiency measure.
| cbsmith wrote:
| > I can understand how they got here: Most computer-provided
| evidence will be something like a printout of an email thread
| or a saved Excel sheet. Proving that the hard drive, printer,
| CPU, drivers, application software, etc. are all untampered
| and working properly is a huge amount of work that will very
| rarely end up finding a problem.
|
| Eye witnesses, expert witnesses, etc. can be unreliable. It's
| very difficult to prove that they have not made any errors.
| However, we don't ask juries to presume they are infallible.
| NoboruWataya wrote:
| > Eye witnesses, expert witnesses, etc. can be unreliable.
| It's very difficult to prove that they have not made any
| errors. However, we don't ask juries to presume they are
| infallible.
|
| There is a difference between being infallible and being
| correct. No one assumes that computers _cannot err_ , just
| that they have not erred unless there is reason to believe
| they have. Likewise, if a witness gives coherent evidence
| and no one has any reason to assume they are wrong or
| lying, that evidence will not generally be disregarded
| simply because humans are fallible and therefore the
| evidence is presumed to be flawed.
| kube-system wrote:
| No, this is dumb:
|
| > required the prosecution to prove that a computer was
| operating properly at the relevant time before a document
| produced by such a computer could be admitted as evidence.
|
| But I'm sure you actually mean to agree, and that your comment
| was just garbled in transit. :)
| minimalized wrote:
| It's over simplification of a nuanced situation. Over
| simplification in either direction is a problem.
| kube-system wrote:
| A presumption is not a simplification of anything. It is a
| starting point.
| minimalized wrote:
| That's fair. But as the article shows, that presumption
| ends up weighing heavily in favor of computers working
| correctly. The nuance is in the difference between
| plumbing, which generally works well, and the bespoke
| business logic of individual applications, especially
| when said logic was responsible for the existence of
| critical evidence in a criminal case. The difference
| matters enormously and is not observed at all in the law.
| koliber wrote:
| There's an excellent Radiolab podcast episode about how often
| cosmic rays cause computer errors in practice. It's engaging and
| educational: https://radiolab.org/podcast/bit-flip
|
| I always knew about the theoretical cosmic ray bit flips. Before
| listening to this episode, I did not stop to think how often they
| actually cause problems.
| remisharrock wrote:
| Am I the only one with exactly this episode that does not load
| ? All the others are ok ?
| archargelod wrote:
| Same for me. After some searching I've found a mirror that
| still works [0].
|
| [0] - https://podcastaddict.com/wissenschaft-auf-die-
| ohren/episode...
| tzs wrote:
| I tried to observe bit flips and failed. I'm not sure if there
| was a problem with my methodology or my apartment was well
| shielded or what.
|
| My methodology was simple. On a Linux home server that had
| plenty of spare memory (non ECC RAM) I ran a process that
| simply alloced a large buffer and filled it with a pattern. It
| would then periodically scan through the buffer looking for
| changes to the pattern.
|
| I ran this for over a year which should have been long enough
| given the amount of RAM I was using and the rates that I found
| in the literature for cosmic ray induced bit flips resulted in
| several flips.
|
| My method would have missed a flip if the page it happened on
| had been paged out sometime in the past and that paged out copy
| still existed, and the bit flip happens between the time of the
| last scan and the time the kernel decides to discard that page.
| On the next scan it would page fault and load the good page.
|
| But the system was very lightly loaded and almost never
| actually had to page things out, so most of the time if a bit
| flipped it should have still been there by the next scan and so
| I don't think this explains why I saw no flips.
|
| A few years later I got a Mac Pro, which had ECC memory. I used
| that mostly at work from 2008-2017. I got another Mac Pro in
| 2009 which I used at home from 2009-2017. I'd occasionally look
| at the memory status in System Report which should say "ECC
| Errors" if the ECC had to fix any errors and only ever saw
| "OK". I'm not sure if that resets on boot and I only looked
| occasionally so if it does reset than it is quite likely I
| would have missed an error statuses.
| dunham wrote:
| I recall seeing system log messages about ECC error
| correction on Sun Sparc machines back in the 90's. I don't
| know if it was radiation or flaky memory though.
| Arch-TK wrote:
| I think it's possible that we've historically mis-attributed
| memory errors to cosmic rays when in fact they were caused by
| heat.
|
| If you ran that machine in a hotbox at 85degC for a year I
| think you probably would have experienced higher error rates.
| Bonus points if you also force the data to swap out a lot so
| it gets transferred through as many data paths as possible.
| toast0 wrote:
| My experience from a ~ 2000 server job with all ECC RAM is
| that errors cluster a lot. Easily 90%+ of the systems never
| had any ECC errors (correctable or no) in their whole life.
| Some systems would get one error a day. A small number got
| 100s per hour until we shut them down. Even less got
| thousands per hour. One in my whole career got like 50,000 in
| the time between when it started running very slowly (from
| processing ECC interrupts) and the next hourly reporting
| interval. You might get close to the literature numbers by
| taking that one instance and dividing it over the total ram
| byte hours. Although that was probably hardware failure and
| not comsic rays, but then you don't get a different ECC code
| for comsic rays.
|
| Also, there was an expectation that comsic ray induced bit
| errors would grow as ram circuit features shrunk, but it
| ended up not happening; reasons unknown or at least I never
| saw anything suggesting a reason.
|
| Getting errors in a small sample of RAM is unlikely, unless
| you specifically induce them by using debug features or
| misconfiguring your system (but some systems conspire to
| misconfigure themselves, making it easier to observe! a
| couple years ago, retail motherboards really liked setting
| the ram voltage too low)
| moffkalast wrote:
| > 50,000
|
| It's not 3.6 roentgen...
|
| Joking aside that's incredibly fascinating, I never thought
| that ECC memory has that much of a performance impact.
| Might be more optimal to just get a large jerry can of
| water and put that over the server as radiation shielding
| lol.
| toast0 wrote:
| ECC interrupts were never a problem at reasonable counts.
| It's just this machine where the memory was falling apart
| where it was a problem. Our system was robust to a
| machine halting, but not so great at dealing with a
| machine running very very slow. Plus, it wasn't easy to
| connect and shut down the service (maybe we should have
| just killed it from IPMI, but this was the only time it
| happened, so learning experience).
|
| I had a similar issue one time where a Pentium III era
| server rebooted and came up with a comically small amount
| of memory, maybe 2-4 mb instead of 128 mb. That wasn't
| too bad, because it was a very lightly service; important
| to be on its own machine for reasons, but didn't need
| much. Just ran a little slow when it was running from
| swap. I think it did trigger a swap usage alert, and then
| it was like why is it swapping, why is it so slow, wait
| _why doesn 't it have any memory!?_
| thedrexster wrote:
| > It's not 3.6 roentgen...
|
| I'm told it's the equivalent of a chest x-ray...
|
| I see you, brother! :D
| bee_rider wrote:
| Were some of your servers in lead-lined coffins while
| others weren't?
|
| Jokes aside, that it seems really unlikely that cosmic rays
| would be clustered past, like, a couple hours, right? It
| isn't like some neutron star is, like, tracking your server
| as the world spins (well, I hope not, I mean who'd you piss
| off for that to happen?).
|
| Anyway, this fits my totally unscientific expectation that
| cosmic rays are just sort of like an informal description
| of hardware bugs that nobody can reasonably find
| beforehand. A server that seems to be hit by lots of cosmic
| rays probably has a dodgy connection somewhere inside it,
| but I mean maybe it's the RAM, swap that out or replace
| it... but maybe inside the chip SOC, so what are we going
| to do, bust out the electron microscope to check all those
| connections?
| toast0 wrote:
| You'd have to ask our host; these were all rental
| dedicated servers, none of us ever got to see them at
| all. Pictures of their racks never included coffins
| though. I don't think we pissed anybody off enough for
| them to get out the cosmic ray gun, but I'm also thinking
| the people we did piss off didn't have cosmic ray guns
| anyway. ;)
|
| In terms of diagnostics, we pretty much just asked for
| ram replacement, if that didn't work, cpu replacement, if
| that didn't work, motherboard replacement. If that didn't
| work, the chassis / rack position is clearly cursed,
| don't give us anything there again, please. :D I don't
| know what they did with the hardware we didn't like,
| maybe send it to the manufacturer, maybe give it to
| customers they don't like, maybe surplus it.
| ghaff wrote:
| This particular server wasn't mine as a product manager.
| But we had what passed as a distributed server in the
| mid-80s and our biggest company was a retail
| insurance/finance company of some sort. No ECC.
|
| So, at scale, they were getting failures constantly. (It
| didn't help that the QIC tape backup was basically write-
| only.)
| prpl wrote:
| The incidence of cosmic rays is biased vertically, especially
| close to sea level where you get more muons than electrons,
| so that is one thing you would want to think about - you'd
| want to maximize your surface area toward that because your
| detection volume is so low.
|
| As a rule of thumb, you get 2 muons through your head a
| minute - but of course your head has a huge volume compared
| to memory chips.
| mhitza wrote:
| Using cosmic rays as a practical dns hijacking tool (defcon)
| https://www.youtube.com/watch?v=aT7mnSstKGs
| dekhn wrote:
| I always assumed computers were "perfect"- at least, always
| executing their instructions according to a spec. FDIV and
| other bugs convinced me otherwise. Then, I worked on a system
| (a large processing accelerator) where some small fraction of
| machines had chips that would fail on a limited set of
| operations (producing the wrong result for a known input,
| vaidated by offline computation). Now I treat computers as
| statistically correct- you have to assume there is some level
| of corruption/error occurring due to hardware bugs or physical
| events.
| altruios wrote:
| you would enjoy Dave Ackley's perspective on 'robust
| computing' vs 'efficient computing'. But his one of his ideas
| is allowing for error: which can speed up calculations if you
| can also reduce the error as the calculation goes on.
| dekhn wrote:
| Well... there's several things going wrong here and his
| perspective only seems to cover some of them.
|
| In the case of a scientific simulation, we can find
| algorithmic speedups that are "lossy"- an example would be
| approximations to n-body systems, where you need to
| calculate n-squared interactions (between all pairs). You
| can calculate all n-squared interactions which produces teh
| completely correct result. But since atoms that are far
| away don't interact strongly (falls off as 1 over r squared
| or more). So you can maintain a neighborlist- all atoms
| within a distance R- cheaper than you can calculate
| n-squared interactions, with some tiny error that is
| unknown. It's assumed in many cases the errror is neglible
| and the speedup is huge.
|
| Or you can switch to using a particle-mesh method which
| involves taking a fourier transform, doing some work in
| fourier space, then an inverse transform. The results are
| nlogn, and the error is small (and known). Speedup is
| signfiicant but takes much, much more computational skill
| and infrastructure.
|
| Next, the case I'm referring to of an accelerator running a
| tensorflow job, it's a totally different scenario- here,
| some random subset of machines will repeatedly return the
| wrong result- say, for a matrix-vector operation. Maybe
| garbage numbers, maybe all zero, maybe some infs. When that
| gets summed into your gradient, it often causes blow-up and
| the entire job terminates. It's not clear whether it makes
| sense to make high-performance jobs have to be robust- I
| see them as special cases where you're working hard to make
| sure the computing substrate is effectively 100% reliable
| (by sending/fixing those machines).
|
| Other folks have observed that some amount of small noise
| injection to the gradient can help training, but the sorts
| of errors I've seen almost immediately terminate the
| training job. I don't mind intentional noise addition, but
| noise due to hardware that is provably, reliably, and
| repeatedly miscalculating results? Not so much.
| Nomadeon wrote:
| As we went from zero to 10K+ embedded systems (full PCs with
| significant RAM) the issues got weirder.
|
| The best was a one-off error log along the lines of "unknown
| type System.DateTime". Huh? That's a system defined type that
| just went missing. Never saw it again.
|
| Another at a different employer was a crash that occurred after
| a check condition that absolutely should have gated the crash
| from being reached. Single threaded. Simple microcontroller.
| Had to reflash it to flip the bit back. After doing the math on
| how much RAM we had in the wild vs. cosmic bit flip rates
| reported in super computers, we had to expect one flip per
| year.
|
| If it's a safety critical system, server or not, use ECC RAM!!
| repiret wrote:
| I am of the opinion that far more than safety critical
| systems should use ECC. You should use ECC anytime bit flips
| might cost you more money then the ECC does, which is why I
| insist on ECC for my desktop computers.
| eschneider wrote:
| Given a large enough installed base, any unlikely but
| possible problem will occur for some segment of the user
| population. Guaranteed. :/
| repiret wrote:
| At work we make a hardware product, and among other things, it
| has a small microcontroller on it. We keep several hundred
| instances of our product to run automated integration tests. At
| that scale, we find SRAM bit-flips one of the microcontrollers
| every few weeks. We have a handful of software measures that
| are able to stop most instances of bit-flip from having a user-
| visible effect.
| bcrl wrote:
| That kind of rate smells like a hardware bug. Insufficient
| decoupling caps maybe?
| repiret wrote:
| That seems plausible. I wasn't deeply involved in the
| investigation, but the silicon vendor supports the cosmic
| ray hypothesis.
| bcrl wrote:
| The rate of cosmic rays impacting hardware should be
| extremely low, unless there's some kind of environmental
| factor (like being at a very high altitude). Or maybe
| there was a special train going by as in:
| https://www.jakepoz.com/debugging-behind-the-iron-
| curtain/
| readyplayernull wrote:
| Nowadays, in Windows 11, after a few hours the Start icons stop
| responding.
| skybrian wrote:
| I'm wondering how it would work otherwise. Would computer systems
| need to be certified to be acceptable for ordinary record-
| keeping?
|
| Extending this rule to LLM's would clearly be disastrous. At a
| minimum, a record-keeping system needs to be written the old-
| fashioned way.
| minimalized wrote:
| The alternative is treating systems based evidence the same as
| witnesses. They're examined and cross referenced because they
| are rightly not presumed to be reliable.
| jack_riminton wrote:
| I like this idea but since software and the data that runs
| through it changes constantly how would a system be
| interrogated if an incident were several years ago and the
| complete data picture cannot be reproduced?
| slotrans wrote:
| Then the maintainers of the system have failed in their
| duty, the evidence cannot be verified, and it must be
| thrown out.
| numpad0 wrote:
| I'm guessing the other possibility is requiring a third party
| entity, like a university professor in CS, to testify
| unlikeliness of computer malfunction due to cosmic rays &&
| bugs, for each and every computer generated evidences.
|
| If the `alleged theft amount by defendant` == (unsigned
| int)((int)-1), that should cover those cases, at least in
| theory. The profs would just blindly vouch it, but at least
| it'll be a point.
| orangesite wrote:
| Nope nope nope and nope. The author of the software and their
| source code shall be interrogated on the witness stand by one
| of their peers and a proof checker.
|
| edit: gender neutral language
| numpad0 wrote:
| That'd be ideal! Require expert testimonials and judges
| with Master's in CS and EE or better for any court sessions
| involving computers. Up to UK people though.
| jcrawfordor wrote:
| If you scroll down to page 3, there is a concrete proposal. The
| citations for this are worth reading as they expand on the
| idea, e.g.
| https://journals.sas.ac.uk/deeslr/article/view/5240/5083
|
| The proposal is directly informed by the Post Office scandal in
| that it basically requires the prosecution to produce the same
| already-existing documents that revealed that Horizon had
| significant known faults.
| gotstad wrote:
| Well, in many safety-critical industries (e.g. manufacturing of
| medicinal products) you actually need to maintain objective
| evidence of its validation.
| jobs_throwaway wrote:
| There's dumb, there's outrageously dumb, and then there's this
| olliej wrote:
| It was pretty standard for the time - therac-25 was another
| example where the logic is "computer don't make mistakes
| therefore code running on computers does not make mistakes"
| which is strictly true in the sense that it's exceedingly
| unlikely a computer will execute code incorrectly.
|
| Well into the 90s the general populace considered computers
| infallible, it really took mass adoption and experience with
| how frequently they crashed to change that perception.
| roughly wrote:
| It's interesting to see the negative reactions to this while an
| awful lot of us are employed specifically and payed exorbitantly
| because computers regularly are not operating correctly in weird
| subtle ways that are hard to figure out. Especially as society
| leans into ML models to solve computationally hard problems, the
| legal notion that the computer is "correct" by default absolutely
| needs to go out the window.
| ygjb wrote:
| We are paid exorbitantly to work hard to maintain the illusion
| that the computers are operating correctly.
|
| Any reasonably competent software developer or engineer should
| know that just because the customer dashboard is green and
| everything is working as expected doesn't mean that there isn't
| an absolute dumpster fire raging in the background (at least,
| having worked in tech for 20 years across many different
| verticals, that has been my experience).
| jack_riminton wrote:
| Exactly, and edge cases can almost never be fully ruled out
| with sufficiently complex systems
| TheNewsIsHere wrote:
| And that's even with things operating to some semblance of
| regularity and conformance to some established baseline.
|
| Not even covering the "kick it down the road" issues that
| sometimes (and sometimes don't) evolve into "features that
| aren't bugs" that have workarounds on workarounds on
| workarounds that will eventually have their own bugs. The
| stuff we know we shouldn't do / that isn't good, but that
| happens anyway.
|
| I subscribe to the philosophy that computers cannot be held
| accountable, because they're only operating as intended, or
| at least as implemented. CentOS used to bill itself as a
| "bug for bug" compatible RHEL clone. I have always thought
| that to be a good perspective.
|
| Humans are ultimately responsible for the results of
| computing. In my life that means I review the reports
| output by my ERP and sanity check numbers before I sign tax
| returns or remit quarterly tax payments. I trust that the
| numbers going in are correct, and I trust that (as was said
| up-thread) statistically my ERP is reliable, but at the end
| of the day someone has to be accountable for how the output
| is used.
| aiejrilawj wrote:
| Lots of people here saying judges and lawyers are incredibly
| stupid. For decades there was a giant mystique surrounding
| software and computers, the belief that computers didn't make
| mistakes. And who promoted that belief? Computer and software
| companies. So they could make absurd amounts of money. Credit
| where its' due. Courts should have been much more critical, but
| they were getting hit with a cultural tidal wave generated by us.
| barryrandall wrote:
| The products were promoted using practices approved by the
| legal system. If those practices aren't appropriate to use when
| targeting judges and lawyers, then they're not appropriate for
| anyone.
| kemitchell wrote:
| The authors seem to be torching a semantic straw man here. The
| same abuse of terms affords their spicy title.
|
| I'm not a UK lawyer, but the law they quote says nothing about
| the _logic_ machines are programmed to follow presumptively
| creating reliable evidence. It _could_ be read to say that
| computers should be presumed to be executing the instructions
| they 're given reliably, unless evidence shows otherwise. It's
| about malfunction, not misapplication.
|
| Perhaps some of the Horizon case decisions showed judges
| improperly presuming that Horizon calculated correctly, and not
| just that the computers were running Horizon correctly. But the
| article doesn't show they did, or even explicitly say they did.
| Conflating two separable issues, it fails to address whether or
| why different presumption rules for each might be desirable.
| chrisjj wrote:
| They are unfortunately conflated in law.
|
| " The Law Commission failed to address the strongest arguments
| against repeal without replacement, .... It ignored the advice
| of the experts they cited who all argued that the focus of
| courts should be on the reliability of computer evidence, ....
| The Law Commission's comments and conclusions revealed that
| they had not understood the nature of computers and complex
| software systems as described in the sources upon which they
| relied.' " https://www.computerweekly.com/opinion/The-cause-of-
| the-Post...
| Terr_ wrote:
| I would like to get those commission members to sit down and
| use a computer program which asks them their name, does a
| fancy "analyzing" animation, and concludes that that they owe
| $1 million in fees and are 99% likely to be the perpetrator
| of an unsolved murder.
|
| I mean, hey, the _computer_ is operating correctly, it can 't
| make a mistake, so give me my money and turn yourself in...
| graemep wrote:
| "It could be read to say that computers should be presumed to
| be executing the instructions they're given reliably, unless
| evidence shows otherwise. It's about malfunction, not
| misapplication."
|
| You are making a technical distinction the law does not make.
| The law relates to the output - to the evidence generated from
| the system as a whole. The presumption is that the logic is
| correct too - which is why it is such a terrible assumption.
| kemitchell wrote:
| Where do statutes or court decisions say that? To my eye,
| none of the authorities quoted in the article do. It's merely
| implied.
| fanf2 wrote:
| Right.
|
| In the specific case of the Post Office miscarriages of
| justice, the system as a whole included Horizon central
| office staff who made manual and sometimes incorrect
| adjustments to the individual Post Office ledgers, and the
| discrepancies were later blamed on the postmasters.
| Arch-TK wrote:
| Hmm, that's genuinely concerning. I would say a more appropriate
| thing to say is that for simple things like audio/video evidence
| (at least before the AI-video era of today) and maybe logs
| computers can be assumed trustworthy, but for everything else
| they should be assumed to be as unreliable as a human witness. At
| the end of the day, a human designed the computer and wrote the
| software for it.
| orangesite wrote:
| We have the economic and mathematic machinery to, in cases where
| uncertainty would adversely affect a defendant, require a
| rigorous statement of proof for the prosecution to win the day.
|
| Bonus: Many opaque systems would have to be aired in an open
| court room to ascertain whether their invariants do, in fact,
| survive scrutiny.
| advael wrote:
| I think there's a huge and fundamental difference between the
| assertion that a computer program has malfunctioned and the
| assertion that a computer program does not accomplish what it
| intended to do at all, that the algorithm is incorrect, or that
| there simply is no known reliable way for any computer program to
| do the thing.
|
| The latter problem is more important, but by lumping this
| together with "malfunction" and giving technologists basically a
| complete pass on the entire hard part, this kind of rule is a
| loophole wide enough to pass a jetliner through
| ciscoriordan wrote:
| Related to that:
| https://www.schneier.com/blog/archives/2009/05/software_prob...
|
| Defense attorneys in a DUI case got their hands on the source
| code for the breathalyzer. It turned out to have terrible
| programming, e.g. calculating new averages by averaging a new
| value with the previous average. The case went all the way to the
| New Jersey Supreme Court, which still found the device to be
| acceptable.
| cochleari_major wrote:
| "Which would cause the first reading to have more weight than
| successive readings", funny enough looks like the opposite of
| what's going on.
| AdamJacobMuller wrote:
| Maybe it averages from newest -> oldest
|
| With software that bad, who can say
| weaksauce wrote:
| yeah the language is slightly ambiguous enough where you
| can't for certain know. first vs newest.
|
| the issue is that taken as a whole the quantization of the
| samples into 8 bins is a much bigger issue along with the
| problem of the no hardware watchdog or hardware malfunction
| alarms. along with the poor testing methodologies too.
| danaris wrote:
| Well, of course, because that statement is deeply incorrect--
| the described mistake would cause the _most recent_ reading
| to have more weight.
|
| If you have a set of readings, say, [0.1, 0.02, 0.3, 0.05,
| 0.08], normally when you average them you would get 0.55--the
| mean of the set.
|
| Calculating the average by "averaging the new reading with
| the previous average" would mean new + old / 2 every time.
| That means that for each reading after the first, your
| "averages" would be: [0.06, 0.18, 0.115, 0.195].
|
| If we add a new reading of 0.01 to each of these, in the
| first case, we would get an average of 0.46, and in the
| second case, 0.1025. As you can see, even taking into account
| the already-very-skewed numbers, the second case biases it
| _much_ further in favor of the new reading (which, in this
| case, is very low compared to the existing readings).
| eastbound wrote:
| For an insurance, I had to perform the average of questions
| for contracts such as "Did it go to court":
|
| 1-Yes
|
| 2-No
|
| 3-Unspecified
|
| Of course the average was around 2.011.
| epgui wrote:
| On the face of it, that's absolutely insane.
| krisoft wrote:
| Idk, hard to guess based on the layman description, but maybe
| what they are describing is a rolling average? That would be
| quite a standard low-pass filter for applications like this.
|
| Hard to get upset over that. What matters is not the signal
| processing but the validation. You take a bunch of people
| with various blood alcohol levels measured by some already
| accepted lab technique and you verify that your new
| measurement technique is measuring within some acceptable
| error bound of that.
| secondcoming wrote:
| That's not the formula for a rolling average though
| aidenn0 wrote:
| An exponential moving average is a form of weighted
| rolling average that would match the plain-text
| description in the complaint and is often used in signal
| processing as an IIR low-pass filter (an unweighted
| rolling average over a window is an FIR filter)
| wnoise wrote:
| It is the formula for an exponentially weighted moving
| average.
|
| Which is not appropriate here...
| hiatus wrote:
| The full opinion can be found here:
| https://law.justia.com/cases/new-jersey/supreme-court/2008/a...
| linsomniac wrote:
| Vonnegut's book _Player_Piano_ has, basically, LLM-driven
| society, but on vacuum tube technology. In one section the court
| says: "We replaced all the vacuum tubes and got the same verdict,
| so we're confident on the judgement."
| tmpz22 wrote:
| My dad and I were watching a TV show where someone received a
| time traveling fax. I remarked that the software on the Fax
| machine may have just had a bug.
|
| He immediately remarked "they're scientists (physicists who sent
| the fax) and it was impossible that they wouldn't have accounted
| for that".
|
| I've been a software engineer for 10 years. He's a well-read
| hard-working blue-collar guy, working as a taxi driver and behind
| a deli most of his career. I just nodded and moved past it.
|
| People want to anthromorphize AI. People want to yield divine
| knowledge to computers. Any sufficiently advanced technology is
| indistinguishable from magic indeed.
| josephg wrote:
| Heh. Ask him if the people at the deli he works at are
| generally competent.
|
| I have a somewhat awful belief about humanity: that about 70%
| of people work jobs they're not very good at. Most of the time
| this is ok, because so many jobs are bullshit anyway. Bad real
| estate agents still sell houses. Grumpy sales people still sell
| product. Social media managers - well, let's not go there. But
| it has some strange consequences - like how most therapists are
| (in one study) less useful for the patient than journaling. Or
| in my opinion, the prevalence of crap software.
|
| Ask about his work. Ask if his coworkers made a fax machine, if
| it would work reliably. If he gets it, you'll know. You'll see
| it in his eyes.
| cryptonector wrote:
| My father is a chemist and he has said things like this to me.
| He has extreme faith in people who wear lab coats.
| pjerem wrote:
| What I find frightening is not only the risk of default but also
| the risk of hacking. It's easy for someone who gain remote access
| to create (or remove) data, records, logs ... on a system.
|
| It's also extremely easy for the computer owner or IT people to
| do the same.
| michael1999 wrote:
| It's nice to see a reasonable proposal. If you are going to
| present criminal evidence based on the output of a computer
| system, it is only reasonable to demand access to a bug tracker,
| the QMS control documents, audits, and a chain of custody. If you
| can't produce that easily, then your evidence shouldn't be worth
| much.
|
| The big win would in this case would be that when the vendor
| conspired to hide bugs from their own tracker, they would have
| been creating criminal liability for their employers. Which
| Fujitsu and their subs richly deserve.
___________________________________________________________________
(page generated 2024-04-16 23:01 UTC)