[HN Gopher] A former pilot on why autonomous vehicles are so risky
___________________________________________________________________
A former pilot on why autonomous vehicles are so risky
Author : obscurette
Score : 57 points
Date : 2023-05-14 15:58 UTC (7 hours ago)
(HTM) web link (spectrum.ieee.org)
(TXT) w3m dump (spectrum.ieee.org)
| mihaaly wrote:
| I have a feeling that she has the eyes of the highly trained
| professional driver (pilot) with good senses and ability to react
| well.
|
| Not something that can be told about the bulk of the drivers on
| the road.
|
| Although I see the point of and agree with the necessity of
| 'crytal clear' situation, knowing what exactly can be expected
| from the assitance, I believe that the average driver should have
| as much help as possible. The roads are pretty unsafe yet,
| because of the human drivers.
| nytesky wrote:
| Did any of you read the article? She acknowledged there are some
| cases where it's working in well defined environments.
|
| But she clearly shows how we are currently in the most dangerous
| phase "car drives except handing off to human drivers in a
| crisis".
|
| " The policy should be that either the computer is driving or you
| are driving. And by driving I mean steering--people do fine with
| regular cruise control. The act of keeping your hands on the
| wheel and guiding the car's lateral motion is enough to keep your
| brain engaged. So, no L3 [full self-driving, but the driver must
| be ready to take the wheel], which is too confusing, and no
| hands-free L2 [partial self-driving]. I am not against the
| passing of control per se, but there should just be two modes of
| operation, with crystal clear feedback about which mode you are
| in."
|
| FSD has been 5 years away for... 15 years now?
| StrangeATractor wrote:
| Fun story, I was in a loaner car from a dealership which had
| lane assist. I was driving in the desert around a corner
| against a cliff so I couldn't see around it very well. Right
| around the corner there was a small boulder in the road which I
| had to swerve to avoid, except the lane assist fought me on it
| since it would cross the painted lines. It wasn't like a near
| death experience or anything, but it really made me think on
| how the systems meant to protect us can be what causes a life
| threatening problem if they aren't designed well. If I had to
| swerve to avoid another rock or a wayward vehicle, compensating
| for my own vehicle's unexpected response to driver input is at
| the bottom of the list of things I'd want.
| GaryNumanVevo wrote:
| Had the same experience while driving through Tahoe in a
| Toyota Camry rental. Immediately turned that "feature" off
| because the last thing I want is my car fighting me if I have
| to make a split second decision.
| m-ee wrote:
| Even without any obstacles a rented jeep Cherokee felt like
| it was constantly trying to kill me on mountain roads. The
| lane assist also turned itself back on every time the car
| started.
|
| Another baffling feature was that the car refused to move
| after stopping at stop sign. Turns out there's a "safety
| feature" to disable the engine if the seatbelt is not
| buckled. Thankfully I was just looking for parking and the
| only consequence was some inconvenienced cars behind me but
| that could get really bad in the wrong situation.
| cassianoleal wrote:
| On my dad's car, the middle back seatbelt has a pretty odd
| behaviour. It doesn't stretch at all if the car is moving,
| but it is allowed to contract. This means:
|
| - if the person sitting in the middle has failed to buckle
| up or at least stretch the belt enough before the car
| started moving, now they can't do it until the car stops
| again;
|
| - whenever the person in that place breathes out, they're
| running the risk of not being able to breathe back in next.
|
| - when the car accelerates, there is a tendency for the
| person to weigh back into the seat. That causes the belt to
| contract.
|
| Repeat the 2 latter points enough, and it becomes
| practically impossible to breathe or even to undo the
| buckle so you get trapped quite literally dying of asphyxia
| and in pain from the belt pressing hard against you.
| PeterisP wrote:
| I recall seeing a video of a car who managed to drive in a
| blocked zone and stop on rails, and eventually got hit by a
| train as it was unable to drive out - arguably because the
| driver opened the door, and afterwards the car
| automatically switched to parking or something.
| DwnVoteHoneyPot wrote:
| An extreme example of this is the Boeing 737 MAX.
| 0zemp2c wrote:
| > FSD has been 5 years away for... 15 years now?
|
| this was only Elon-time hallucinations that made for great
| marketing
|
| you won't get to Mars on Elon-time either, but that doesn't
| mean you won't get there
| croes wrote:
| >you won't get to Mars on Elon-time either, but that doesn't
| mean you won't get there
|
| Doesn't mean you will either.
| EA-3167 wrote:
| It wasn't just Musk, it was Lyft and Uber, Waymo and the rest
| are all still doing it. Lyft's CEO just put out a statement
| saying "Soon now".
| hbarka wrote:
| The greatest trick Musk played was marketing the lane-keeping
| assist feature of Tesla as "Autopilot" or FSD. It is the best
| lane-keeping assist I'll give it that but FSD not in five years
| and not until commodity LIDAR and map-sensing. Mobileye
| deserves credit for jumpstarting Tesla's lane-keeping assist
| early on.
| thedrbrian wrote:
| >FSD has been 5 years away for... 15 years now?
|
| FSD is going to be enabled on HW2 anyday now. Oh look at that
| HW4 just rolled out.
| detaro wrote:
| > _FSD has been 5 years away for... 15 years now?_
|
| the past few years it has been "later this year"
| candiddevmike wrote:
| Shit like this makes you realize how behind the curve NHTSA and
| other government regulators are. Why aren't they putting more
| guardrails around self driving? What do they stand to lose by
| enforcing common sense ideas like the ones in the article?
| Veserv wrote:
| They person interviewed, Missy Cummings, was appointed by the
| Biden administration to be a senior safety advisor to NHTSA.
| Elon Musk directed his fans (who are also largely
| shareholders) to viciously attack her which included sending
| death threats [1]. She was then ordered to recuse herself
| from any Tesla related matters due to a petition driven by
| those same Tesla shareholders.
|
| [1] https://amp.cnn.com/cnn/2021/10/28/cars/tesla-ntsb-
| cummings/...
| philipov wrote:
| What do they stand to lose? Cushy industry jobs.
| jackmott42 wrote:
| Because the USA is a country ruled by large corporations
| and/or wealthy individuals, not by the people. The large
| corporations wants to be free to do self driving experiments,
| so they do it.
| natch wrote:
| I think most people would agree that the middle zone is not
| ideal for most drivers.
|
| However, there may be no other way to someday reach level 5 for
| general driving, other than to live with that middle zone for a
| while (keeping in mind that it's already safer than humans
| driving alone) in order to conduct the training needed to reach
| level 5.
|
| You don't just wish, or regulate, level 5 into existence. You
| need to build toward it by having real systems deployed in the
| real world. Those real systems are not going to be level 5 at
| first or even level 3. This is where we are today, for general
| driving we only have level 2 and some level 3 if that.
|
| Training is needed not only for the systems but also for
| drivers (including other drivers in traditional vehicles, they
| are also being trained unwittingly) and pedestrians, and
| regulators. Everyone is learning here. It's not just an
| overnight, snap of the fingers kind of thing.
| twelve40 wrote:
| This training in the wild sounds scary at first, until i
| remembered that we've done something similar with air travel,
| basically "training" with live people and fatalities. But
| over the years we did reduce those from 40 crashes/mil to 0.1
| crashes per million flights. I would not want to fly on those
| early jets in the 50's, but enough people actually took that
| chance!
| nytesky wrote:
| I believe the article speaks to the lack of data: we don't
| know if L3 is safer. Safer than a drunk driver or a tired
| long haul, maybe. But a typical driver that would also use L3
| responsibly, it may not be safer.
| nytesky wrote:
| One other huge disadvantage of L3. As driver physically drive
| less and less while models learn, their skills will wither
| and when the car gives up control it will be to out of
| practice drivers.
| klyrs wrote:
| > However, there may be no other way to someday reach level 5
| for general driving, other than to live with that middle zone
| for a while (keeping in mind that it's already safer than
| humans driving alone) in order to conduct the training needed
| to reach level 5.
|
| Gotta call bullshit on this. If human-piloted cars were
| bedecked in sensors, recording drivers' inputs and outputs,
| we could collect all the training we could need for L5
| driving. It would take a ton of stringent curation, but
| that's going to be necessary anyway. It's not like Tesla has
| magically learned not to hit emergency vehicles stopped in
| traffic yet.
| ma2rten wrote:
| It seems very ignorant to bet against AI given all the progress
| that have been made in that last year (!).
| pmontra wrote:
| Most people in 1969 would probably say the same about betting
| against humans on Mars well before the end of the 20th
| century.
| htag wrote:
| It seems overly optimistic to expect a breakthrough.
| jstx1 wrote:
| It seems intellectually lazy to assume that the present
| continues reliably into the future. You need a more concrete
| argument than "we've been making progress so far so we'll
| keep making progress", or if you're on the other side of the
| argument, you need something better than "this hasn't
| happened yet so it will never happen".
| kuboble wrote:
| If the progress of neutral networks-based AI has thought me
| anything then I expect more and more applications where it's
| amazingly capable some time, capable most of the time and
| completely fails still relatively often.
|
| I am really skeptical at this point if it can ever master
| something to the point where we can safely trust our lives
| with it.
| WA wrote:
| Two entirely separate fields, both with a similar problem:
| not good enough. Impressive, but makes mistakes and an 80%
| solution is not what you want for self-driving cars.
| ianburrell wrote:
| The progress recently has been with LLM. Great when you want
| to talk to your car or car needs to read street sign. Not
| useful if want to drive the car. It is possible that the
| techniques transfer over but self-driving is pretty different
| domain. My guess is that can share general techniques but the
| LLM breakthrough isn't useful for self-driving.
| natch wrote:
| > we are currently in the most dangerous phase
|
| I did read the article. It's ironic given her anti-Tesla stance
| how Tesla exactly follows what she recommends... "No hands-free
| L2" is pretty much exactly what Tesla urges and enforces its
| drivers to follow. Is the enforcement perfect? No, of course
| not; humans are simultaneously stupid and creative.
| amluto wrote:
| Try reading it again. She thinks that either the driver
| should control the car or the car should control the car.
| Hands on a wheel while the car steers is not a driver
| controlling the car. Tesla does exactly what she objects to.
|
| Wayne did a study several years ago IIRC and determined that
| even their trained, professional drivers being paid for this
| exact task could not reliably, effectively and quickly take
| over driving when a self-driving car made a mistake. It's a
| task that humans are really quite bad at.
|
| I know _I_ am. Steering a car on an average road: not a big
| deal. Taking over on time when a new, borrowed Tesla tries to
| mess up on that same road: not so easy. Also, it takes more
| concentration than just steering the car normally.
| EdwardDiego wrote:
| Apparently data and people who use it to make decisions have
| an anti-Tesla stance.
| robotburrito wrote:
| I can see it. The other day a self driving Cruise car was
| extremely aggressive with me, almost feeling aggressive as I
| tried to cross an intersection on foot. It was stop/starting
| quickly towards me to try and get me to walk faster it feels
| like.
|
| I think that these self driving cars need some kind of feedback
| system for pedestrians and cyclists. Even an emoji face that
| shows facial expressions would help. When I see a real driver at
| an intersection, I feel like we come to some agreement about what
| is going to happen next based on our facial expressions. Or I can
| at least guess what may happen and take an action. With the self
| driving car there is 0 feed back. It's just an empty car that may
| or may not run me over.
|
| Additionally, I also almost got run over by a dude head down
| driving in the Mission the other day, typing on his phone with
| both hands as his he just felt comfy having his Tesla do all the
| work.
|
| At the moment all of this feels so sketchy. I hate my life being
| used for beta testing software lol.
| CamperBob2 wrote:
| You know what's risky? Letting _humans_ drive. 30,000+ deaths a
| year in the US alone.
| vba616 wrote:
| Is that risky?
|
| There are something like 230 million licensed drivers.
|
| So if I was the non-existent "average driver" with no control
| over risk, it seems to me I should expect a fatal accident
| roughly once every 100 lifetimes.
|
| Or every 100 million miles.
|
| I don't perceive the state of the art in autonomous driving as
| anywhere near that kind of reliability.
| piloto_ciego wrote:
| I'm not in favor of how Tesla has used BS marketing to make
| their product seem more capable than it is, nor am I excited
| about how all these companies Over promise and under deliver.
|
| However... driving is one of the most dangerous things we do.
| 1/100 people (roughly) die in a traffic related accident in
| their lives. That's bonkers and unacceptably high.
| vba616 wrote:
| >That's bonkers and unacceptably high.
|
| Well, that's your opinion.
|
| My estimate of my risk of death is 100/100, so 1/100 does
| not seem that high to me.
|
| What I think could legitimately be called "bonkers and
| unacceptably high" is the prevalence of medical errors as a
| cause of death in the US.
|
| "Analyzing medical death rate data over an eight-year
| period, Johns Hopkins patient safety experts have
| calculated that more than 250,000 deaths per year are due
| to medical error in the U.S."
|
| https://hub.jhu.edu/2016/05/03/medical-errors-third-
| leading-...
|
| Since I don't drive drunk, and usually miss rush hour, I do
| think it's reasonable for me to consider going to the
| doctor the most dangerous thing I do.
| sebzim4500 wrote:
| I see a lot of people arguing from their intuition (or experience
| in other fields) that what Tesla is doing with their driving
| assistance features is dangerous. There are millions of Teslas on
| the road, shouldn't the first step be to find out check whether
| they crash more often in practice? If not, then what's the big
| deal?
| mberning wrote:
| " Cummings: There is no evidence of mitigation. At NHTSA we
| couldn't answer the question that you're less likely to get in a
| crash--no data."
|
| Compared to what? I find this very hard to believe. Surely
| forward collision avoidance systems have stopped a number
| accidents over the years. To say that there is no evidence for
| it, no data even, seems to suggest that you might have a data
| problem. Hard to take them seriously when they don't have good
| data, yet maintain their very negative position.
| vba616 wrote:
| The reasoning you are using seems to be popular - I found
| something similar on a page about Waymo.
|
| If an autonomous driving system clearly prevented or would have
| prevented a crash that happened to a human, then that's one
| life saved, is the philosophy. The minimum benefit from
| everyone using the system is at least one life.
|
| But some people do not see it that way. So you are probably at
| cross purposes with them. I'm sure the skeptics understand
| there is evidence for what _you_ think proves mitigation - just
| not for what _they_ think would prove it.
|
| I haven't got what it takes right now to describe at length the
| alternative way of looking at things, plus I could have it (or
| phrase it) wrong. Maybe someone else will comment.
| MBCook wrote:
| This is one of the things that really annoys me. We have _so
| much_ data and case studies of the effects of automations (of
| different levels of sophistication) and user interfaces and how
| those interact with other things including distractions through
| FAA investigations.
|
| To say nothing of other safety critical industries.
|
| We know what works ok. We know what's dangerous. We know what's
| lethal. There are lessons to look at.
|
| And yet Tesla and a lot of people seem content to re-learn the
| same lessons through blood over and over again. Like this is 100%
| incomparable to anything that's ever happened before.
| pigbearpig wrote:
| Sounds like a very good person to have in that position. Being a
| single pilot in an aircraft, she would not only have years of
| experience relying on automation, but also risk-reward decision
| making involving life or death. Self-driving cars are not a
| "restart the container" situation.
| [deleted]
| jameshart wrote:
| 'Former pilot' seems like a fairly irrelevant part of the resume
| of a human-autonomous vehicle interaction researcher with a PhD
| in systems engineering and a series of professorships either side
| of an NHTSA advisory position.
| sigmoid10 wrote:
| For most people "former navy pilot" is more likely to establish
| credibility by association. Not that this kind of journalism is
| worthy of praise, but it is pretty prevalent.
| natch wrote:
| She is pushing a competing technology. They forgot to mention
| that.
| lightedman wrote:
| It's actually fairly-relevant, as airplanes from commercial
| craft to military craft have a TON of autonomous systems.
| vba616 wrote:
| >'Former pilot' seems like a fairly irrelevant part of the
| resume
|
| Does it?
|
| "Tesla's Autopilot AI team drives the future of autonomy of
| current and new generations of vehicles"
| natch wrote:
| Her being a major stockholder and board member of a LIDAR
| company is also important context, but was left out for some
| reason. She is pushing a technology approach Elon has called "a
| fool's errand" which explains a lot of her incongruous
| hostility toward Tesla.
| vba616 wrote:
| How it is important context? The article isn't about LIDAR.
|
| And if LIDAR is the wrong way to go for technical reasons,
| then if she pushes for better autonomous driving technology,
| she is working against her supposed interests.
| natch wrote:
| When they leave things out it's up to you to connect the
| dots, or not. Her association with a LIDAR company suggests
| she is not a neutral voice here when it comes to evaluating
| solutions.
| vba616 wrote:
| The topic you are debating, I can't relate to the
| article. Can you clarify?
| thrill wrote:
| Everything that slows down the selling of Teslas is labeled
| by Elon as a "fool's errand".
| natch wrote:
| Flippant. By that logic he wouldn't work at Twitter or
| SpaceX. The point is she has an agenda to make money off
| her stake in a competitor, so her opinions should be seen
| in that light.
| Veserv wrote:
| What company, Veoneer? The company where she resigned from
| the board and sold all of her shares prior to accepting her
| appointment as a senior safety advisor to NHTSA?
|
| We should believe Elon Musk who has the largest current
| conflict of interest in the world over a woman who has no
| current conflict of interest and deliberately took action to
| terminate any potential conflict of interest she may have had
| (to be fair that is legally required to hold such a post)?
| ghaff wrote:
| On the one hand, she's been writing about this sort of thing
| for years--including at a time when when a lot of people here
| and elsewhere had bought into the fully autonomous vehicles
| next year Koolaid. And there are probably a lot fewer people
| running human factors labs that relate to this topic than there
| are former fighter pilots. That said, being a researcher who
| _also_ has first-hand experience as one of the US Navy 's first
| female fighter pilots does bring additional real world
| credibility--and, yes, probably makes people more likely to
| click on the headline.
| natch wrote:
| They are trying to make digs against auto pilot so it fits into
| that narrative.
| piloto_ciego wrote:
| Other former pilot here... I'd say that HCI and understanding
| how automation works in the user environment is probably more
| useful than her PhD in systems engineering given the context.
|
| Switching/using correct/monitoring autopilot modes is (somehow)
| still a major problem in aviation.
|
| It's not surprising that drivers (who basically get no
| training) have a hard time with this too, nor is it surprising
| that manufacturers are blinded by their own BS (see 737 Max for
| further reference).
|
| Comically the GA airplanes with Garmin units in them were
| wildly better designed than the "professional FMS" boxes
| Honeywell sold. In particular, the flag ship "Honeywell Apex"
| units were terrible in comparison to the touch screen Garmins.
|
| That's wild, because the Honeywell box is fully integrated with
| the airplane and an order of magnitude more expensive.
| asasidh wrote:
| It is difficult to get a man or woman to understand something,
| when their salary depends on them not understanding it.
| EGreg wrote:
| This aphorism can be applied to pretty much the entire AI
| industry right now. Similar to Web3 industry before it, except
| AI is potentially far more dangerous.
|
| https://en.m.wikipedia.org/wiki/Existential_risk_from_artifi...
|
| Many of the top AI experts are now coming out sounding
| warnings. AI Experts have predicted a chance of 10-20% human
| level extinction from AI, have a history of underestimating AI,
| but companies are rushing to the precipice, and it is just the
| early days.
|
| Everyone should listen to this warning:
| https://m.youtube.com/watch?v=xoVJKj8lcNQ
| croes wrote:
| Her job would still exist even with FSD.
|
| But Elon's income on the other hand depends on the success of
| Tesla's FSD.
| JimtheCoder wrote:
| "But Elon's income on the other hand depends on the success
| of Tesla's FSD."
|
| Well, his income and net worth have done pretty well without
| delivering FSD, so maybe not?
| croes wrote:
| But not without promising FSD.
|
| What would happen if he cancels FSD?
| JimtheCoder wrote:
| People would still buy Teslas?
| anonuser123456 wrote:
| It's also difficult to get posters to read articles before
| commenting on them.
| natch wrote:
| The article did not disclose her conflict of interest.
| Perhaps the poster was aware of it.
| [deleted]
| 0zemp2c wrote:
| [flagged]
| Jtsummers wrote:
| > This woman appears to be in her fifties...claiming autonomous
| driving will not happen in her lifetime
|
| She doesn't claim that. You claim she claimed that. Why make
| things up?
|
| In the interview she talks about existing autonomous driving
| applications that are operating _today_. How can you go from
| reading that part of her statements to your gross
| misinterpretation?
| [deleted]
| yawpitch wrote:
| I'm pretty content that "the day when AI in cars can handle ALL
| conditions on the road, ALL OF THE TIME" (emphasis mine) won't
| happen in your lifetime, or indeed the lifetime of any
| currently gestating fetus.
| joshribakoff wrote:
| It is also true that humans in general cannot drive [without
| crashing].
| Espressosaurus wrote:
| With suitable training they can.
|
| The insurance industry expects the average insured driver
| to file a claim every _18 years_. That 's covering all
| conditions, all of the time. No "hand off to someone else
| when I get scared" because humans can't do that.
|
| That's honestly pretty good.
|
| Wake me when autonomous driving can get within a factor of
| two of that.
| croes wrote:
| And if the have an accident they are liable.
|
| Who is if a fully autonomous driving car has an accident.
| jibe wrote:
| Probably insurance companies, like in human crashes.
| croes wrote:
| Including fatal accidents?
|
| What about software or sensor errors?
| watwut wrote:
| They provably can.
| 0zemp2c wrote:
| human can't handle "ALL conditions ALL OF THE TIME" either
|
| this is a false goal that doesn't exist in the real world
|
| cars, planes, trains, all automation of any kind, all the
| code you ever wrote, nuclear power plants...all this and more
| operates with some fallibility that is deemed acceptable
| given the benefits
|
| no one is excusing AV accidents that seem to defy common
| sense (randomly driving into barriers etc)...but humans fail
| in these ways too, every day
| joshribakoff wrote:
| In case you're living under a rock, it already happened. Are
| they perfect and driving everywhere in every city? No... but
| here in SF it is not unusual to see cars with no one in them,
| they're driving all over the city, at all times, in parking
| lots, through school zones, etc
|
| Also you're trying to discredit a well credentialed individual
| (who worked in automotive safety regulation) based upon
| age/gender and other irrelevant factors. I feel the need to
| call that out as toxic.
| AQuantized wrote:
| I don't think the poster is trying to discredit her based on
| age, just contextualizing her comment about it not happening
| in her lifetime by estimating what period that's likely to
| refer to.
|
| The comment about her appeal to authority on the basis of
| being a fighter pilot is also more a counterpoint to
| crediting her additionally on that basis above any non-
| expert, since that skillset isn't actually particularly
| related to autonomous driving.
| hattmall wrote:
| There have been self driving vehicles in well constrained
| areas since the 90s or even earlier. They have been capable
| of complex route navigation, obstacle avoidance and
| responding to a variety of conditions. The technology hasn't
| really matured an order of magnitude past the automated
| vehicles that have been moving things around in factories for
| decades. The technology improvement has come in computing
| power and sensor tech where now it's just cheaper and easier
| to include this in more vehicles. There need to be major
| leaps before this can really become widespread. I would say
| at best we are at the Commodore 64 era of self driving cars.
| I think it's going to take even longer to get to the iphone
| era considering the result of failure could be death.
| lumost wrote:
| The rollout beyond sf has been ... slow. Which begs the
| question, how autonomous are autonomous vehicles? Is this a
| technology or an economics problem, or both?
| 0zemp2c wrote:
| shrug, I'm older than her, so the age thing doesn't resonate
| with me
|
| I appreciate her credentials, but I don't see why her
| statements are somehow unassailable
|
| and I stick by my point that her history as a fighter pilot
| has no bearing
|
| not sure where your "gender" point comes from? other than
| using the pronoun, I made no claims based on gender
| aatd86 wrote:
| I agree with the fact that "experts" are quite often wrong
| and it's too easy to be derailed by appeal to authority.
|
| On the other hand she has a point. As long as the tech
| won't be 99.99xxxx safe, it will introduce moral hazard
| (she calls it "risk-homeostasis" but I don't think that
| kind of neologism is particularly expressive).
|
| At a meta-level, coining some new term could make someone
| raise an eyebrow. Perhaps she just didn't know about it.
___________________________________________________________________
(page generated 2023-05-14 23:01 UTC)