[HN Gopher] Rivian software update bricks infotainment system, f...
___________________________________________________________________
Rivian software update bricks infotainment system, fix not obvious
Author : carlivar
Score : 119 points
Date : 2023-11-14 17:22 UTC (5 hours ago)
(HTM) web link (electrek.co)
(TXT) w3m dump (electrek.co)
| eschneider wrote:
| This is a bit of a nightmare scenario and why when remote
| updating, you always test update to your own fleet first. Always.
| kevin_nisbet wrote:
| Yes. And also things like rolling out the update in batches,
| and then also things like golden images, where if there are two
| crashes or failures in the first 24 hours of the update, change
| to the last known good software version.
| toddmorey wrote:
| It sounds like it was tested on their own fleet but they
| accidentally pushed the wrong bits when deploying the update
| more widely out to customers.
| carlivar wrote:
| Maybe they should have an additional phase between test
| deploy and customers such as "employee personal vehicles".
| eschneider wrote:
| The usual "best practice" thing for IoT deploys, is to deploy
| to "your" devices, what for everyone to go green, then allow
| that build to deploy more widely. In a well-functioning
| system, it shouldn't be possible to swap bits between those
| stages.
|
| But who knows what these guys were doing. :/
| bink wrote:
| Reading between the lines of their public comments it
| sounds like they did run a full test through their test
| fleet, their employees, and then were rolling out to
| customers when the promotion process was "fat fingered".
| Maybe someone accidentally promoted the wrong release
| version.
| baz00 wrote:
| This is why I have a Dumbcar connected to a Smartphone via
| bluetooth.
| barbazoo wrote:
| That's why _I_ have a Dumbcar connected to a smartphone via FM
| for audio only :)
| thatguy0900 wrote:
| I've tried that before, but it sounded terrible. What dongle
| do you use?
| barbazoo wrote:
| I mean, totally, it doesn't sound great but that's all I
| can do, my car doesn't have bluetooth :)
| BobaFloutist wrote:
| Does it not have an aux port? USB C to Aux cables are
| pretty cheap and simple.
| barbazoo wrote:
| I know I know but then there's a cable from where the
| phone is to the middle console where the aux port is. But
| true, it'd for sure be better sound quality.
| ska wrote:
| > I know I know but then there's a cable from where the
| phone is to the middle console where the aux port is.
|
| This doesn't seem unmanageable, no?
| barbazoo wrote:
| Not at all but not the best option for people that care
| about convenience over quality.
| bonton89 wrote:
| These all sounded like crap to me as well. I've heard
| earlier models had much higher radio power (that was
| possibly illegal) and worked better. They were an archaic
| tech by the time I was buying the adapters off amazon.
| toddmorey wrote:
| Just a counterpoint: my dumb car has been undrivable way more
| often than my electric car.
|
| They never deployed bad software updates but they sure have
| designed & deployed bad fuel pumps.
|
| In some ways it's all engineering and quality control.
| baz00 wrote:
| I can go nearly anywhere and replace the fuel pump though.
| qmarchi wrote:
| It's crazy to me that this is possible in the first place.
| Standard practice is to have a fleet of test vehicles that are
| effectively production except in an early release group.
|
| Or, you know, having an A/B boot partition scheme with a
| watchdog. Things that have been around for decades at this point.
|
| Disclaimer: Former Googler, Worked closely with Automotive.
| mlyle wrote:
| The code went through early release tests successfully; the
| problem came with how it was more broadly released.
|
| They should have had further staging of the rollout
| (randomizing when it is offered to users).
| MichaelZuo wrote:
| The 'early release tests' weren't testing an identical copy
| of the actual update?
| DannyBee wrote:
| It's probably closer to:
|
| The test vehicles accept test/prod signed versions
|
| Regular vehicles only accept prod signed versions
|
| They are otherwise identical.
|
| The test vehicles were sent test signed versions
|
| The prod vehicles were sent the exact same update, signed
| with test.
|
| This would not be uncommon since the test vehicles probably
| occasionally run test releases for debugging.
|
| Further, the update is probably multiple signed pieces, and
| the only part accidentally signed with test was likely
| infotainment software.
|
| Or something like this.
|
| It's hard to believe they wouldn't test sending badly
| signed updates, so i have to imagine it's a particularly
| weird badly signed update.
|
| In other words, i would not assume they are idiots.
| AlotOfReading wrote:
| It'd be pretty silly to implement an OTA scheme that
| didn't check signatures before installing updates. That
| would mean any random attacker could soft-brick the
| module by sending an invalid image, which a development
| image should be to a production vehicle.
|
| You could get this situation if the application code
| accepted signatures the bootloader does not though. I can
| imagine that accidentally occurring.
| whalesalad wrote:
| A/B partitions tends to solve that. You will only switch to
| the new partition when the update is 100% verified installed.
| If it doesn't complete in an atomic manner, your device will
| just boot into the previous healthy partition.
| AlotOfReading wrote:
| A/B gets complicated in the real world. BL1 may not support
| A/B for example, so to implement A/B bootloaders you may
| need a shim that can read/write NVM to handle that. Your
| HSM may not have slots for multiple keys to have different
| signatures, so upgrading one may trample the other if your
| update code doesn't check that.
|
| Lots of ways to screw this up, especially in automotive
| where you're likely to be dealing with TI and their
| (in)secure boot.
|
| I've solved this problem god only knows how many times now
| and I've rarely found an automotive board that doesn't
| introduce fun, new edge cases. OTA can't exceed x kilobytes
| of memory, the processor isn't fast enough to verify
| signatures and write the image in < x seconds, can't write
| the image to flash unless the signature is verified, but
| the image doesn't fit in RAM, the server delivering the
| update is 3+ networks away from the device receiving the
| update, etc.
| Dalewyn wrote:
| With all due respect, that all sounds like Programmer
| Induced Problems(tm).
|
| Cars are a long solved problem, being around for over a
| century. Telephony and computing hardware and
| infrastructure today are in the realm of the ludicrously
| good compared to even just a few decades ago, even if we
| consider bottom of the barrel worst case scenarios. If
| software somehow can't work a solved problem using
| ludicrously good hardware, the programmers (and their
| managers) are the problem.
| whalesalad wrote:
| I agree here. A partition is just a partition. It's
| taking one disk and abstracting it into two. This is not
| hard.
|
| The rovers on Mars, the Voyager, these problems have been
| solved for a long time. The compute in a Tesla these days
| can probably run Crysis.
|
| You can do this OTA to a Raspberry Pi running Nerves via
| remote SSH and it works really well. The Nerves runtime
| utilizes A/B partitions for OTA updates.
| mypalmike wrote:
| I once worked at a startup that sold a very expensive
| "enterprise" network appliance that didn't have
| partitioning. I had worked on network appliances before
| which did have this capability, so I asked the VP of
| engineering about it. He said it just wasn't a priority
| given all the other work that needed to be done. I
| wouldn't be surprised if Rivian had the same startup
| mentality that might lead to such a situation.
| AlotOfReading wrote:
| Let's use two of the examples I gave above. How would you
| go about modifying the silicon to support new features in
| the boot rom, and how do you get around the pigeonhole
| principle when that silicon vendor doesn't ship enough
| bits of OTP memory to store multiple keys? M-x butterfly
| doesn't quite get there, but maybe there's another Emacs
| command I could use?
| whalesalad wrote:
| You choose different silicon? You get a different vendor?
| You build your own boards and chips?
|
| For a vehicle that costs $100k+ it shouldn't be hard to
| double or triple the budget for onboard compute
| considering it is vital to the operation of the entire
| vehicle.
| AlotOfReading wrote:
| Note that I never claimed the automotive world has
| functional people processes. Functional people processes
| make a lot of technical issues much easier, but they're
| usually off the table in traditional manufacturing. The
| security team insists on x requirements for security
| reasons. Hardware team insists on this chip because it's
| the only one that makes the budget work.
|
| A shockingly large part of my job is telling both that
| one team won't be getting what they want and to work it
| out among themselves. Rinse and repeat between dozens of
| boards because the relevant teams don't talk to each
| other and none of them read the "design requirements to
| ensure we don't have to tell you no" doc either. One time
| they didn't even tell us there was a board until the end
| of December, when delivery was scheduled for Feb.
| whalesalad wrote:
| I live in Detroit so what you're saying does not surprise
| me in the least bit. I have some friends in automotive at
| some of the big 3, and suppliers, and I've heard some
| really terrible stories.
| mlyle wrote:
| If the comment I replied to _originally_ contained a
| mention of A /B partitions, I missed it.
| hef19898 wrote:
| I am still not sure why I would update software on car, a
| piece of hardware that, IMHO, shoupd be able to run air
| gapped 24/7. Exceptions: recurring bugs, GPS maps and
| security updates. All of which can be done either during
| service (preferred, if they brick it, they are liable) or by
| plugging in something. OTA updates just seem completely
| pointless...
|
| Edit: Also, why the heck isn't the entertainment system
| completely air gapped from the software running the car?
| enragedcacti wrote:
| Rivian consistently ships a lot of new features and
| improvements, you can see the changelogs here [1]. I think
| you can pretty fairly critique a lot of them with: They are
| just solving a problem that they created by making it too
| techy, or they are shipping stuff they should have
| completed before releasing the product. I do think its hard
| to argue that the updates aren't adding anything of value
| though. There's convenience stuff like pet mode or bird's
| eye camera view that were added after release, but there
| are also things like new driving modes (soft sand and snow)
| or improved DC charging curves and smarter battery
| conditioning that legitimately improve the quality of the
| product as a vehicle.
|
| > Edit: Also, why the heck isn't the entertainment system
| completely air gapped from the software running the car?
|
| As for this, the entertainment system can control basically
| every feature of the car and is often the primary or only
| way to accomplish certain things. Even in much much dumber
| cars the infotainment is still part of the CAN bus and is
| able to interact with the rest of the vehicle.
|
| https://rivian.software/category/public/
| hef19898 wrote:
| Funny, our 2020 MY Jaguar controls car functions from the
| digital screen in front of the driver, the middle console
| screens only control AC, entertainment, phone, navigation
| and other non-car related stuff. No idea how the
| architecture looks behind all that so. But seriously,
| even if on the same bus, just don't the media player,
| radio and connected phone access to the systems actually
| running the car from engine to brakes. And please,
| please, finish developing the embedded software running
| on car before shipping said car. Then it can be air
| gapped, if not it requires OTA and _internet_ access,
| raising all kinds of security issues...
| CamperBob2 wrote:
| _And please, please, finish developing the embedded
| software running on car before shipping said car._
|
| Sorry, but things no longer work that way, and never will
| again. This is a _good_ thing, as long as processes are
| improved to avoid situations like this one.
| refulgentis wrote:
| Rollouts don't solve problems, they limit who they effect.
| mlyle wrote:
| Is not reducing the effective cost of a bad update by 10x
| or more worthwhile?
|
| Sure, but if you are rolling out to 1% of users per hour,
| you detect the problem in a couple of hours and much fewer
| than 2% of users will have applied the update. This is a
| relatively small support problem.
|
| While if you roll out to everyone at once, you'll detect
| the problem sooner (within an hour) but have 10x as many
| affected.
| michaelt wrote:
| To me it's all-too-understandable how this is possible.
|
| Maybe they've got a test fleet, but it accepts code signed with
| the test build key.
|
| Maybe they've got a watchdog timer, but it doesn't get
| configured until later in the boot process.
|
| Maybe they've got A/B boot partitions, but trouble counting
| their boot attempts - maybe they don't have any writable
| storage that early in the boot process.
|
| I wouldn't be surprised if, as a newer company, they'd made a
| 'Minimum Viable Product' secure boot setup & release procedure,
| and the auto-fallback and fat-finger-protection were waiting to
| get to the top of the backlog.
| psunavy03 wrote:
| Exhibit A of why a Minimum Viable Product still needs a
| proper Definition of Done which includes quality standards.
| qmarchi wrote:
| So, using Polestar as a reference as it's both a vehicle that
| I've worked on, and one that I personally drive.
|
| > Maybe they've got a test fleet, but it accepts code signed
| with the test build key.
|
| Polestar solves this by only delivering signed updates to
| their vehicles. The vehicle headunit will refuse to flash a
| partition that isn't signed by the private key held by
| Polestar. Pulls double duty to prevent someone from flashing
| a malicious update, as well as corruption detection.
|
| > Maybe they've got a watchdog timer, but it doesn't get
| configured until later in the boot process.
|
| Based on what the Rivian reports are showing (Speedometer,
| cameras, safety systems are working), they likely are running
| their infotainment as a "virtual machine" within their
| systems. Again, something that Polestar does.
|
| Implementation of a watchdog with a "sub-system" like this is
| relatively braindead simple.
|
| > Maybe they've got A/B boot partitions, but trouble counting
| their boot attempts - maybe they don't have any writable
| storage that early in the boot process.
|
| Generally, A/B partitioning is part of the bootloader, the
| first program that executes after the reset (on many modern
| processors) pin is released. This also leads to reboot
| counters and such being stored as part of the NVRAM that is
| available at boot.
|
| Opinion: Maybe I'm biased, but maybe if you can't develop
| something yourself, there's reason for you to get an off the
| shelf option that handles a lot of these things.
|
| Disclaimer: Former Googler, Worked closely with Automotive.
| refulgentis wrote:
| Opinion: I'm a little confused as to how you're confused as
| to how test might not match prod sometimes.
|
| Observation: "[if you write buggy software], there's reason
| for you to get an off the shelf option"
|
| Question: Are you saying if they used Android Automotive
| this could never have happened?
|
| Reference: similar event for Android, last week:
| https://linustechtips.com/topic/1538248-pixel-phones-
| using-m...
|
| Disclaimer: Former Googler, did not Work closely with
| Automotive, Worked closely with Android.
| qmarchi wrote:
| Answer: To clarify, more that a company should stick to
| their core competencies unless there's a drastic need or
| opening in the market that could be filled (and to build
| a new competency).
|
| In this particular case, there's nothing particularly
| unique that Rivian is doing with their Infotainment
| system that couldn't already be handled by an incumbent
| in the space, (Android Automotive, QNX, etc.) especially
| given how modular the systems themselves are.
|
| As State Farm says, "We know a thing or two because we've
| seen a thing or two".
| hermitdev wrote:
| > As State Farm says, "We know a thing or two because
| we've seen a thing or two".
|
| That's Farmers, delivered by J.K. Simmons, not State
| Farm. State Farm is Jake, in his khakis.
| Gud wrote:
| To be honest, I don't think Polestar set a very high bar
| for software quality. I am currently renting a Polestar 2
| from Hertz, and sometimes the HUD doesn't work(it's 50/50
| if it will turn on). That means, I don't see speed, battery
| charge, etc, while driving. Infotainment system is working
| though.
| qmarchi wrote:
| You're definitely correct in the fact that Polestar isn't
| the highest in software quality, but it was the examples
| of what they did right that I wanted to focus on.
|
| They're garbage when it comes to their mobile app and
| some of the controls on the infotainment system.
|
| > sometimes the HUD doesn't work(it's 50/50 if it will
| turn on).
|
| You should definitely reach out to Hertz and ask for a
| car swap. Sounds like there's a bad connection between
| the display and the IHU. Both screens are operated by the
| same system, so it's unlikely to be a software failure.
| BlueTemplar wrote:
| Sounds ridiculous. How is that even road legal ?!?
| FireBeyond wrote:
| Teslas occasionally need to reboot / hard reset their
| software too, when driving no less, and during that
| period all that information, and most of the controls,
| are unavailable (like windshield wipers, etc.)
| xyst wrote:
| When a car company is losing money on every car sale. C level
| execs going to cut corners
| dewski wrote:
| This is a bad take.
| worik wrote:
| Why?
| xyst wrote:
| Rivian layoffs earlier this year [2] combined with reports
| of $33K loss per sale [1]. Rivian is hemorrhaging money
| right now.
|
| RVN IPO'd at $150/share. Now it's trading at $16/share.
|
| All of these indicators of poor leadership to me. No
| sustainability. Burning cash. Poor company outlook. Poor
| products.
|
| [1] https://tfltruck.com/2023/10/rivian-financial-results-
| losses...
|
| [2] https://www.theverge.com/2023/2/1/23581642/rivian-
| layoff-ev-...
| OneLeggedCat wrote:
| This is an inadequate comment.
| worik wrote:
| What amazes me is that any grown up person thinks it is a good
| idea to update vehicles as if they were telephones
|
| Owners should have to bring the vehicle into a shop to have
| changes made, and they should be very rare.
|
| This lazy, control freakery of the worst kind
|
| Something very bad is going on happen and people will die
| before we realize that it is a stupid dangerous practice
| qmarchi wrote:
| I understand the sentiment, but think about the alternatives.
|
| There are a few different kinds of updates that can be
| applied, each with their own protective layers.
|
| Infotainment updates, like what happened to Rivian aren't
| that dangerous. You lose "convienience features" like maps,
| air con, etc, but generally nothing that could kill you or
| someone else.
|
| Then there's system updates, which is where danger noodle
| things happen. Automotive manufacturers are significantly
| more risk averse to updating these components, and generally,
| if _anything_ within the system looks wonky, it's an
| immediate revert.
|
| If I, as a Polestar owner, wanted to get an update for my
| vehicle, the nearest service center is 1.5h away. If I lived
| in Montana (United States), it would be realistically
| impossible for me to update my car. Thus, if we want to
| enable competition within the markets, we shouldn't have
| regulations that force a new manufacturer to have a global
| network just to add CarPlay to a screen.
| neoromantique wrote:
| >Infotainment updates, like what happened to Rivian aren't
| that dangerous. You lose "convienience features" like maps,
| air con, etc, but generally nothing that could kill you or
| someone else.
|
| Also speedometer, which is hardly a convenience feature.
| fyrn_ wrote:
| The dashboard panel is working, at least according to
| Twitter updates. "Only" affects the infotainment console.
| vore wrote:
| As the update only affects infotainment and not critical
| systems, it seems like a reasonable tradeoff to me. Just
| because a car can fail in ways that kill people doesn't mean
| all parts of a car are equally critical.
| windexh8er wrote:
| This isn't true. If you look at the release notes for any
| of Rivian's updates they all include vehicle related
| firmware changes. This is not simply infotainment.
|
| Beyond that "infotainment" includes driver critical
| information - like the speedometer which, for many
| affected, means there's no working driver screen.
| Daneel_ wrote:
| The article states that the speedometer is unaffected:
|
| "Speedometer, charging, backup cameras, locks, lights,
| wipers, and turn signals are all still functional with
| the 2023.42 error."
| bradleyjg wrote:
| The art of shipping software--like on a disk, where once it's
| out the door, it's out the door and you may never get another
| shot--is dead or dying. Even in some embedded areas of the
| industry now.
| LastMuel wrote:
| On the other hand, we update irreplaceable spacecraft
| billions of miles away with new software.
|
| It should be fine to push software updates out, as long as
| the correct safety and fallback procedures are in place. It
| simply has to be designed to handle failure and procedures
| need to be in place to mitigate risks.
|
| It sounds like that wasn't the case here. Also, why wouldn't
| you have a small initial release pool when you have such a
| large potential for disruption?
| DannyBee wrote:
| Rivian does have a test fleet, and they test it for weeks
| before releasing. This particular issue is because they
| apparently distributed the firmware signed with the wrong cert.
|
| Not a bug in the software itself.
|
| That is independent of testing the software, but still a
| distribution issue.
| jandrese wrote:
| Yeah, but how did the vehicle not just reject the wrong cert
| and refuse to flash the update?
| DannyBee wrote:
| The firmware is probably not just a signed package but
| signed binaries in the package as well. One is probably
| signed with the wrong cert.
|
| This would not cause the updater to fail unless it verified
| the certs of all the binaries in the package, which most
| don't
| mytailorisrich wrote:
| My 2c based on your comment:
|
| * " _signed with the wrong cert_ " should mean the software
| package is rejected before it it is installed.
|
| * software upgrades are tricky and there should be at least 2
| versions available so that fallback to the previous is
| possible and automatic in case of issues.
| DannyBee wrote:
| The software package probably is signed right but contains
| multiple signed binaries of which one is signed wrong.
|
| Or is multiple signed packages and one is wrong.
|
| Or the test cars accept prod and test certs.
|
| Or some combo of the above.
|
| There are lots of ways this could have broken that doesn't
| amount to rivian not being able to write software
| adolph wrote:
| _The vehicles are drivable but software and displays go black. It
| appears that the 2023.42 software update hangs at 90% on the
| vehicle screen or 50% on the app screen and then the vehicle
| screens black out. All systems appear to still work except for
| the displays._
|
| This is what I do with my Prius to get a comfortably distraction-
| free driving environment. Sounds like a feature not a bug.
| altairprime wrote:
| Technically, the NTSB could order an immediate recall for all
| Rivian vehicles due to this issue, as the disabled defroster
| controls are a critical safety issue in cold and/or humid
| environments. Tesla was forced to issue a recall notice over
| the controls being buried in a menu; Rivian's "defroster
| unavailable during driving due to manufacturer error" is far
| worse -- especially given the mass and torque of their
| vehicles, relative to unarmored road users.
| sturza wrote:
| Instrument cluster display going black is a functional
| safety/QM issue. No blinker, transmission direction, speed etc
| confirmations.
| bri3d wrote:
| It looks like they correctly isolated the safety critical
| components on the instrument cluster and they are still
| functional without infotainment: https://twitter.com/RivianSo
| ftware/status/172443804967573962...
| wannacboatmovie wrote:
| Interesting to note that Ford's approach of updating software is
| far more conservative and car-like. It can be done fully offline
| via USB, but requests that you kindly upload the log files
| written to the memory stick back to them when complete, in the
| instructions as a necessary step. Presumably so they can track
| and stop incidents like this before they happen fleet-wide.
|
| Rivian seems more like a "ship it and we'll fix it in the next
| sprint!" company.
|
| How do other manufacturers handle updates?
| sturza wrote:
| A/B partitions
| barryrandall wrote:
| The last time I built something like that, it used partition
| 1 for the current version, 1 for the last version, 1 with the
| as-shipped version, and 1 that could restore A or B from the
| internet or USB.
| post_break wrote:
| Fords approach is flawed however. You can still update sync
| with a bad update and bork it over usb. Ask me how I know.
| r00fus wrote:
| Pray tell, how painful was your discovery?
| thrill wrote:
| Inexcusable, really.
| cs702 wrote:
| It's easy to underestimate how hard and expensive it is to build,
| deploy, and remotely upgrade software that runs reliably on a
| fleet of diverse cars (different models, different years,
| slightly different components from batch to batch, etc.). It
| makes updating a mobile phone OS look trivial in comparison.
|
| So far, only Tesla seems to be able to update car software
| remotely, regularly and reliably. I'm certain it's neither easy
| nor cheap.
|
| All things considered, _physical buttons and dials are probably
| easier and cheaper_ , because they don't require software
| updates!
| Aurornis wrote:
| > (different models, different years, slightly different
| components from batch to batch, etc.). It makes updating a
| mobile phone OS look trivial in comparison.
|
| Not really. Vehicle computers aren't vastly different on every
| model year and every trim level or option package. These parts
| are standardized, tested, and carried across model years.
|
| Even with changes, the teams would be expected to have the
| different variants in their development and test cycles. The
| 2020, 2021, and 2022 model infotainment systems likely share a
| lot more in common than an iPhone 13, iPhone 14, and iPhone 15
| with all of the non-Pro, Pro, and Max variants.
| wannacboatmovie wrote:
| This isn't a bunch of Windows PCs home-built from a hodgepodge
| of components.
|
| They designed, built, and shipped all the hardware. There is
| _ABSOLUTELY NO_ excuse for not having a database of the exact
| hardware configs by serial number. They have the ability to
| test every single shipped configuration.
|
| If they don't, they have already failed as a car company.
| AlotOfReading wrote:
| I guarantee they have a database with the hardware configs.
| It's required by NHTSA to do recalls and notices. They'll
| undoubtedly be using that to inform the right people to come
| in.
|
| The update servers almost certainly don't talk to that system
| though.
| xgbi wrote:
| From what I read somewhere, Tesla was able to do that because
| they have remote ssh capability.
|
| In at least one instance, they fixed the cars manually by
| running a massive remote command on all cars after a messed up
| update:
| https://lobste.rs/s/v42zil/former_tesla_employee_ssh_d_as_ma...
|
| I wouldn't call that very reliable , but they indeed do it
| regularly
| SoftTalker wrote:
| It sounds like, in this case, the updates clobbered the ssh
| authorized keys (or equivalent in their system) and so now
| they cannot access the cars remotely. So they are going to
| have to go into the shop and have the authorized keys
| restored.
| FireBeyond wrote:
| And it's not like they'd ever abuse that ability, like when
| someone pokes around in their car and discovers references to
| a new unannounced model, and then Tesla reaches in, force
| downgrades the vehicle to older software with no references,
| and then disables the ethernet port on the vehicle, and for a
| final fuck you disables its ability to ever get another
| update.
|
| They'd never do that, except when they did do that.
| wil421 wrote:
| > So far, only Tesla seems to be able to update car software
| remotely, regularly and reliably. I'm certain it's neither easy
| nor cheap.
|
| My Jeep Grand Cherokee has OTA for over 5+ years. BMW has been
| doing it since 2018.
|
| I'm almost positive a family member had it with GMC on star
| back in the late 2000s.
| willio58 wrote:
| I don't think the Jeep or BMW infotainment systems are nearly
| as fleshed out or complex as Rivian's, especially not
| Tesla's. Maybe I'm wrong!
| wil421 wrote:
| That's a huge plus for me. CarPlay or nothing. BMW is
| becoming closer to a Tesla like screen. GM is supposed to
| drop CarPlay in favor of whatever they are doing on their
| EVs.
|
| I don't want my ICE/EV to become a SaaS app where I'm
| paying $500 a year to use my own car.
| vel0city wrote:
| > GM is supposed to drop CarPlay in favor of whatever
| they are doing on their EVs
|
| IIRC its not just their EVs its all cars.
| duped wrote:
| Updating software is orthogonal to the complexity of the
| software application being updated, unless you have
| horribly designed your architecture. I know, because I've
| made that mistake.
| phpisthebest wrote:
| Well then we need to ask why is their infotainment systems
| so complex? and does it need to be?
|
| I want my infotainment systems go connect to Android Auto.
| That is is.
|
| Make it do that, and only that.
|
| This drive to make EV's as complex as possible is one of
| the reason i am not planning on buying one
|
| EV's are suppose to be SIMPLER than ICE. Make me a Simple
| Car with simple controls, and just replace the ICE with a
| battery and Electric Motor, give me an app for my Phone
| that can do the Charging Trip Calculators and interface
| with other systems.
|
| I do not want a compplex SaaS app on wheels
| cobalt wrote:
| android auto is just a different brand of infotainment
| system
| worik wrote:
| And if it fails it is inconvenient
| bdamm wrote:
| Well, good news, Bollinger has made your product!
| etchalon wrote:
| My Volvo XC90 gets regular OTA updates without issue, and so
| did my Land Rover Discovery before it.
| bri3d wrote:
| > All things considered, physical buttons and dials are
| probably easier and cheaper, because they don't require
| software updates!
|
| Almost all automotive control modules have firmware, whether
| that firmware is parsing touchscreen inputs or a rotary
| encoder.
| NotYourLawyer wrote:
| Well sure, but the rotary encoder can't get moved to a
| different menu tree by a software update, and I can use it
| without taking my eyes off the road. I know which I prefer.
| dylan604 wrote:
| if (cpu == A) do code
|
| else if (cpu == B) do other code
|
| They invited the multiple combination vampire into their house.
| They know what devices are being used. If you don't want a
| dedicated update per piece of equipment, it'll be a large
| binary with lots of branching. Saying they don't know what
| device is where is just lazy. Ask the device what it is, and
| have a branch for it. If the device IDs itself as something
| unknown, don't do anything.
| tech_ken wrote:
| > All things considered, physical buttons and dials are
| probably easier and cheaper, because they don't require
| software updates!
|
| If it ain't broke it's ripe for disruption
| matrss wrote:
| > All things considered, physical buttons and dials are
| probably easier and cheaper, because they don't require
| software updates!
|
| I am pretty sure there is a market for a dumb modern car, but
| no one is building it. I am thinking of an electric car without
| anything "smart" in it. Modern safety features can stay, if
| they work completely self contained and without requiring an
| external connection ever over the lifespan of the car.
| NotYourLawyer wrote:
| I'd buy that car _today._
| VyseofArcadia wrote:
| Forget updates entirely. My car is one of the few places I
| expect to get software that works the first time.
|
| If you absolutely must have updates, then at least not OTA
| updates. Have them done at the dealership or service center so
| any issues can be dealt with immediately.
|
| Come on, is this engineering or hacking? This is a car, not a
| CRUD app. Get. It. Right.
| w0m wrote:
| random new features via OTA updates was one of the deciding
| factors when i bought my car ... :)
|
| I also mostly WFH so... yea. lol.
| xyst wrote:
| Tech junk shouldn't go in cars, period. Cars shouldn't be as
| pervasive and prevalent in society (at least in USA). Yet here
| we are. Car manufacturers have spent an insane amount of money
| over decades to get to this point (buying legislators, forcing
| highway infra, subsidies, profit driven strategy over
| sustainability)
| FireBeyond wrote:
| > So far, only Tesla seems to be able to update car software
| remotely, regularly and reliably. I'm certain it's neither easy
| nor cheap.
|
| Tesla, whose computer systems quite regularly need to be hard
| rebooted _while the car is driving_? That Tesla?
| code_runner wrote:
| I had to do this once or twice (its very very infrequent in
| my experience) and one time it was genuinely terrifying, as I
| had lost blinkers etc where a few interstates all intersect
| and merge etc.
|
| I still do love the car though.... but a very sketchy moment
| that I shouldn't have brought on myself while driving in that
| situation.
| samsquire wrote:
| This is actually a topic that I think about from time to time:
| how to do aggressive changes to software while they are running.
| In Ruby world you have monkeypatching. And Linux kernel has
| livepatching.
|
| For example, if you have a distributed system and you want to
| upgrade a component that every caller uses: you have a large
| exercise on your hands where you might have to roll out a change
| over time and then clean up your incremental branches where you
| have to handle two control flow paths through the code. It
| reminds me of Google's protobuf required field discussions.
|
| It reminds me of repository-per-microservice and a Java library
| that other microservices use and updating a dependency and having
| to deploy the change to every service.
|
| It's like trying to change wheels on a car while the car is
| moving or refueling a jet in flight.
|
| Unison lang is trying to solve this problem I think, by allowing
| multiple versions of a function to be available.
|
| https://www.unison-lang.org/
|
| Migrations in databases are painful too.
|
| One solution I've thought of which is probably overengineered is
| that API call sites are an abstract object and their schema and
| arguments is centrally deployed, I called this "protocol
| manager".
|
| The idea is you write all your code to use a "span" and have
| contextual data in a span, and you can include or exclude data in
| a span with a non-software rollout. Your communication schema of
| RPC and API calls is a runtime decided thing, not hardcoded.
|
| If you have N deployed versions of code and you want to upgrade
| to X, you have to test 1..N to X versions. So nobody does that.
| jbott wrote:
| You might be interested in learning about Erlang - it supports
| hot code reloads natively:
| https://oozou.com/blog/understanding-elixir-otp-applications...
| fabianlindfors wrote:
| The database aspect of this problem is particularly interesting
| to me. I've previously built Reshape [0], a zero-downtime
| migration tool for Postgres, and am now working on ReshapeDB
| [1], a full database designed from the ground up to tackle this
| problem.
|
| [0] https://github.com/fabianlindfors/reshape [1]
| https://reshapedb.com
| 1970-01-01 wrote:
| Lexus did the very same thing about 8 years ago:
|
| https://www.consumerreports.org/lexus/what-to-do-if-your-lex...
| scardycat wrote:
| Bringing CI/CD mindset to cars is probably not a great idea.
| Software updates to commuter vehicles should have a high bar for
| operational standards, and a simple thing such as an expired
| certificate should have never been deployed. Having isolated
| networks in vehicles helps but doesn't prevent broken updates
| from, eventually, bricking the cars.
| nomel wrote:
| I think this shows more of a fundamental flaw in their update
| mechanism, than anything.
|
| I don't think a botched update is a big deal. It happens, and
| should be _expected_ , in a sane design. The fact that the
| customer noticed is a big deal.
|
| There are many implementations that could be used for an "auto
| rollback" feature. They either failed to implement that in a
| sane way, or they were goobers, and assumed things would always
| be rosy.
| babypuncher wrote:
| I would be pretty pissed if I went out to my garage to head
| to work one morning and found that a damn software update
| bricked my car overnight. This shouldn't even be a thing, why
| does a _car_ need regular software updates to keep
| functioning?
| BuckRogers wrote:
| It doesn't. People and these tech companies are tools. And
| do it largely in search of ways to take more of your money.
| It's not a favor.
| theandrewbailey wrote:
| If ICE tech was the hot new thing in cars, things like
| spark plugs would have a chip so that it would fire _n_
| times then break, but don 't worry, there's a
| subscription for new ones and they will be automatically
| ordered when the car says so. If the credit card on file
| expires, your spark plugs won't work anymore, even if you
| just replaced them.
| jdminhbg wrote:
| There's never been any car that 100% will work in the
| morning when you go to the garage. It's all tradeoffs.
| theandrewbailey wrote:
| Now we can add 'bad software update' to the list of
| things that can go wrong with cars. We didn't use to have
| that.
| jdminhbg wrote:
| At the same time, we're losing tons of mechanical
| problems that use to go wrong with cars. The amount of
| time lost to car malfunctions is way down over the past
| couple decades, even with slight regressions like this
| one.
| bink wrote:
| It doesn't need regular updates to keep functioning. It
| offers regular updates as they add new features. For
| instance, in this update a new feature was added to allow
| for proximity locking at home but disable proximity
| unlocking. That would lessen the number of times the car
| would lock and unlock accidentally as you walk in and out
| of the garage. No one was forced to install the update.
| theandrewbailey wrote:
| Cars 20 years ago, even most of them 10 years ago, never
| got any updates unless they got recalled. Nothing broke,
| nothing got hacked, and most are probably still working
| fine.
|
| What happened to cars today? I refuse to believe that it's
| solely because these are _electric_ cars, as if the way the
| car stores and uses energy dictates that it must be part of
| the internet of things.
|
| Edit: there were electric cars over 100 years ago. I bet
| they never got software updates.
| ryandvm wrote:
| As software eats the world, it becomes more and more
| apparent to the non-developers of the world that software
| engineering is not, and never has been, a real
| engineering discipline.
|
| Tech Support: "Oh your garage door is bricked after last
| nights update? Yeah, apparently the [totally
| uncredentialed] contractor that wrote that update is only
| 3 weeks out of coding bootcamp and was just copying and
| pasting from ChatGPT. Lmao"
| jdminhbg wrote:
| Cars 20 years ago didn't have realtime traffic on big
| touchscreens that you can use to look up your destination
| and plan out a route that also lets you schedule
| fueling/charging stops, oh and also stream humanity's
| entire library of recorded music, books, and podcasts.
| It's a tradeoff that the vast majority of people want.
| mckn1ght wrote:
| All that stuff should be done via smartphones and the
| screen in the car should be a dumb display for it.
| gitfan86 wrote:
| The Tesla update is slow probably for this reason. It is
| probably verifying that it can rollback at any point of
| failure.
| yjftsjthsd-h wrote:
| I would naively expect it to just do A/B updates, which
| unless I'm forgetting something shouldn't incur a speed
| penalty? (Other than that the update doesn't get _applied_
| until restart)
| liminalsunset wrote:
| I believe one of the reasons it is slow is because it is
| also updating the firmware on any number of connected ECUs
| over the CAN bus. This typically means the image has to be
| sent over a 500kbit/s bus so there is a limit to how long
| it has to take.
| 1234letshaveatw wrote:
| From a few days back- Its software has been a "key
| differentiator" https://electrek.co/2023/11/10/rivian-using-
| software-to-scal... kind of humorous in hindsight
| qudat wrote:
| What a nightmare. This is where software engineering meets "real"
| engineering, where a "bug" has potentially life threatening
| consequences.
| nomel wrote:
| > where a "bug" has potentially life threatening consequences.
|
| What are you referring to? That is not relevant to this story,
| and would require a deep understanding of the system to make
| such a claim of negligence.
|
| "The issue impacts the infotainment system. In most cases, the
| rest of the vehicle systems are still operational ..."
|
| Also, you can't do an update while driving.
| jawns wrote:
| Based on the photo included in the article, what they're
| calling an infotainment system is actually two separate
| components, one of which appears to be taking the place of a
| traditional dashboard. If that's the case and there's no
| other way to monitor speed, fuel levels, engine temperature,
| warning lights, etc., I'd say that's quite a bit more
| worrisome than just not being able to play your favorite
| music while driving.
| nomel wrote:
| I understand, but the risk of life wouldn't be from the
| bug, it would be from conscious choice of driving without a
| speedo. There's a _critical_ distinction there.
|
| In this case, mileage/battery are still present, and I
| would assume safety critical warnings would still be
| displayed.
| jonplackett wrote:
| If you need thr car in an emergency and it's unusable,
| that almost counts.
| thrill wrote:
| "almost" only counts in horseshoes and nukes.
| prmoustache wrote:
| If it is unusable you just don't use it.
|
| That any car can have a dead /failing battery/fuel
| pump/whatever one morning or fail and leave you stranded
| on the road to an hospital has never been considered a
| safety risk.
| ct0 wrote:
| You've never been to death valley without air conditioning Or
| Russia without heat. I think the infotainment system in this
| case has a broken climate control function. There are
| workarounds, but why if you don't have your phone?
| nomel wrote:
| This is a great point. I would claim that it would be a bad
| choice to initiate the update (it's a manual process,
| requiring intent) when you're _in_ these conditions. But, a
| less tech savvy person may not understand that updates can
| be risky, and give it a shot at a remote charging station.
| BlueTemplar wrote:
| I have been talking recently to someone whose job involves
| sometimes driving to other continents, and they mentioned
| that cars more recent than ~2003 were out of the question
| because outside of the EU you cannot expect random
| mechanics to have the computers required to interface with
| the car's computers - required for repairs.
| PeterisP wrote:
| If that's the only problem, then that would be fixed
| simply by taking "computers required to interface with
| the car's computers" i.e. a <$200 OBD-II device with you
| in the car, not choosing a car with 20+ years of wear and
| tear.
|
| Also, since many of these places have a lot of used cars
| that once were in the EU or other places with legally
| required computers, this is changing as now most of the
| mechanics even in relatively poor remote places have to
| have these devices as now many of their local customers
| don't have "computerless" cars any more, and the existing
| pre-2003 cars won't last forever even there.
| qudat wrote:
| > What are you referring to?
|
| Not the specifics of this article, but more generally about
| the gravity of the situation car makers (and their software
| engineers) operate under. The very idea that an OTA software
| update that causes a bug within more critical features of a
| car could be life threatening. So my point isn't about the
| specifics of this particular bug, rather the capacity for a
| bug that could kill.
| gravitronic wrote:
| I used to work for a company that built satellite receivers that
| would be installed in all sorts of weird remote environments in
| order to pull radio or tv from satellite and rebroadcast locally.
|
| If we pushed a broken update it might mean someone from the radio
| company would have to make a trip to go pull the device and send
| it to us physically.
|
| Our upgrader did not run as root, but one time we had to move a
| file as root.. so I had to figure out a way to exploit our
| machine reliably from a local user, gain root, and move the file
| out of the way. We'd then deploy this over the satellite head end
| and N remote units would receive and run the upgrade
| autonomously. Fun stuff.
|
| Turns out we had a separate process running that listened on a
| local socket and would run any command it received as root.
| Nobody remembered building or releasing it but it made my work
| quick.
| nomel wrote:
| > Turns out we had a separate process running that listened on
| a local socket and would run any command it received as root.
| Nobody remembered building or releasing it but it made my work
| quick.
|
| No offense, but what a shit show. It makes me assume no source
| control, and a really good chance that state actors made their
| way into your network/product. This almost happened at a
| communication startup I know, with three letter agencies
| helping resolve it. State actors really like infiltrating
| communication stuffs.
| gravitronic wrote:
| oh, yeah, this place was a total shit show. BUT we were
| ISO9001 certified!! So we had source control (CVS) and a
| Process (with a capital P) to follow. In this case that code
| was added in a previous development iteration because someone
| needed to run something as root when a user pressed a certain
| button on the LCD panel in front and this was the decoupled
| solution they wrote intentionally. Somehow I feel like that
| makes it worse than if it was a malicious three letter agency
| lol.
| nomel wrote:
| Package this up and send it to https://thedailywtf.com
|
| It's beautiful.
| singleshot_ wrote:
| The person who built and released this might not have ever
| worked for your company, which might be why no one remembers
| building or releasing it.
| martin8412 wrote:
| Stuff like this is why I don't want OTA updates in my cars. Let
| the car dealership deal with it during regular maintenance.
| They'll be on the hook for fixing it before handing the car back
| to me.
| ben_jones wrote:
| But what if your car doesn't have the latest emojiset or social
| sharing functionality within the notes app?
| galangalalgol wrote:
| Don't even need OTA. A seattle radio station bricked a bunch of
| mazdas.
|
| https://www.autoblog.com/2022/02/09/seattle-radio-station-br...
| cozzyd wrote:
| Amazing. Can't wait for some car software stack to be so
| poorly designed that an FM transmitter can remote takeover.
| galangalalgol wrote:
| It quite possibly could have with a well formed digital fm
| payload.
| cozzyd wrote:
| We'll hopefully no car manufacturer is dumb enough that
| the FM radio software can affect vehicle control
| functions but I wouldn't be surprised ..
| aftbit wrote:
| I'm pretty sure they are all dumb enough for this. This
| is why people can steal (modern) cars by ripping a
| headlight out and accessing the CAN bus through the gap.
|
| https://kentindell.github.io/2023/04/03/can-injection/
|
| AIUI the military uses "CAN bus firewalls" to prevent
| this. There is also some sort of encrypted/authenticated
| CAN bus protocol in the works. Neither are common in
| production cars as of 2022 or so, though they may be soon
| enough.
| dilyevsky wrote:
| There is nearly zero regular maintenance to be done on EVs
| though. No oil, no belts, no fuel filter, spark plugs etc. Even
| the brakes will likely last entire lifetime of the car
| nkingsy wrote:
| Hertz seemed to find teslas cost double ice counterparts to
| maintain.
|
| Maybe it's auto company smoke but source:
| https://fortune.com/2023/10/27/tesla-elon-musk-hertz-evs-
| ren...
| dilyevsky wrote:
| That's because it is run by idiots who ran it into
| bankruptcy
| aftbit wrote:
| I'm not sure maintenance costs are really the relevant
| part. It seems like the problem is that Teslas are cheaper
| now and thus Hertz's fleet is worth less than it was
| before. Additionally, they find that Teslas suffer more
| damage, likely from collisions or similar. Routine
| maintenance costs are not mentioned in that article at all.
|
| >In short, the declining value of the Tesla cars in Hertz's
| fleet--a decline directly caused by Musk's price cuts--has
| hit Hertz squarely in its profits.
|
| >Without explaining precisely why, Scherr said Hertz is
| suffering a higher incidence of damage specifically with
| its EV fleet, where the repair costs are roughly double
| that of a comparable gas-fueled car.
|
| >"Studies of current EV ownership evidence lower incidence
| of damage and collision than for ICE vehicles, not higher
| as we are experiencing," he revealed. Musk's price cuts
| then become an acute problem when one of the Hertz EVs
| sustains so much damage that the cost of repair is more
| than the asset itself.
|
| >"Where a car is salvaged, we must crystallize at once any
| difference between our carrying value and the market value
| of that car," Scherr explained. "The [price] declines in
| EVs over the course of 2023, driven primarily by Tesla,
| have driven the fair market value of our EVs lower as
| compared to last year, such that a salvage creates a larger
| loss and, therefore, greater burden."
|
| >In short, Hertz then needs to book a noncash accounting
| charge. Together with the higher repair costs this led to
| significant profit margin headwinds.
|
| https://archive.ph/leFdf
| BlueTemplar wrote:
| Sucks to be them, but then maybe they should have
| listened to Musk when in the very beginning of Tesla he
| said the goal was to make EVs cheap ?
| martin8412 wrote:
| Bullshit. BEVs eat through tires because they're heavy. The
| air cabin filter and pollen filter need frequent replacement.
| In the case of Tesla, you better check the undercarriage so
| you can potentially spot control arms soon to fail. The
| shield for the battery should be inspected.
| dilyevsky wrote:
| You can't rotate/change tires at local tire shop? It's not
| under warranty anyway.
|
| > The air cabin filter and pollen filter need frequent
| replacement
|
| Yeah "regularly" like every two years lol, you gon wait for
| that long to update your software and pay $500 to a dealer
| to do it?
| wannacboatmovie wrote:
| EVs should be subject to mandatory German-style inspection by
| law to counteract this delusion.
| dilyevsky wrote:
| Hey if you want to pay a dealer 300-500 euro to inspect
| your tires and swap air filter I'm not against that - you
| do you. Also if you buy a german car (ev or not) then yes a
| mandatory inspection is warranted
| rconti wrote:
| Just EVs?
| SoftTalker wrote:
| Also worth considering that a manufacturer like Rivian is
| pretty small. Every town has a Ford dealer. There are many
| states, however, that don't even have a single Rivian service
| center.
| 0xffff2 wrote:
| Coverage is even thinner than I would have guessed.
| California has 6 Rivian service centers, but they're strongly
| clustered in the Bay area and LA/OC/SAN. Even in California
| if you live in Fresno/Bakersfield/Santa Barbara etc you're
| looking at several hours round trip to visit an official
| service center.
| evanelias wrote:
| Regardless of OTA vs dealership-only updates, software bugs can
| have problematic effects long after the update occurred.
|
| So far I've had to take my Chevy Bolt to the dealership twice
| due to major software problems causing the "service needed"
| indicator to be lit (equivalent to "check engine"), and I've
| owned it for barely over a year.
|
| The first time, some random bug made the car think there was
| something wrong with the transmission under some extremely
| specific set of circumstances, and as a safety precaution it
| would refuse to shift into drive if not serviced within 100 key
| cycles.
|
| The second time, it was a bug with the software that manages
| battery health making the car think the battery had a severe
| problem. In that situation, as a safety precaution, the car
| refuses to charge above 40%, disables regenerative breaking,
| limits the HVAC usage, and slightly limits max acceleration.
|
| This is getting very irritating. I bought an EV because I
| thought it would require fewer maintenance visits to the
| dealer!
| nicholasjarnold wrote:
| Is it possible, as a licensee of the Rivian vehicle system, to
| disable the automatic OTA updates without having expert-level
| knowledge or tooling?
|
| Also, yes, I'm specifically avoiding using the word "owner" above
| for obvious reasons.
| 55873445216111 wrote:
| Rivian "licensee" here. So far all updates have required you to
| press a button (in the car or on the app) to launch the update
| installer. Not sure how many weeks you can ignore it for as I
| never tried.
| kevinventullo wrote:
| My 2019 car is not connected to the internet. Instead, I use
| Apple CarPlay for everything.
|
| Is there any reason not to do it this way?
| gmane wrote:
| Then automakers can't sell your information for fat stacks of
| cash.
| lotsofpulp wrote:
| This, and automakers do not want to become dumb appliances
| (even though they are for 90% of people).
| pookha wrote:
| ..And authoritarian governments can't take complete custody
| of your movements. Can't shutdown your car either. Seems to
| be a win-win for both sides (private and public industry).
|
| https://qz.com/1522309/how-chinas-electric-car-
| surveillance-...
|
| https://apnews.com/article/4a749a4211904784826b45e812cff4ca
| ChadMoran wrote:
| Quite a few reasons but primarily CarPlay doesn't know about
| your vehicle's charge and road trip navigation requires
| external inputs to determine your charge route.
|
| Apple could potentially offer an API to have "reverse" CarPlay
| where the car's app can feed information into iOS. I recently
| rented an Mercedes EV which had Apple CarPlay and it was a
| weird experience having to manage two sets of experiences.
| callalex wrote:
| The exact API you describe already exists in CarPlay.
| ChadMoran wrote:
| Today I Learned... that's great to hear!
| kyleee wrote:
| Manufacturers should be required to expose an agreed upon
| spec API that provides range estimate, battery state, etc. So
| CarPlay and other apps can access the info in a standardized
| way
| ChadMoran wrote:
| 100%. I've owned Teslas since 2017 and always found the
| infotainment to be very good. Though I've really preferred
| Apple Maps routing recently. The descriptions of when to
| turn are much more human.
| callalex wrote:
| Apple holds up their end of the deal by supporting the EV
| Routing feature in maps. It's entirely up to manufacturers
| to expose the information at this point.
| darknavi wrote:
| Well for one Rivian (and Tesla) don't support CarPlay or
| Android Auto.
| callalex wrote:
| More potential for anticompetitive vendor lock-in (sugar-coated
| as differentiation), and more opportunities to profiteer from
| stalking behavior (sugar-coated as telemetry to improve user
| experience).
| aftbit wrote:
| My cars are from 2019 and 2001. I don't use CarPlay or any
| internet features in any of them. Instead, I just use my
| Android's screen itself for navigation and bluetooth for phone
| calls and music.
|
| Perhaps there are advantages to tighter integration with my car
| (at least the newer one) but IMO they are outweighed by the
| risks of things like this, or even just getting a software
| update that borks a small feature that I like.
| lotsofpulp wrote:
| CarPlay and Android Auto has no risk like this. It is great
| because it is just a protocol for your phone to be able to
| use the screen in the car. Your data stays on your phone, and
| there is no risk of intrusion.
| nicolaslem wrote:
| This is the kind of thing that keeps my awake at night.
|
| Does anyone here have some practical tips to turn an embedded
| Linux machine into an appliance? The kind of system that a
| botched update cannot brick but only momentarily disable until a
| non-technical user presses a factory reset button of some sort.
| elitepleb wrote:
| A/B updates as implemented in android,
| https://bootlin.com/pub/conferences/2022/elce/opdenacker-imp...
| hospitalJail wrote:
| >Does anyone here have some practical tips to turn an embedded
| Linux machine into an appliance?
|
| Lol
|
| I suppose this is the negative about having sensors that make
| sure water gets hot enough to be sanitizing, but not so hot
| that it wastes energy. And I'm sure you can imagine 100 other
| uses of having a microcontroller/CPU process data and do
| feedback. (I'm sure there are EE only ways of doing it, but
| theoretically possible and useful are two different thigs)
| ct0 wrote:
| Will insurance carriers cover damages due to botched updates?
| Imagine 10 years from now the power/control that electric
| delivery companies would have over retailers like amazon. One
| botched update away from a complete backup for delivery vans.
| eigenvalue wrote:
| Can't imagine how much it would suck to be the engineer who fat
| fingered it and caused a huge crisis for the company,
| inconveniencing tons of customers and costing millions. Even if
| there should be processes in place to prevent it in the first
| place, you'd still know you were the "but for cause" of the
| problem.
| xyst wrote:
| Looks like this car brand is circling the drain. Glad I never
| bought into the hype.
| seattle_spring wrote:
| It's circling the drain because of one bad software update?
|
| Sounds more like you've just bought into the doom and gloom
| that a few specific news outlets have been pushing.
| karaterobot wrote:
| What kinds of changes are generally included in these over the
| air updates? I have this sudden urge to shake my fist at a cloud
| and tell the gods that cars shouldn't need updates in the first
| place, if the car was ever deemed ready for production and then
| sold to customers for money. But, maybe I'm wrong, and it makes
| perfect sense. All I can think of would be something like a
| periodic update to navigation data, is that it?
| ezfe wrote:
| It's possible to deem software ready to sell but find
| improvements later.
|
| Simple example: my Subaru was sold to me with an interesting
| design decision that caused the radio to come on whenever the
| car was started. This was not a bug. Every Subaru worked this
| way for years. A year into ownership I received an OTA update
| that added a "not playing" state on startup.
|
| This was never a safety issue and was likely not a defect. It
| was, however, stupid and needed to be changed.
| cryptoegorophy wrote:
| Tesla updates are sent in batches and you can opt in for advanced
| updates I guess to be earlier. Normally when I see that there is
| an update on Reddit then it takes 1-2 weeks at least to get to my
| car with the "advanced" updates on.
| xyst wrote:
| Tesla. Rivian. All cut from the same cloth. A car should be
| simple. Yet we are stuffing all of this tech junk into it and
| trying to repackage is as something else to pump the numbers.
|
| Car companies suck at tech. Let's be realistic. They should stay
| their lane and focus on improving the car and physical aspects
| (safety, reducing carbon output, longevity, ease of
| repairability, reducing supply chain issues)
| bhauer wrote:
| > _Tesla. Rivian. All cut from the same cloth._
|
| I'm not aware of any Tesla OTA updates bricking the
| infotainment system. At least since I've been paying attention.
| I don't see them quite as similar as you suggest.
| margalabargala wrote:
| There have been plenty.
|
| https://www.reddit.com/r/TeslaLounge/comments/112oqln/new_te.
| ..
|
| https://teslamotorsclub.com/tmc/threads/failed-software-
| upda...
|
| Two examples of many.
|
| I'm not aware of any _fleet-wide_ issues that accidentally
| bricked Teslas, but as one-offs they do happen; and unlike
| this Rivian update, a botched Tesla OTA generally leaves the
| car undriveable and needing to be towed. These Rivians will
| at least still drive, as long as you don 't need fancy
| extraneous luxury features like a...speedometer.
| avereveard wrote:
| > In most cases, the rest of the vehicle systems are still
| operational
|
| Like what do you mean "in most cases" I can understand a broken
| infotainment needing reset but imagine if you had to tow your
| truck I'd be furious.
| Havoc wrote:
| > the vehicle is not bricked
|
| What a time to be alive. Software updates (almost) turning cars
| into paper weights lol
| emmelaich wrote:
| Poor title; physical repair is not required. Physical _presence_
| is required.
| Someone1234 wrote:
| The article doesn't really state what is required to repair the
| vehicle. I'd assume if it was as simple as loading a flash
| drive and plugging it in, then Rivian would have provided a way
| for customers to self-fix. The second a single body panel is
| removed to gain access to the headunit, it is a physical
| repair.
|
| So without more info we cannot know if it is accurate or not.
| WirelessGigabit wrote:
| What's the impact on your insurance should you get into an
| accident?
|
| The speedometer screen is gone, so does that not imply the
| vehicle is inherently unsafe to drive?
| MisterTea wrote:
| Can I please just buy a car with a motor and battery? Why does
| every god damn vehicle have to come littered with screens and
| chips all together like some tentacle monster?
|
| All I need is a gauge cluster screen that can display the normal
| info like stored and heading while also letting me configure the
| cars performance and safety features. Then let me mount a double
| DIN radio that isn't dog shit. I've not seen a single new car
| with these dumb screens with a sound system that's not tinny
| muddy garbage with zero adjustment save for "bass" and "treble"
| settings. I mean all that technology and you can't be assed to
| put an eq in there. HVAC never needed more than two or three
| knobs anyway.
| easylion wrote:
| need a easy way to do restore to previous version offline. take
| 100 bucks extra if required to have a backup ssd. Don't want to
| be camping and then realizing i'm stuck because of some junior
| dev not being competent enough
| seattle_spring wrote:
| Why would you intentionally upgrade your vehicle software while
| camping? It's not like this stuff installs automatically, you
| have to explicitly accept the installation. Waiting a few days
| or even a few weeks before hitting "install" is completely
| normal.
| ralmidani wrote:
| Move fast and break things that move fast...
|
| I don't really like or trust most (if not all) of the established
| automakers, but there is something to be said for having several
| decades (over a century in some cases) of experience building
| potential killing machines vs. a company that's not even 15 years
| old. The established players have put out cars which suffered
| freak malfunctions, but Rivian (and Tesla) seem to be struggling
| more with QA.
|
| Non-rhetorical question: do companies have safeguards for
| critical components like braking systems, or are they also prone
| to catastrophic failure if a software engineer pushes a bad
| commit?
| ezfe wrote:
| The moving fast components were unaffected by this issue...
| Someone1234 wrote:
| I wonder if the way Microsoft's XBox is designed may be something
| to look towards in terms of hardware reliability/fallback.
| Specifically they utilize a Hypervisor which rarely needs
| updates, running different operating environments which need
| frequent updates.
|
| - Better isolation of different parts of the system (e.g.
| infotainment unit, instrument cluster, et al).
|
| - Better isolation for updates (e.g. run a "beta" update, and a
| "stable" update side-by-side).
|
| - Automatic error detection and rollback (e.g. if a VM keeps
| restarting after an update).
|
| - Ease of offering features like rollbacks to end-users.
|
| - Rare hypervisor updates can be held to a much higher standard
| relative to other VM updates.
|
| The only downside of hypervisor-based systems is slightly higher
| hardware costs. But even that is largely mitigated by modern
| architectures that natively support virtualization.
|
| PS - You can also look to any containerization. I specifically
| brought up the XBox because it is a hardware product, just like a
| vehicle.
| antoniuschan99 wrote:
| Wondering why there isn't an option for a factory reset (eg.
| press and hold with a paperclip for 10 seconds)
| glonq wrote:
| As somebody who has spent many years doing embedded+iot related
| to remote fleet firmware updates, this is the kind of thing that
| lurks in my nightmares.
|
| I'd love to be a fly on the wall at Rivian engineering/operations
| this week!
| janitor61 wrote:
| This is tangentially Rivian related, but does anyone else see the
| inherent danger of stylized tail lights that are just a single
| red bar across the back of the car? Travelling on the freeway at
| night I can't really gauge the distance to the car in front of me
| if it's far ahead and if there's no discernable left and right
| brake lights. I'd believe Rivians and other cars like that are
| more at risk of high speed rear-end collisions.
| rurp wrote:
| This reminds of the terrible turn signals Mini used, which look
| like flashing arrows pointing in the opposite direction of the
| turn[0].
|
| Getting cute with basic stuff like tail lights is forgettable
| or annoying at best, and absolutely can be dangerous.
|
| [0]https://jalopnik.com/congratulations-mini-you-made-the-
| stupi...
| latchkey wrote:
| I built a whole remote software update mechanism for a control
| binary that ran on 25k+ servers across multiple data centers.
|
| Rest assured that after the first time I messed it up (which
| required ssh into each box individually), I wrote a lot of unit
| and integration tests to make sure that it never failed to deploy
| again. One of the integration tests ensured that the app started
| up and could always go through the internal auto update process.
| This ran in CI and would fail the build if it didn't pass.
|
| While I fully understand that this is hard to get right 100% of
| the time, a mess up of this level by a car manufacturer is pretty
| amazing to me.
| aaronbeekay wrote:
| As somebody currently working at an automaker on software
| systems, the amazing thing to me is that a mess up of this
| level doesn't happen weekly. It's rough out here.
| bozhark wrote:
| What's the priority then, telemetry data? Why is it rough out
| there?
| donmcronald wrote:
| > While I fully understand that this is hard to get right 100%
| of the time, a mess up of this level by a car manufacturer is
| pretty amazing to me.
|
| I feel like it's going to happen to someone that makes network
| devices eventually. I'm always scared to update my (several
| hundred) UniFi devices. Their update process isn't foolproof
| and they push auto-updates via the UI pretty hard.
|
| Several years ago they caused some people's devices to
| disconnect from the management controller when they enabled
| 'https' communication. Prior to that, if you were pointing
| devices at 'https://example.com:8080...' they would ignore the
| 'https' part and do an 'http' request to port '8080'. Then they
| pushed their 'https' update which _expected_ an 'https'
| connection and didn't fall back to the old behavior for anyone
| that was mistakenly using 'https' in their URL initially. Some
| people on their forums complained about having to manually SSH
| to every device to fix the issue.
|
| It was caused by an end-user mistake, but they knew it was a
| potential issue. AFAIK, their attitude on it hasn't changed and
| a lot and at the time their response was that they knew it
| would break some people, but that it wouldn't be that many
| (lol).
|
| IMO, the issue with those systems is that basic communication
| back to the update / config server is part of the total package
| which is too complex (ie: a full Debian install). I'd rather
| see something like Mender (mender.io) where the core
| communications / updates come from a hardened system with
| watchdog, recovery, rollback logic.
|
| Think of how crazy it is to have something like pfSense doing
| package based updates rather than slice based updates. At least
| with boot environments they could add some watchdog and
| rollback type logic, but it'll still be part of the total
| system instead of something like a hardened slice based setup
| where the most critical logic is isolated from everything else
| and treated like a princess.
|
| Do you have any insight on package vs slice based systems for
| updates? Did you isolate update logic from the rest of the
| system or am I out of touch with that opinion?
| code_runner wrote:
| out of morbid curiosity.... how long did it take to ssh into
| and fix all of those servers? I imagine even automating a fix
| (if possible) would still take a good amount of time.
| latchkey wrote:
| gnu parallel and sshpass is your friend.
|
| The way I built my app was that I could install it cleanly
| via a curl | bash.
|
| So, I just had a simple shell script that iterated through
| the list of IP addresses (from the DHCP leases), ran curl |
| bash and that cleaned up the mess pretty quickly.
| FireBeyond wrote:
| As annoying as this, I find this laughable, too. Rivian updated
| users on the situation. Then, whines Electrek:
|
| > That's the last update we had over 10 hours after Rivian
| customer vehicles were fed the bad software update.
|
| "Over 10 hours"!
|
| I suppose it isn't Tesla, who yeets updates over the fence, that
| break new things, yeets another update that fixes that problem
| but introduces another one, then reverts back to two versions
| prior, before the issue. The Tesla that gets firmware fixes from
| vendors that have a test harness that should take 36+ hours to
| run, but says YOLO and flashes it onto a random car they have
| lying around and emails the vender back 3 hours later saying
| "LGTM, WFM, thanks!"
| wnevets wrote:
| This maybe crazy but if you're writing software for hardware that
| cost tens of thousands of dollars it should be impossible to
| brick it with an update, especially if that update is OTA.
|
| The future is going great.
| fsckboy wrote:
| I wish the economics of mass production didn't turn pennies into
| millions that need to be eliminated, because I've always thought
| the "don't disconnect from power" and "update bricks it" type
| problems could be solved by having extra EPROM to download into,
| the way linux keeps the previous kernel around after an update.
|
| Or at least the ability to re-init/download from scratch, like a
| borked macbook disk. And hey, not the extra ability to do that,
| make it "the way it works" so you're always testing it.
| j45 wrote:
| As long as they are good for fixing it, this might what being a
| Pioneer or Early Adopter is about.
| gunapologist99 wrote:
| This is why I don't really want my car to have any antenna (that
| receives/interprets code) or receive OTA updates, ever.
|
| I'd like to please force any attackers to at least be within 50
| feet of my TPMS, instead of being literally anywhere on the
| planet.
___________________________________________________________________
(page generated 2023-11-14 23:00 UTC)