https://hackaday.com/2021/02/11/tesla-recalls-cars-with-emmc-failures-calls-part-a-wear-item/ Skip to content Logo Hackaday Primary Menu * Home * Blog * Hackaday.io * Tindie * Hackaday Prize * Submit * About February 11, 2021 Tesla Recalls Cars With EMMC Failures, Calls Part A 'Wear Item' 86 Comments * by: Lewin Day February 11, 2021 * * [eMMC] It's a problem familiar to anyone who's spent a decent amount of time playing with a Raspberry Pi - over time, the flash in the SD card reaches its write cycle limits, and causes a cavalcade of confusing errors before failing entirely. While flash storage is fast, compact, and mechanically reliable, it has always had a writeable lifespan much shorter than magnetic technologies. [tesla-emmc-flash]Flash storage failures in the computer behind Tesla's famous touch screen are causing headaches for drivers. Of course, with proper wear levelling techniques and careful use, these issues can be mitigated successfully. The surprising thing is when a major automaker fails to implement such basic features, as was the case with several Tesla models. Due to the car's Linux operating system logging excessively to its 8 GB eMMC storage, the flash modules have been wearing out. This leads to widespread failures in the car, typically putting it into limp mode and disabling many features controlled via the touchscreen. With the issue affecting important subsystems such as the heater, defroster, and warning systems, the NHTSA wrote to the automaker in January requesting a recall. Tesla's response acquiesced to this request with some consternation, downplaying the severity of the issue. Now they are claiming that the eMMC chip, ball-grid soldered to the motherboard, inaccessible without disassembling the dash, and not specifically mentioned in the owner's manual, should be considered a "wear item", and thus should not be subject to such scrutiny. Certainly An Odd Wear Item [Replacing-Tesla-Tegra-EMMC-chip-0-47-screenshot]The chip in question, a sub-$7 eMMC chip packing 8GB of storage. Historically, major electronic parts in automobiles are not considered consumables. While it's not uncommon for some cars to face issues with engine control units or body control modules, they're not typically treated as wear items to be replaced at nominal intervals. Thus far, precedent has considered these parts as something to last the lifetime of the vehicle, and to be replaced in the case of unexpected malfunction. The Tesla case is different in that the eMMC failure is, by and large, inevitable. Rather than being a case of isolated malfunctions in a small percentage of cars as would be expected from the occasional manufacturing defect, this is a issue affecting every car that rolled off the line up to a certain date. Failure rates are up to 30 percent in certain build months. With the computer and touchscreen being in charge of so many vital vehicle functions, it's not a defect that can be easily ignored by the end user. [Replacing-Tesla-Tegra-EMMC-chip-2-7-screenshot-1]Replacing the chip involves reflowing the board, and carefully pulling off the offending part with tweezers. Tesla's assertion that the eMMC chip should be considered a 'wear item' is a dubious one at best. Flash memory does wear out, it's true, as Tesla points out when discussing the limits of the technology. Many parts on a modern car wear out over time - brake pads, belts, and air filters are all common examples. The difference is that these parts are all designed to be replaced by the end user or a typical mechanic. Trying to claim that a ball-grid array chip, permanently soldered onto a PCB and buried inside the dashboard is a wear item is patently ludicrous. If it were, we'd expect to see several things. There'd be a recommend time and mileage upon which the eMMC would be changed to avoid surprise failures, and this would be listed in the manual. Additionally, Tesla's repair process would involve desoldering the eMMC chip from the board and replacing it directly. Given that Tesla are instead replacing the computers as a whole is indicative that the part is not being treated as a wear item by anyone, anywhere. Obviously, the chip can be replaced, but it's no easy job. Once the computer's main board has been extracted from the car, the storage must be backed up over JTAG. Then, it must be carefully reflowed to remove the chip, in a delicate process that has a significant chance of damaging other components on the board. If the chip was a wear item, it wouldn't require specialist BGA reflow equipment to change. We'd see Tesla doing it routinely, replacing a sub-$7 chip rather than swapping out entire mainboards instead at the costs of thousands of dollars. Granted, there are parts of modern cars that are also time consuming to replace - such as timing belts, water pumps, and so on. However, again, in these cases, automakers make it clear that these are wear items ahead of time, create maintenance schedules for them, and standard processes to change them. Nobody would put up with swapping out their entire front suspension setup every time their brakes wore out - automakers realised brake pads were wear items and designed accordingly. Tesla simply dropped the ball, writing too often to the flash memory, which isn't easily replaceable. The proper solution is trivial. Either stop logging so much to flash storage, or make it easier to swap out. And maybe put the logs in their own partition. While SD cards probably aren't up to snuff for storing the car's operating system, they'd make a cheap place to store non-critical logs that probably are never read anyway. Alternatively, put the eMMC chip on a removable module, or just use an M.2 drive with automotive-rated connectors. The issue is claimed to only effect models built prior to March 2018, which run on an NVIDIA Tegra 3. Later models are based on the Intel Atom, and feature a larger eMMC chip on board. These modules are yet to demonstrate the same failures, and Tesla claim they should not suffer the issue. We'll see. * [share_face] * [share_twit] * [share_in] * [share_mail] Posted in car hacks, Hackaday Columns, News, SliderTagged emmc, flash , flash storage, tesla Post navigation - Phishing With Morse Code Tiny Motors Enable Experimental Piano Performance - 86 thoughts on "Tesla Recalls Cars With EMMC Failures, Calls Part A 'Wear Item'" 1. Thomas says: February 11, 2021 at 7:15 am As a Tesla owner, I know this is going to be an issue. I saw it before I bought my Model Y. Why they didn't go for removable flash is beyond me. Even as tech advances, the packages will change. But at least there are adapters to deal with that. In this case, it's far more complicated than replacing the flash on an iPhone, of which most consumers couldn't do today. Report comment Reply 1. Chris (MN) says: February 11, 2021 at 8:14 am I'm not sure this is a problem in the Model Y and newer Teslas. But I'm also not sure that it's a good idea to make it removable. Removable flash can have failures over time. A better solution may be to just use better software technique that reduces logging and handles flash writes appropriately plus using a (pseudo)-Single-level-cell flash chip for the main subsystem. Those can be made to last literally decades even with high writes and with better software, bascially will never wear out (well, the PCB itself probably will eventually... maybe 50-100 years?). Entertainment storage should be removable, though. Report comment Reply 1. ThisGuy says: February 11, 2021 at 8:50 am No, the proper way is not to have important functions like the defroster rely on an eMMC chip (or a computer for that matter). Shit like this is why I think Tesla doesn't belong in the automotive industry. Because they show time and time again they don't understand safety engineering and imho show a massive contempt for their customers. Report comment Reply 1. Escalion says: February 11, 2021 at 9:03 am eMMC would be fine if they weren't writing logs to it. Simply move the logs to a removable flash drive and the problem is resolved. I would even go as far to say, 2 SD cards - one for user configuration files and one for logs. This would leave the eMMC as read-only and avoid the whole shenanigan they found themselves in. Report comment Reply 1. Dude says: February 11, 2021 at 11:32 am Flash memory also has read disturb and a chance to corrupt the data anyhow, so even if you were doing zero logging, the data corruption can still render your windshield wipers inoperative which is a perfectly avoidable design choice. Report comment 2. Jul13 says: February 11, 2021 at 12:29 pm Not the point. A safety feature like defrosting should not fail for any of those reasons. Sd card or Emmc either should be able to fail without disabling a safety feature Report comment 2. Ostracus says: February 11, 2021 at 11:30 am "No, the proper way is not to have important functions like the defroster rely on an eMMC chip (or a computer for that matter). " 555. ;-) Report comment Reply 1. Dude says: February 11, 2021 at 11:34 am Why a defroster needs anything more than a simple switch or a relay is beyond me. Report comment 3. John says: February 11, 2021 at 12:20 pm What really boggles my mind is that it seems likely that every engineer would know about write limitations of flash. Even me programming lowly atmel chips knows that emmc and eeprom have write limites. eeprom on the atmel by nature of being tiny will reach that limit extremely fast if being used as some sort of watch dog timer ram, ie chip wakes up, checks eeprom, upon calibration or recalibration writes to eeprom, watch dog timer hits, chip checks eeprom, sees incorrect shutdown flag, pulls last calibration if within time limit. The eeprom is rated to 100K writes so given large writes pre cycle and an eeprom size of 1KB you can kill the chip rather quickly. Flash might only have 1000 cycles. I took this into consideration for another project. What is more likely is that the culture at Tesla so stressed that considerations like these were not made. Hard to say, dev teams can get into very nasty political modes where certain devs refuse to take criticism and due to seniority will merge garbage code. Report comment Reply 1. Dude says: February 11, 2021 at 12:34 pm When Elon says "jump", you just jump - or you're fired. Report comment 2. cb88 says: February 11, 2021 at 1:38 pm Probably system level design presumed that logs would be logged to tmpfs, and uploaded to the cloud... its probably an oversight that they are getting written to flash. Perhaps they are being written to flash backed tmpfs.... We already know that Tesla's are logging massive amounts of data from the telemetry .... so there is really no reason for them to be logging anything to flash ever. Report comment 3. Shannon says: February 11, 2021 at 1:45 pm 100k isn't a limit, it's just the guarantee from Atmel, it'll actually handle many more writes. Some of you may remember when this was livestreamed, I think it hit a couple million writes before failing. https://hackaday.com/2010/05/28/ russian-roulette-for-eeprom/ Report comment 2. ev enthusiast says: February 11, 2021 at 12:14 pm consider the vibration, but then again a vibration resistant flash interconnect could have been used. but then path lengths would be longer, but that can easily be worked around.. oh well. hopefully this sort of coverage encourages them to put in 128Gb chips and still intentionally use a fraction of the space along with wear level algorithm balancing the rewrite cycles Report comment Reply 2. stickben says: February 11, 2021 at 7:22 am Well, flash wears, why wouldn't it be a "wear item"? Wear leveling is something that most products incorporate, regardless flash will not last forever. Tesla is just like any car company, the first adopters are the ones that get to find all the bugs. First gen cars are typically riddled with them. Report comment Reply 1. Gravis says: February 11, 2021 at 7:35 am The fact that eventually wears out is not in dispute, what is in dispute is that the chips were ever intended to be replaced. If they had only written to the memory on rare occasion then it could have easily lasted millions of miles of driving. Report comment Reply 1. stickben says: February 11, 2021 at 8:17 am Is the board not replaceable? Did this person have to throw away their car? I would think the case for "replaceable chips" is more often than not - a disposable item. The chip is technically replaceable, but the cost would probably exceed the value of the board. It doesn't make sense from a business standpoint. Yes, its a design flaw. Yes, Tesla should fix the design and the persons car. Don't hold your breath on individually replaceable board level components, it wont happen for a multitude of reasons. Report comment Reply 1. macfuddy says: February 11, 2021 at 8:53 am I think the point is that by calling it a "wear item," Tesla is implying that wearing out was the intended behavior. However, because they didn't make it replaceable, the whole multi-$K mainboard must be replaced when the chip fails. I believe Gravis is saying it's hard to believe that the chip was intended to wear out, not that the chip should be individually replaceable. Report comment Reply 1. stickben says: February 11, 2021 at 9:26 am Yes, I agree. "Wear item" is a cop out to avoid warranty. They should fix peoples cars, and redesign to avoid the issue. Maybe refurbs would be the way to go. Making things fixable by individuals would not work for most people. Hack-a-day readers are comfortable plugging things in, grandma isn't. Report comment 2. Ren says: February 11, 2021 at 9:59 am I'm reminded of Honda back in the early 1990s(?). Their timing belts started breaking after 80K miles. So, their recall was: Update the Owner's Manual to say... Replace the Timing Belt every 60K miles (In spite of that, I do own a Honda) Report comment 2. Andy Dodd says: February 11, 2021 at 1:50 pm There is also the fact that everyone else in the automotive industry is smart enough to NOT hammer their flash memory with logfile writes in production. Everyone else in the industry has been using devices rated for far fewer write cycles with no issues with hitting those limits, with a simple rule: Don't write to flash memory in production unless you absolutely have to. Report comment Reply 3. Alexander Wikstrom says: February 11, 2021 at 7:28 am I find the statement: "Now they are claiming that the eMMC chip, ball-grid soldered to the motherboard, inaccessible without disassembling the dash, and not specifically mentioned in the owner's manual, should be considered a "wear item", and thus should not be subject to such scrutiny." To seem very much inline with how Tesla and other tech companies operates. Obviously nothing can ever actually be "incorrectly made"... One big method of handling flash chips without "wear" becoming as big of an issue is to use something else for actual "day to day" activates, for an example some battery backed RAM that then only writes to flash if our main power source disappears. For an electric car with a 100 kWh battery in it, running a small bank of RAM for months should be fairly trivial after all... Give it a super capacitor and it can quickly save its contents to flash if the battery ever were to suddenly disappear. In this setup the wear on the flash will likely be less than the wear on the car in general. (This is after all fairly common in a lot of other applications where flash storage is used in a high wear environment, like battery backed disk caching on some RAID cards for an example.) Though, if one would like to treat the Flash chip as a wear item, then it shouldn't be soldered to a board, unless it is a daughter board. (Ie, the storage should be sitting in a socket/connector.) Then we have: "The issue is claimed to only effect models built prior to March 2018, which run on an NVIDIA Tegra 3. Later models are based on the Intel Atom, and feature a larger eMMC chip on board. These modules are yet to demonstrate the same failures, and Tesla claim they should not suffer the issue. We'll see." It doesn't really matter that the eMMC chip is larger, it still wears. Only exception is if it is so large that the time for it to wear out becomes similar to the life expectancy of the car. But even then, one generally prefers that a device doesn't get unusable due to a relatively cheap thing breaking, so the eMMC flash should probably be on a connector regardless. (One reason I personally don't like SBC with eMMC flash, to me, it is just future e-wast waiting to happen. Why throw away a whole SBC just because the flash died, though here we usually can just boot of something else...) I though do like Lewin Day's suggestion of keeping the log and other non vital data on a separate storage device that is far more trivially replaced. Report comment Reply 1. Chris (MN) says: February 11, 2021 at 8:19 am It *does* matter how big the flash unit is. For the same write throughput and technology, a larger flash unit will last much longer. Report comment Reply 1. Alexander Wikstrom says: February 11, 2021 at 8:49 am I suggest reading the whole argument, not just the initial part of it. "Only exception is if it is so large that the time for it to wear out becomes similar to the life expectancy of the car." Now, the cars that does have the current eMMC problem has 8GB of Flash, and some of these cars experience the problem after a relatively short time. Doubling the amount of Flash only really doubles the time to failure, nothing more than that. But as I have already stated, it is only a fix if it is expected to last the lifetime of the product. Though, some people might view a car as something that only needs to have a 5 year life time, but if one is more environmentally and economically conscious one would want it to last at least 4-5 times longer than that. Report comment Reply 1. JayCop says: February 11, 2021 at 9:21 am Just going off memory so I could be wrong, but I seem to recall that the logs were only taking up a small portion of that 8GB, the rest being system files. If they were (for example) using 500MB for logs an increase to 16GB would be a 32x improvement. Report comment Reply 1. RW ver 0.0.1 says: February 11, 2021 at 10:48 am Your math is a bit off there, if it's 500MB logs, 7.5GB system files, on a 16GB it can be 7.5GB system files, 8.5GB logs, so 17x Report comment 2. Dude says: February 11, 2021 at 11:46 am Flash memory is erased in blocks, and the wear-levelling algorithm shuffles the blocks around, so updating a single byte can lead to writing anything from 4 kilobytes to half a megabyte on the flash. This is called "write amplification". This is why it's generally a stupid idea to keep logs on a flash device - appending a few bytes to a file repeatedly makes the wear leveling and garbage collection algorithms destroy the chip over time. Report comment 2. Dude says: February 11, 2021 at 11:42 am >Doubling the amount of Flash only really doubles the time to failure Not necessarily, if it's using MLC techniques and larger block sizes to increase capacity. See "Write amplification". Report comment Reply 1. Alexander Wikstrom says: February 11, 2021 at 1:38 pm Yes, but write amplification would be largely the same in both cases. But yes, write amplification can drastically reduce the life expectancy of flash. ie, one can quickly make a drive able to survive 100+ TB of writes through its life die from only a few hundred GB of actual data being stuffed into it. Logging being one such application where one adds in a few hundred bytes at a time every now and then.... Each time shuffling about far more data than the actual content being written just to find room for it. One of the best reasons for why battery backed RAM drives for logging is a wonderful thing. Report comment 2. Ostracus says: February 11, 2021 at 11:34 am "Obviously nothing can ever actually be "incorrectly made"..." You're holding it wrong. Report comment Reply 4. William J. Jackson says: February 11, 2021 at 7:30 am Yes, this was an unforeseen problem that sandbagged Tesla. All they can do is fight their way out of it - hopefully they may be able to get some recourse from the BGA makers?? This is a time and labor issue. Tesla should do a time and motion optimization. There will be a number of different modules. They can set up a central repair depot to repair each type of module and set aside a stock of these to be shipped to the dozens of service bureaux all over. Then they can schedule cars in for repair with known repair modules at hand to minimise the task. A trained crew that does modules can be used. No matter how this is done, it will be a large hit - one they must eat. All they can do is minimise it and supply a loaner if possible. A sad unforeseen event. Sockets would have mitigated this - if only it was predictable? Someone suggested that it could have been anticipated and with changes to the way Linux addresses/refreshes/wear levels - it could have been avoided. Report comment Reply 1. Rcx says: February 11, 2021 at 7:50 am Flash wear is not an unknown quantity: Any professional electronics design which includes it should do (and on the whole does) an analysis of the expected lifetime of the flash. It's fairly easy to do when you have a known write rate to the flash. For almost all embedded automative systems this is likely very well controlled (Tesla claim the write endurance of their flash is well within industry standard: this is true but only half the equation. Most automative systems do not write to their flash continuously, and certainly not at the kind of rates that Tesla does). With embedded linux it's a little more difficult (the naive raspberry pi like approach is not good here), because you're not sure how much data will be written, but it's easy to mitigate by e.g. having a seperate partition or even chip for your root filesystem (kept strictly read-only outside of firmware upgrades), your configuration options (which will generally have a very low write rate), and your logs. Then you make sure your system still functions even if the log partition fails. It's not clear whether Tesla did any initial assesment of the lifetime of the flash, but it is fairly well established that they later exacerbated the wear on the chip as they developed the software and pushed it out to the cars: as their firmware got larger and larger, the space available for wear levelling on the flash got smaller and smaller, and the amount of logged data increased, shortening the lifetime significantly. If the software had stayed the same it might not have been an issue. They clearly either didn't keep an eye on the issue or thought it was not important enough to spend any effort on fixing. They also neglected to ensure the system still could work even if the flash became unwriteable. This was not an unforeseeable issue, in fact anyone with any experience in embedded linux should have considered it. Report comment Reply 1. jack324 says: February 11, 2021 at 11:12 am I fully agree with you. Since the moment I became aware of flash wear, it's been a consideration in anything that uses flash storage that own or build. Even if it's just picking an sd card for some trivial ultra non critical data storage, there's a tiny whisper in the back of my head telling me to keep flash wear in mind. That ENGINNEERS didn't consider it is both astounding and depressing. Report comment Reply 1. Dude says: February 11, 2021 at 11:54 am Engineers probably did consider it, but as with the "autopilot" system, anyone who makes a complaint that could be considered as a showstopper is let go because of "performance reasons". Seriously. There was an article in Forbes back in the day where they interviewed one of the original engineers behind the Autopilot, who tried to raise concerns but was basically fired for speaking out. That was before the first wave of autopilot deaths hit the media. Report comment Reply 2. Ostracus says: February 11, 2021 at 11:37 am "It's not clear whether Tesla did any initial assesment of the lifetime of the flash, but it is fairly well established that they later exacerbated the wear on the chip as they developed the software and pushed it out to the cars: as their firmware got larger and larger, the space available for wear levelling on the flash got smaller and smaller, and the amount of logged data increased, shortening the lifetime significantly. " Wonder if cellphones will run into this as programs get bigger and bigger? Report comment Reply 1. Dude says: February 11, 2021 at 11:57 am The number one reason for replacing a phone of a previous generation is because Android craps out after running out of flash on the phone's main chip. Can't put everything on the SD card. For some reason, even 16 gigabytes is running short these days. Report comment Reply 3. AndreasKlein says: February 11, 2021 at 2:07 pm All the geeks here, should work for Tesla and make milions per year. Report comment Reply 2. Dude says: February 11, 2021 at 11:50 am >an unforeseen problem People did predict that the whole "touchscreen in the middle controlling everything" would lead to trouble with the electronics breaking down, bugging out, or becoming obsolete within the lifespan of the car. Report comment Reply 5. Joel B says: February 11, 2021 at 7:39 am "Nobody would put up with swapping out their entire front suspension setup every time their brakes wore out.." No, but don't put it past a car maker to do something equally as stupid. The lower control arm bushings on my Civic are cast into the control arm. Want to replace the bushing? Replace the entire part. I can get why they do it, but what a waste. Report comment Reply 1. Wolf says: February 11, 2021 at 7:57 am On my 2013 Ford Escape, replacing the battery. Step 1. Remove the windshield wipers: https://youtu.be/-lJiRmC9igc Report comment Reply 1. Rick says: February 11, 2021 at 8:52 am I started looking into changing the battery on my '14 Escape and I noped out of it and had the dealership do it during a scheduled maintenance. Ever try adding coolant to the reservoir? Report comment Reply 1. Escalion says: February 11, 2021 at 9:06 am I had the Turbo fail on my '16 Renault Trafic. I noped out at the point I had remobed the exhaust, sub frame, steering rack and DPF. By this point I could get to the Turbo but wished I hadn't. Report comment Reply 2. zeldatp151 says: February 11, 2021 at 11:43 am When I had a Chevy HHR, replacing the bulbs for the headlights meant taking off the front wheels and removing the plastic shroud around the wheel wells. After the first time, I decided to cut a square our of the plastic shroud, and put little door with a hinge in its place so in the future I could just turn the wheel and reach in with my arm to change it. Report comment Reply 1. Ostracus says: February 11, 2021 at 1:13 pm Wonder if aftermarket parts correct for these oversights? Report comment 2. RuneTorgersen says: February 11, 2021 at 9:07 am Changed battery in a '14 Escape - took about 2 hours, as you have to remove either the Air Filter Assembly (and various other pieces), or (easier) the window cowl to get the battery out.... Report comment Reply 3. RW ver 0.0.1 says: February 11, 2021 at 9:18 am So if you're Canadian, starting to worry about your battery in a 2013 up Escape, get CAA... they will replace your battery for the price of the battery, hint; if you wanna keep the car 5 years, get the best they've got. The $74 for a years membership sounds like it might be cheaper than the shop rate. Don't know if that will work for AAA too. Report comment Reply 2. RetepV says: February 11, 2021 at 8:41 am Look at modern motorcycle engines. The cylinder assembly is a fixed part of the upper engine casting. And the cylinders themselves are Nikasil-coated aluminium and also a fixed part of the upper engine casing. Non-replacable. If the cylinders wear out, you can bore them and recoat them about 3 times (and buy larger pistons every time). But after that, all that's left is to throw away the whole engine block. With a dirtbike, with one cylinder, I guess that's not too bad. Only one cylinder to bore, and only one piston to replace. But modern 4-cylinder motorbikes all have such an engine. Report comment Reply 3. Donald Trump says: February 11, 2021 at 11:39 am Made of Aluminum? I've replaced suspension bushing with polyurethane on 3 cars by now. Including a Civic. It's always a pain, but worth it. Aluminum has a fatigue life of about 1 million full strains, 10 million 10% strains, depending on alloy and manufacturing. Making the bushings non-replaceable is the manufactures way of preventing the aluminum part from breaking in half as you roll down the road. Report comment Reply 6. Grawp says: February 11, 2021 at 7:39 am In the first company I worked for we had products based on several SBCs (RPi1 among them). We did use SD cards but they were completely read-only except for system updates. It is not so hard to build such system using Buildroot but it is not completely easy because some GNU/Linux components like to non-optionally require access to a mutable storage and even if you use tmpfs you have make sure that it doesn't run out of memory... Report comment Reply 1. Alex W says: February 11, 2021 at 7:47 am This is exactly the most annoying aspect of this whole thing: almost everyone who's done serious SBC development knows not to fall into this trap. "You'll wear out your flash!" is almost as common a critique as "3D prints aren't food safe!" To simply not have bothered implementing the design correctly is unforgivable. Report comment Reply 1. Chris (MN) says: February 11, 2021 at 8:18 am The major eMMC problems were with early cars. I don't totally blame them (although they shouldn't act so huffy about it *now*) as I also installed a bunch of embedded linux systems that used flash and eventually had to have their flash units replaced for the same reason. At the time (a decade ago) it wasn't quite as understood (I mean by the broader IT/linux community) as it is now how crucial it is to handle logging, etc, properly. They DIDN'T have to make it so hard to replace, though... Report comment Reply 2. X says: February 11, 2021 at 8:20 am Using that standard we would have no cars, no computers, no houses, no nuthin. Humans are not perfect, we don't make perfect things and it is foolish to expect otherwise. Report comment Reply 1. Craig Hildreth says: February 11, 2021 at 9:05 am This is like building a house with no breaker box. We'll just replace all the wiring when it fries. Report comment Reply 3. Foldi-One says: February 11, 2021 at 8:32 am Its probably that age old but very simple issue of the E.Engineers and Software folks not singing from the same hymn sheet - the software human is going, wow we have so many sensors, and so much data, we should log it all, often, to help with our self driving (perhaps), or maybe to prove how well our brake pad last, etc. But the system was built and designed assuming less insane levels of logging, so much longer than the expected lifespan of the car to wear out through the software updates. Report comment Reply 1. Ostracus says: February 11, 2021 at 11:40 am Cloud, cloud, cloud, and throw in some ubiquitous 5G. Solve one's logging problems. Report comment Reply 2. arcol says: February 11, 2021 at 2:22 pm > We did use SD cards but they were completely read-only except for system updates. It is not so hard to build such system using Buildroot I just name one thing: systemd and its companions (hostnamed, etc). I could not figure out how to set up hostname on a network booted RPI. Some systemd components just *WANT* to have access to the filesystem, you can not masque it away using tmpfs. Report comment Reply 7. Jamie says: February 11, 2021 at 7:47 am Surely they could just cache any writes to the flash in RAM until a full power down event occurs? Even with an unexpected power failure you could probably store enough energy in a capacitor to write the changes to flash before every last watt of juice is dissipated? Report comment Reply 1. andarb says: February 11, 2021 at 10:33 am That would require the foresight they seem to have missed. That being said, it could easily have been an engineer saying "It'll be fine" when someone pointed out the concern. Or a manager saying "It'll be fine" when an engineer pointed out the concern. (I pointed out the concern in a product and was told "It'll be fine") Report comment Reply 1. Dude says: February 11, 2021 at 12:09 pm When the upper management is a bunch of hacks who try to "cheat" their way through everything and do the minimum to have something happen, the engineers lower down start to only do what is required of them and nothing more. When the management does not want to hear what the engineer have to say, the engineers have to play stupid and leave out all the supporting features and "scaffolding" that would actually be required for the thing to work. In other words, they make a hack out of the new feature request, and the product becomes technologically fragile - bodges on bodges. Report comment Reply 8. M.2 SADA SSD Cards and not eMMC cards says: February 11, 2021 at 8:02 am Going to point out the Raspberry Pi can now use a third party Argon One M.2 Case that adds things like, you know, a POWER BUTTON and most importantly things like the ability to actually operate an M.2 SATA SSD Compatibility style drive. Accepts any size of M.2 SATA SSD with Key-B or Key-B&M. Actually uses USB 3 so it's also much quicker as well. Using simply a micro SD as the main "hard drive" for the Raspberry Pi or Tesla for that matter is not a good idea for long term use. Even if it's a halfway decent quality micro SD card. Already had several break in various ways as it is. This isn't exactly something they are just now experiencing either. Report comment Reply 9. RetepV says: February 11, 2021 at 8:30 am If you use a flash/eMMC, or whatever similar thing like that, as storage, your software has to take into account that you can only rewrite cells a finite amount before they will start to fail. Memory caching, bad sector management, distributing writes evenly over all the sectors, and plain simply not writing so much to it in the first place. When I worked at TomTom, years ago, it was also a big issue then. All fixed by changing the strategies of what and how to write to the flash memory. Report comment Reply 10. RW ver 0.0.1 says: February 11, 2021 at 8:43 am Just wondering how hot you can get these things before they fall off the board, or start moving. Whether it uses higher temp solder because automotive or not. It's known that flash memory can be re-annealed by heating it to 250C "for several hours". I think that's too high, but wondering if it's a dose/time thing, such that getting it to 170C maybe and leaving it overnight at 170 might have an effect. The object being that you can make a clip with a heater in it, that maintains 170C and just snake your arm up the dash with that, clip on edge of board over the chip, plug it in and leave it. Potentially a much cheaper repair. Report comment Reply 1. tekkieneet says: February 11, 2021 at 8:53 am Considering the common reflow temperature peaks at around 240C/260C for lead and RoHS solder, the annealing is enough to desolder them. It is tricky to make reliable vibration proof connectors that also have to handle a large temperature cycling. Don't expect commercial grade sockets to be used without having it goes through the shake & bake cycles. i.e. forget the usual FLASH cards etc. Report comment Reply 1. arcol says: February 11, 2021 at 2:27 pm they could have solder it to the board with and adaptor pcb, just like how any esp module is made. So the problem is now only a soldering iron and not a reflow station. Report comment Reply 2. tekkieneet says: February 11, 2021 at 10:06 am BTW Your PCB will turn brown and deliminate as they can't handle high temperature for a long duration. PCB Glass Point (Tg) classifications: Low Tg: around 130degC Middle Tg: >=150degC High Tg: >=170degC Report comment Reply 1. Padrote says: February 11, 2021 at 2:50 pm Huh? Tg doesn't apply to cured epoxies... Report comment Reply 3. Donald Trump says: February 11, 2021 at 11:46 am The final nail in the 'wear item' claim should be pointing out the melt temp in the solder used on the flash. Bet it _isn't_ the lowest melt point solder on the board. I suggest you try converting your cars dash into a reflow oven and tell us how it goes. What's the flashpoint of under car dash smegma? Report comment Reply 11. CM says: February 11, 2021 at 9:07 am This may be a dumb question, but why couldn't the firmware and hardware be updated during a TSB campaign? Update firmware to write to an SSD on a daughterboard over JTAG? As long as the cabling is sufficient length, the daughterboard could be installed in a more accessible location. Report comment Reply 1. tekkieneet says: February 11, 2021 at 10:03 am Why bother with JTAG? JTAG is a Test Interface not a general purpose bus nor it means FLASHing. A microcontroller/board with a JTAG interface means that it is a slave and cannot toggle JTAG pins. Report comment Reply 12. dlcarrier says: February 11, 2021 at 9:52 am The eMMC is a wear item, but Tesla's architecture is destroying that item. It's akin to having a brake that is stuck partially on, on every vehicle, except in this case, the brakes would should a few million miles, but the defect would suddenly and without warning ware them out in the tens of thousands of miles. A recall would need to replace the brakes and fix the issue with them being stuck partially on. Report comment Reply 13. Tesla eMMC issue says: February 11, 2021 at 10:16 am Funnily enough my Model S 2016 is in the shop tomorrow to rectify this very issue. I received an alert email that Tesla would replace the component for free back in November as kind've a warranty extension, noted it because I heard of the issue originally from RichRebuilds YT channel. Then lo and behold almost by fate, in December the alert popped up on my display, storage device degraded or something. Booked it in for repair even before the recall was announced so I've gotten in there before the mad rush! Getting a courtesy car and everything. I have noticed only a few minor strange goings on in the car, notably some Spotify album tracks cached on the storage device are corrupted and never play, the media player freezes up for a minute or so. For whatever reason the car is not downloading fresh files to fix the issue. Aside from that I've never been denied control of any aspect of the car, air con, etc. But rarely I have had to do the two finger salute to reboot the centre console, but even then you can still drive the car. Had to do this once when it stopped playing the clicking sound when the turn indicators are active. Was driving at the time so I had no idea whether the indicators were actually working or not. Either way quick reboot had it up and running in a few minutes whilst I still drove it. All in all I cannot complain. Report comment Reply 1. Donald Trump says: February 11, 2021 at 11:56 am Are they going to replace it every 4 years, do they have a real fix or is the next owner screwed? BTW just some history. Mopar had an engine where the ECU _was_ a wear part. 2.2 four banger in ever POS in the late 80s/90s, the ECU would be destroyed when the oxygen sensor went bad. Also every car ECU with chinesium electrolytic caps from slightly later. Would have hoped Testa could do better, but here we are. Report comment Reply 2. Dude says: February 11, 2021 at 12:14 pm >All in all I cannot complain. >it stopped playing the clicking sound when the turn indicators are active That's such a basic (safety) feature that one cannot but facepalm at the lengths you are willing to go through to justify your purchase and excuse the company for getting simple things wrong. Report comment Reply 1. Dude says: February 11, 2021 at 12:19 pm I mean, if it was a $18,500 Toyota, and the dashboard indicators were failing because the center console software was bugging out, I would just go NOPE and return it to dealership for a full refund. No way I'm gonna touch that pile of bull**** - it can only spell trouble later on; endless garage bills and fighting the company on fixing issues that shouldn't be issues in the first place. But when it's a $100k luxury car, you just deal with it? Report comment Reply 14. Vinalon says: February 11, 2021 at 10:51 am Flash makers, not BGA makers. BGA refers to the way that the Flash was packaged, in a plastic shell with a "Ball Grid Array" of solder bumps underneath. And Flash write endurance is very well-understood. Most junior embedded engineers would tell you that it is a terrible idea to frequently write rapidly-refreshing data to Flash memory. Tesla was not "sandbagged", and the have no recourse with the manufacturers because the products are working as designed. Report comment Reply 15. Olivier says: February 11, 2021 at 10:54 am Ha. Cant say i'm surprised. Im afraid all Tesla owners should prepare for more recalls in the next few years. Report comment Reply 16. h2odragon says: February 11, 2021 at 10:56 am So they can pull a Ford; and go "we're so sorry we screwed up here." Give all affected owners a new car at some fair discount, keep the old ones to study for wear / re manufacture and re-use parts, and just keep them out of the junkyards and secondary market. Given the financial conditions over the last few years it could even get booked as a good investment I bet. They'll bend the numbers to whatever end is needed. Report comment Reply 17. Kirby says: February 11, 2021 at 11:06 am I wonder if Tesla took their "wear item" context from Microsoft. I say this, as a buddy found a major bug in Windows and given that they were a big company wanted Microsoft to update/patch the bug. Microsoft reviewed the issue and came back and instead of calling it a bug they called it a "feature" and so they planned to do nothing about it. Now, the "feature" would cause a blue screen under some pretty routine steps. My buddy did not blink an eye... and simply stated that their firm was planning a news conference and inviting everyone so that they could show and demonstrate this "feature"... low and behold Microsoft decided to fix the bug/feature and nothing more was said. Report comment Reply 1. Jul13 says: February 11, 2021 at 12:32 pm Nothing like doing volunteer work to increase the profits of millionaires. I'll never understand some people Report comment Reply 18. Chris T. says: February 11, 2021 at 11:06 am As a wear item, it should be easily field replaceable. Requiring disassembly of the dash of the car for a "normal wear item" isn't practical. Here comes the era of throwaway cars. Report comment Reply 1. arcol says: February 11, 2021 at 2:33 pm > Here comes the era of throwaway cars. With the glued in battery (the battery itself is a structural part of the car frame) we get there, no worries. Just like how is it done in cellphones these days. The recipe is known and tested. (lighter car, faster production, heck even safer, so it is a win-win for everyone, no?) Report comment Reply 19. Erik T says: February 11, 2021 at 11:11 am Man, 8+ years ago I was putting SSDs into industrial PCs (they weren't a standard item you could just order from our supplier) and overprovisioning them by like 90%. 8 GB partition on a 64 GB SSD sorta thing. Hey, they all still work fine! Report comment Reply 20. Jerry says: February 11, 2021 at 11:20 am Fast Forward to 14 Minutes. Report comment Reply 1. Dude says: February 11, 2021 at 12:27 pm All that is moot, because the car will be in fire in less than 5 minutes from a crash. That was the basis of the lawsuit that Tesla settled out of court with money; when enough damage is done to the car that the battery is compromised, there is little chance the rescuers will get to you before the thing catches fire, and the flames will be unstoppable without killing the occupants of the car in the attempt. Report comment Reply 21. Per Jensen says: February 11, 2021 at 11:33 am The eMMC /does/ have wear leveling. What Tesla fucked up is writing several GIGABYTES of log data from the vehicle sensors/ camera every single day. Writing such big amount of to the eMMC is bad. It HAD to go wrong. Such a n00b mistake to make. Report comment Reply 22. Bret Tschacher says: February 11, 2021 at 1:44 pm Space shuttles and O-rings, Toyota and throttle bodies that would fail and go full throttle, Tesla's and eMMC's, just a small sample of how an object can have such a high rated value and yet be nothing more than another POS. Anyone remember the sales slogan for Zenith appliances? The quality goes in before the Name goes on. Report comment Reply 23. DainBramage says: February 11, 2021 at 2:45 pm Careful, Hackaday. Tesla will sue you out of existence for having the nerve to say anything negative about their silly toy cars, no matter how factual it is. Heck, they even managed to shut Jeremy Clarkson up! Report comment Reply Leave a Reply Cancel reply Please be kind and respectful to help make the comments section excellent. (Comment Policy) This site uses Akismet to reduce spam. Learn how your comment data is processed. Search Search for: [ ] [Search] Never miss a hack Follow on facebook Follow on twitter Follow on youtube Follow on rss Contact us Subscribe [ ] [ ] [Subscribe] If you missed it * [Insect-thu] Would You Like Fries With Your Insect Burger, Ma'am? 150 Comments * [1940-thumb] The Modding, Restoration, And Demise Of A $3M Analog Computer 17 Comments * [DSCF2233_t] Hands-On: The RISC-V ESP32-C3 Will Be Your New ESP8266 57 Comments * [phone-magn] Using Your Phone As A Microscope On The Electronics Workbench 23 Comments * [Sewage_thu] Sewage Maceration Is As Gross As It Sounds 98 Comments More from this category Our Columns * [eMMC_thumb] Tesla Recalls Cars With EMMC Failures, Calls Part A 'Wear Item' 86 Comments * [50ham_thum] The $50 Ham: Digital Modes With WSJT-X 19 Comments * [bugbulb_th] Teardown: Bug Zapper Bulb 13 Comments * [7mins_thum] Getting Ready For Mars: The Seven Minutes Of Terror 46 Comments * [zack-freed] Finishing Your Projects Hack Chat With Zack Freedman 14 Comments More from this category Search Search for: [ ] [Search] Never miss a hack Follow on facebook Follow on twitter Follow on youtube Follow on rss Contact us Subscribe [ ] [ ] [Subscribe] If you missed it * [Insect-thu] Would You Like Fries With Your Insect Burger, Ma'am? 150 Comments * [1940-thumb] The Modding, Restoration, And Demise Of A $3M Analog Computer 17 Comments * [DSCF2233_t] Hands-On: The RISC-V ESP32-C3 Will Be Your New ESP8266 57 Comments * [phone-magn] Using Your Phone As A Microscope On The Electronics Workbench 23 Comments * [Sewage_thu] Sewage Maceration Is As Gross As It Sounds 98 Comments More from this category Categories Categories[Select Category ] Our Columns * [eMMC_thumb] Tesla Recalls Cars With EMMC Failures, Calls Part A 'Wear Item' 86 Comments * [50ham_thum] The $50 Ham: Digital Modes With WSJT-X 19 Comments * [bugbulb_th] Teardown: Bug Zapper Bulb 13 Comments * [7mins_thum] Getting Ready For Mars: The Seven Minutes Of Terror 46 Comments * [zack-freed] Finishing Your Projects Hack Chat With Zack Freedman 14 Comments More from this category Recent comments * Padrote on Tesla Recalls Cars With EMMC Failures, Calls Part A 'Wear Item' * DainBramage on Tesla Recalls Cars With EMMC Failures, Calls Part A 'Wear Item' * Bren on Restored Dreamcast Is A SEGA Fan's Dream Come True * Bren on Restored Dreamcast Is A SEGA Fan's Dream Come True * arcol on Tesla Recalls Cars With EMMC Failures, Calls Part A 'Wear Item' * Mr Name Required on Vacuum Forming Key Cap Covers Doesn't Quite Work Out * Scoldog on Restored Dreamcast Is A SEGA Fan's Dream Come True * arcol on Tesla Recalls Cars With EMMC Failures, Calls Part A 'Wear Item' * arcol on Tesla Recalls Cars With EMMC Failures, Calls Part A 'Wear Item' * Tweepy on AOI By DIY Now on Hackaday.io * ekaggrat singh kalsi wrote a comment on Dual-Disc Polar Printer. * charles5688 liked Bloodhound: Autonomous Radiolocation Drone. * charles5688 liked eMBee ONE Pocket Computer. * Dave B liked Animatronic LEGO Lunar Lander. * B.B started following Finishing Your Projects Hack Chat. * matop wrote a reply on Build your own TurtleBot 3 backbone. * John liked Flip Dot Display. * rchadwick7 liked DIY AOI RGB Light. * androSID wrote a reply on MockA65xx - Universal 6502/85xx CPU replacement. * bobricius wrote a reply on mini (Pi)QWERTY USB keyboard II.. Logo * Home * Blog * Hackaday.io * Tindie * Hackaday Prize * Video * Submit A Tip * About * Contact Us Never miss a hack Follow on facebook Follow on twitter Follow on youtube Follow on rss Contact us Subscribe to Newsletter [ ] [ ] [Subscribe] Copyright (c) 2021 | Hackaday, Hack A Day, and the Skull and Wrenches Logo are Trademarks of Hackaday.com | Privacy Policy | Terms of Service Powered by WordPress.com VIP [impression] [close] By using our website and services, you expressly agree to the placement of our performance, functionality and advertising cookies. Learn more OK