Reprinted from TidBITS by permission; reuse governed by Creative Commons license BY-NC-ND 3.0. TidBITS has offered years of thoughtful commentary on Apple and Internet topics. For free email subscriptions and access to the entire TidBITS archive, visit http://www.tidbits.com/ Six Lessons Learned from Dealing with an iMac's Dead SSD Adam Engst There it was, Wednesday evening, and I was working hard to finish something for the day. Then I received an email from [1]Watchman Monitoring, an essential tool used by sysadmins and consultants to keep track of Macs under their care, telling me about disk errors on my 2014 27-inch iMac's SSD. That was concerning, so I finished what I was doing, restarted in macOS Recovery by rebooting while holding down Command-R, launched Disk Utility, started First Aid, went to make dinner, and promptly forgot about it. The next morning, however, I checked the iMac and discovered that First Aid had failed with an error -69842. Another try (which took longer than it should have) failed with the same error. I could find no indication of what that error meant, but Apple's [2]support documentation was pretty clear about the next step being a reformat and restore. At some point in that process, I booted the iMac successfully again, just long enough for Watchman Monitoring to send an even more ominous warning. I wasn't terribly perturbed by all this, since I knew I had both a bootable duplicate from the night before and a Time Machine backup that had been working until I quit for the day, both on an external hard disk, not to mention Backblaze Internet backups. Plus, I could work on my 2012 13-inch MacBook Air during the lengthy restore process. This is the first time I've had to reformat an SSD, and I didn't know if reformatting would map out bad sectors as it does with hard disks, but it seemed worth a try. But I started to get more worried when Disk Utility threw an error while trying to erase the boot volume (the iMac was running macOS 10.14 Mojave, so it didn't have 10.15 Catalina's bifurcated drive structure). However, when I set the View pop-up menu in Disk Utility to Show All Devices and then erased the actual drive, it succeeded. While still in macOS Recovery, I quit Disk Utility and started to restore from my Time Machine backup. I knew this was going to take a lot of time, and the initial 4-hour estimate kept going up until it was at about 10 hours. After dinner, I checked on it, and it was 61.6% done, which seemed slow, and when I checked several hours later, it was still at 61.6%. Not good. After I forcibly restarted the Mac, it wouldn't even boot into macOS Recovery, instead loading Internet Recovery. That was the first suggestion that the SSD was dead, and indeed, once I could look at the setup with Disk Utility again, it was missing entirely and hasn't shown up since. [3]It's an ex-parrot. Because I'm paranoid that way, I booted from my bootable duplicate (which worked, thankfully), and then set it to update yet another bootable duplicate backup I have on yet another hard disk. Then I went to bed'it's always best to tackle major problems when rested. The next morning, I pondered my short-, medium-, and long-term options. The 'I need to get my work done' Short-Term Solution Although I can get my work done on the MacBook Air, its 13-inch screen running at 1440-by-900 pixels is limiting, especially because I'm accustomed to working on the 27-inch iMac next to a 27-inch Thunderbolt Display. My normal desktop is 5120 by 1440 pixels. Plus, while I don't mind working on a laptop for short periods, the ergonomics of laptop use are horrible, and it's bad for my body to put in my normal work hours on a laptop. The obvious solution was to run my iMac from my bootable duplicate. But this proved problematic. The 2014 iMac supports USB 3.0 and Thunderbolt 2, and my Thunderbolt Display adds a FireWire 800 port. The theoretical speeds (in gigabits per second and converted to the common megabytes per seconds for storage devices) on those are: * Thunderbolt 2: 20 Gbps, which works out to 2500 MBps * USB 3.0: 5 Gbps, or 625 MBps * FireWire 800: 800 Mbps, or 100 MBps My backup drive is [4]a 2.5-inch Seagate 2 TB FireCuda Gaming drive in an [5]OWC Mercury Elite Pro mini external enclosure. The drive supposedly has 128 MB of cache memory, but it's only 5400 rpm, and while its speed has never been a noticeable problem while making backups, it's painful beyond belief to use as a boot drive. USB 3.0 might theoretically be capable of 625 MBps, but when I tested the drive in Blackmagic Speed Test, it averaged just 20'25 MBps for both read and write speeds. Ouch. I didn't think of switching to FireWire at the time, but interestingly, it was slightly better at 25'30 MBps for the same drive. That leads me to: Lesson #1: A bootable duplicate should be fast enough to use as your boot drive, or it's essentially a read-only backup like Time Machine. For comparison, the internal SSD on Tonya's identical 2014 27-inch iMac laid down read speeds of about 750 MBps and write speeds of about 500 MBps. You can see why the hard disk was unacceptable, given that it was 20 to 30 times slower. The solution I ended up implementing eliminated the performance problem. I restarted my MacBook Air in [6]Target Disk Mode by holding down the T key at startup, and then I connected it to the iMac using Thunderbolt 2 and booted from it. Startup was slow but acceptable, and once the iMac was running, performance wasn't noticeably different than normal. Keep in mind that I do very little that stresses the drive. Blackmagic Speed Test showed read and write speeds of about 120 MBps, or 5 to 6 times faster than the hard disk. Interestingly, these numbers were quite a bit below what I got when I eliminated Thunderbolt 2 from the equation and tested the MacBook Air's SSD on its own. So Thunderbolt 2 must be slower than the MacBook Air's internal bus. Although this approach got me up and running, it suffered from a few other problems. Most annoyingly, iCloud and quite a few apps tie their credentials to the internal hardware of the Mac, so switching from it being a standalone MacBook Air to being a boot drive for an iMac caused iCloud and those apps to ask for authentication every time I wanted to move between machines. For reasons I never figured out, even my extensions in Brave disappeared each time I swapped Mac brains underneath the same storage device. Plus, that MacBook Air has been kernel panicking in sleep for some time, something I had previously chalked up to a dodgy Catalina install but that I now realize is related to failing hardware, perhaps with a thermal trigger, since it only happens when the case is closed. I now came up with: Lesson #2: Target Disk Mode is useful only for occasional troubleshooting and emergency use, not as an everyday driver. (I had briefly considered replacing my MacBook Air with a new model, which I've been planning to do soon anyway, and using it to boot the iMac. But the awkwardness of Target Disk Mode made that unattractive. Worse, a test using Tonya's 2016 MacBook Pro failed'I could put it into Target Disk Mode, but the iMac never saw it over USB. It might have worked over Thunderbolt, but I didn't have a Thunderbolt 2 to Thunderbolt 3 adapter.) Regardless, booting from the MacBook Air in Target Disk Mode meant that I could work effectively on my big screens, and I could even access files and apps that existed only on my bootable duplicate. (My iMac's SSD was 500 GB, whereas the MacBook Air is only 250 GB, so the two have similar setups in key ways, but they aren't identical.) The 'I can't work like this indefinitely' Medium-Term Solution Between the split personality problem every time I connected or disconnected the MacBook Air, the slightly inadequate setup, and the kernel panics, I needed a better solution for the next few weeks or months. Once again, there was an obvious solution'replace the internal SSD in the iMac with a new one'and once again, I didn't take it. It was the cheapest and cleanest solution, but it also came with the most downtime and the most risk. I'm probably capable of cutting the screen off a 27-inch iMac'yes, you have to cut through the adhesive that holds it on'replacing the SSD, and reattaching the screen, but there's always a chance of making a fatal (to the iMac) mistake. Worse, in [7]the (extremely useful) TidBITS Talk conversation on the topics, my buddy [8]Ron Risley told a story about how he used [9]OWC's Hard Drive Upgrade Kit to replace the drive in a 27-inch Retina Mac only to have the re-adhered screen fall off. (I may still go down this path since I've now taken over another 2014 27-inch iMac that TidBITS bought for Josh Centers and replaced just a few months ago. He had been having trouble with it'specifically with its Fusion Drive'for some time, so I might open up that iMac and replace its Fusion Drive with an SSD to see if that resolves its problems. If so, I'd know how to do it for my iMac, and then I'd have two restored iMacs!) There is a local repair shop that's open and would probably do the work. Still, between not having any experience with them, not wanting to leave the house during New York's stay-at-home order, the potential for several days of downtime, and the risk that they'd mess something up, I decided against that option too. (Having Apple repair the iMac isn't an option, since Apple retail stores are closed, and none are within an hour drive anyway. There is an Apple Authorized Service Provider in Ithaca, but they're doing repairs only for essential businesses, and we haven't been impressed with them in the past anyway.) I also eliminated the second-most-obvious solution, which I'll discuss shortly, and decided instead to buy an external SSD to boot the iMac. Since the MacBook Air's performance in Target Disk Mode over Thunderbolt 2 was acceptable and still below what USB 3.0 should be able to do, I figured that an external SSD would be as good or better. I could also get a 1 TB model that would relieve the space pressure I had been feeling and would let me move my Photos library back to the SSD from its hard disk exile. The main question here was how much to spend. * The cheapest option was to buy a bare SSD and use a toaster case I have around, but it's sufficiently old that I worried that its controller might not be able to handle the SSD or that its performance might be lousy. It looked like I could get a bare 1 TB SSD for about $100'$150 after tax. * The 'just-right' option was to buy a SATA III SSD in a USB 3.1 Gen 2 case. That variant of USB can theoretically do 10 Gbps, or 1250 MBps, but the SATA III drive interface runs only at 6 Gbps and can provide about 600 MBps of throughput. Prices were around $200 after tax. * The most expensive option was to buy a PCIe SSD in a Thunderbolt 3 case, which would run $300'$400. I'm a little fuzzy on the details here, but PCIe SSDs rely on a communications interface called NVMe, and the upshot is significantly better performance. With a 40 Gbps Thunderbolt 3 interface, an NVMe SSD can do as much as 2400 MBps read and 1800 MBps write. I'd be limited to half that due to having only Thunderbolt 2, but the bigger problem is that it seems that Apple's Thunderbolt 2 to Thunderbolt 3 adapter (which costs another $49) doesn't work with bus-powered Thunderbolt 3 devices. Augh! Lesson #3: When you keep a Mac as long as possible, you can run into connectivity or performance issues when buying new peripherals. Plan ahead with a full collection of adapters. As you might guess, I went for the 'just-right' option and chose the popular [10]Samsung T5 external SSD. It's tiny, has a single USB-C port that supports USB 3.1 Gen 2, comes with the necessary cable to attach it to my iMac's USB-A ports, and costs $200 after tax. I ordered it directly from Samsung to avoid Amazon shipping delays and got it two days later. The Long Term Solution Remember the second-most-obvious solution above? That was to slap a new 27-inch iMac on my Apple Card and move on. I've been starting to think about replacing my iMac anyway, and while it was tempting to use this hardware failure as an excuse, Apple hasn't updated the iMac in over a year, which suggests that now is not the best time to buy. Regardless, at some point before the end of the year, I'll probably replace the iMac, either with a new 27-inch iMac because I love its 5K Retina screen, or with a Mac mini coupled with a new class of curved 49-inch screens running at 5120-by-1440 pixels from the likes of [11]Dell, [12]Philips, and [13]Samsung. These so-called QHD monitors aren't as high quality as would be ideal, having a pixel density of only 109 ppi versus the 218 ppi of the iMac's 5K display and the LG UltraFine 5K Display, but they're priced similarly to the LG screen for a lot more real estate. Oh, the anticipation! Restoring Isn't Always as Easy as It Should Be I thought I was home free once the Samsung T5 arrived, but more obstacles remained. As I started down the path of restoring my data to the Samsung T5, I found myself in a weird set of interlocking Catch-22s caused by not having kept my hardware and software up with the times. Although my MacBook Air has been running Catalina fine since it was released, I hadn't yet upgraded the iMac from Mojave. Why was this a problem? I had reformatted the Samsung T5 from ExFAT to APFS, which is appropriate for Mojave when running on an SSD. I know that the macOS installer creates a Recovery partition during the installation process, but what about when restoring with SuperDuper? I was worried that SuperDuper would have no way to know it should create a Recovery partition when moving an HFS+ drive to an APFS drive. (That was a guess. When moving 360 GB around, you don't want to guess wrong and have to waste another day repeating the copy. However, I subsequently talked with Dave Nanian of Shirt Pocket, and he set me straight by pointing me to [14]this blog post from 2017. SuperDuper will back up the Recovery partition whether the source drive is HFS+ or APFS, and whether the backup drive is HFS+ or APFS. When restoring to APFS, SuperDuper does the right thing as well, creating and restoring to the Recovery partition. When restoring to an HFS+ drive, SuperDuper can restore to an existing Recovery partition if one exists, but if not, you have to reinstall macOS so Apple's installer can create it properly for you. Since I had formatted the Samsung T5 as APFS, all would have been fine. Live and learn!) Not realizing that SuperDuper would have worked, I decided I needed to reinstall macOS, which is often a good idea anyway. I initially tried to reinstall Mojave, which was what I had been running, but that proved problematic. I had a copy of the Mojave installer available, but when I booted using the MacBook Air in Target Disk Mode, it was running Catalina, and wouldn't let me install Mojave because it was too old. When I booted using the version of Mojave on the bootable duplicate hard disk, I was told that the Mojave installer was damaged. (I have no reason to believe that's true, but I couldn't really argue with it. It could have been related to the expired certificate problem'see '[15]Redownload Archived macOS Installers to Address Expired Certificates,' 28 October 2019). So much for my local installer. Lesson #4: Check your archived installers periodically; don't assume they'll necessarily work. Back to macOS Recovery. However, because the Recovery partition on the internal SSD was no longer available, booting with Command-R held down put me in Internet Recovery, which could take up to 10'20 minutes. After that wait, using the Reinstall macOS option presented me with 10.10 Yosemite, which was the version of macOS that came with the iMac. For time reasons alone, I didn't want to install Yosemite and then have to upgrade, and I wasn't entirely sure how I could upgrade to Mojave anyway, since Software Update would try to give me Catalina. Eventually, I used Command-Option-R to boot into Internet Recovery and get the latest version of macOS for my iMac, which was indeed Catalina. I had been planning to upgrade to Catalina after the release of 10.15.4 anyway and had been putting it off, so being forced into the upgrade didn't bother me too much. I had been thinking I'd do it separately from the switch to the Samsung T5 to reduce the number of variables if I encountered any problems, but that wasn't in the cards. Lesson #5: You can put off macOS upgrades for a while, but something will ultimately force you to upgrade. It's better to be ready for that eventuality than to pretend that it can be avoided forever. (What may not be clear from my retelling of the story above is just how maddening this process was, since booting from the MacBook Air brought with it cabling swaps and authentication resets, booting from the hard disk's bootable duplicate took nearly 6 minutes, and booting from Internet Recovery could take 10'20 minutes. I probably spent several hours just rebooting the iMac repeatedly in an effort to get to the point where I could reinstall macOS.) In the end, installing Catalina worked as it should, and I was able to restore the entire contents of my bootable duplicate using Setup Assistant. It took hours, although how many I don't know since I wasn't watching. One question I was unable to answer, but which I'd be curious to test, was whether it would be significantly faster or slower to restore from Time Machine rather than the bootable duplicate. After installation, the first few boots were pretty rough because I have quite a few apps that launch at startup. Until I went into Keychain Access, selected my 'adam' keychain, and chose File > Make Keychain 'adam' Default, I was deluged with dialogs requesting my keychain password. Eventually, though, I managed to log in to every app and get things working again. Lesson #6: Reinstalling macOS and restoring from a backup may not leave your Mac in an entirely functional state. Be prepared for additional work after restoring. I've had only a few days with the Samsung T5 as my boot drive, and while most things seem to be working normally, things aren't exactly as they were. Starting up and shutting down both take longer than seems reasonable (2'3 minutes for startup). Even booting into macOS Recovery is slower than it should be, taking about 8 minutes. What's particularly odd about that is that the Samsung T5 has excellent performance over USB 3.0, even better than the MacBook Air over Thunderbolt 2. It's showing over 400 MBps for both read and write speeds. My best guess right now is that the Mac is still looking for and receiving errors from the internal SSD at startup and shutdown, such that it takes a while for those errors to time out. Even stranger, for reasons I still don't understand, when I restart and log in, Wi-Fi is unavailable for over 2 minutes. And I mean unavailable'clicking the Wi-Fi icon in the menu bar doesn't give a menu, and no networking services appear in the left sidebar of System Preferences > Networking. Between 2 and 3 minutes after boot, Wi-Fi wakes up, and all is normal. This also happened a few times while booting from the MacBook Air in Target Disk Mode, and I haven't yet had time to ponder all the variables to figure out what might be going wrong. Freaky stuff. As you can see, the Wi-Fi icon is grayed-out and no network adapters are available at all (left), but just 23 seconds later, it's working and the Network preference pane is fully populated (right). Happily, I appear to have dodged [16]a virtual memory problem that has plagued some people booting from external SSDs. In some of these scenarios, macOS fails to create virtual memory swap files, seemingly because of an inability to mount the APFS swap volume. There's a workaround involving complicated scripting, but if I ran into this, I'd start by reformatting the drive and reinstalling macOS from scratch, since I believe the macOS installer would set things up correctly. I suspect that this saga isn't yet over, and if I learn any more lessons in the School of Hard Tech Knocks, I'll be sure to share them. In the meantime, I hope following along with my thinking helps your future troubleshooting and recovery efforts. References Visible links 1. https://www.watchmanmonitoring.com/ 2. https://support.apple.com/guide/disk-utility/repair-a-disk-dskutl1040/18.0/mac/10.14 3. https://www.youtube.com/watch?v=vZw35VUBdzo 4. https://www.amazon.com/gp/product/B01M1NHCZT/ 5. https://eshop.macsales.com/shop/owc-mercury-elite-pro-mini 6. https://support.apple.com/guide/mac-help/transfer-files-mac-computers-target-disk-mode-mchlp1443/mac 7. https://talk.tidbits.com/t/dead-ssd-in-a-2014-27-inch-imac-what-would-you-do/11853/?u=ace 8. https://talk.tidbits.com/t/dead-ssd-in-a-2014-27-inch-imac-what-would-you-do/11853/4?u=ace 9. https://eshop.macsales.com/item/OWC/DIYIMACHDD12/ 10. https://www.samsung.com/semiconductor/minisite/ssd/product/portable/t5/ 11. https://www.dell.com/en-us/work/shop/dell-ultrasharp-49-curved-monitor-u4919dw/apd/210-arnw/monitors-monitor-accessories 12. https://www.usa.philips.com/c-p/499P9H_27/brilliance-329-superwide-curved-lcd-display 13. https://www.samsung.com/us/computing/monitors/gaming/49-crg9-dual-qhd-curved-qled-gaming-monitor-lc49rg90ssnxza/ 14. https://www.shirt-pocket.com/blog/index.php/shadedgrey/comments/forward_into_the_past/ 15. https://tidbits.com/2019/10/28/redownload-archived-macos-installers-to-address-expired-certificates/ 16. https://apple.stackexchange.com/questions/318266/mac-os-x-is-not-creating-a-swap-file Hidden links: 17. https://tidbits.com/wp/../uploads/2020/04/Watchman-disk-errors.png 18. https://tidbits.com/wp/../uploads/2020/04/Watchman-SMART-errors.png 19. https://tidbits.com/wp/../uploads/2020/04/Erase-failed.jpg 20. https://tidbits.com/wp/../uploads/2020/04/Hard-disk-vs-internal-SSD.jpg 21. https://tidbits.com/wp/../uploads/2020/04/MacBook-Air-Thunderbolt2-vs-internal.jpg 22. https://tidbits.com/wp/../uploads/2020/04/Bad-Mojave-Installer.jpg 23. https://tidbits.com/wp/../uploads/2020/04/Restoring-slowly.jpg 24. https://tidbits.com/wp/../uploads/2020/04/Samsung-T5-USB3.jpg 25. https://tidbits.com/wp/../uploads/2020/04/Weird-Wi-Fi.jpg .