[HN Gopher] Options for genuine ECC RAM on the desktop in (early...
___________________________________________________________________
Options for genuine ECC RAM on the desktop in (early) 2024
Author : ingve
Score : 92 points
Date : 2024-02-17 07:45 UTC (15 hours ago)
(HTM) web link (utcc.utoronto.ca)
(TXT) w3m dump (utcc.utoronto.ca)
| muro wrote:
| You can also get a threadripper pro
| jrockway wrote:
| The non-pro threadrippers also use ECC now.
| treprinum wrote:
| They always used ECC but in UDIMMs, not RDIMMs.
| _JamesA_ wrote:
| What is the difference between UDIMM and RDIMM and which
| one is preferred and why?
|
| It seems RDIMM's are mostly supported on higher end server
| boards but I haven't seen a technical comparison of the two
| types.
| da768 wrote:
| RDIMMs have buffers on the memory lines to improve signal
| integrity, this is only helpful when there are many slots
| on the same memory controller channel. (ex. Servers often
| support 4 DIMMs per channel)
| loeg wrote:
| I think RDIMMs are available in higher capacities than
| UDIMMs. But the registering adds additional latency. So
| if you need the capacity, RDIMM is the way to go. But if
| you don't, you might prefer UDIMMs.
| badgersnake wrote:
| I bought a second hand first gen EPYC for my desktop. I'm not
| sure it was a great idea overall but it does work fine and I
| have 128gb full ECC, two M2 slots on the board and many many
| PCIe 3x lanes for anything else I want to add. The built in BMC
| is handy for remote access.
|
| The 32 cores is pretty pointless, I very rarely use more than
| half. And it probably uses way to much power.
| da768 wrote:
| SuperMicro finally got into AMD desktop CPU boards, if they added
| USB4 and faster networking to this one we'd have a winner
|
| https://www.supermicro.com/en/products/motherboard/h13sae-mf
| sirlancer wrote:
| I was pleasantly surprised a few days ago to find Supermicro
| sells a 3U chassis that runs 8x Ryzen 7000 series CPUs
| supporting ECC. If one doesn't need more than 128GB of RAM per
| system then they can get much higher clock speeds at a much
| lower power envelope than an EPYC CPU with an equivalent core
| count.
| JonChesterfield wrote:
| That should be a much higher power envelope, no? I'm
| struggling to think of a workload that would like eight
| 8/12/16 core systems on a network more than one 64 or 128
| core chip on the grounds that latency is harder to find than
| bandwidth.
| adrian_b wrote:
| The performance per dollar of Ryzen 7950X is many times
| higher than of any Epyc or Threadripper CPUs, even after
| adding the costs of the motherboards, coolers, cases and
| PSUs.
|
| The DRAM bandwidth per core is identical for Ryzen 7950X
| and for 96-core Genoa Epyc CPUs. On the other hand, the
| Epyc CPUs with high-core count have a better performance
| per watt.
|
| So the initial cost for a cluster of Ryzen servers is many
| times less. Depending on the cost of electricity, if an
| Epyc server is used 24/7, after some years the expenses
| with it may become lower than with Ryzen.
|
| If the server is used intermittently, the total cost of
| ownership may remain lower with Ryzen until the end of
| life.
|
| The only certain advantages of Epyc are the ability of
| aggregating a higher amount of memory, especially if it is
| preferable to have it inside a single box, and the faster
| inter-core communication for applications that use all
| cores (as opposed to the case when the cores are
| partitioned between weakly-coupled applications, e.g.
| between different virtual machines).
|
| The prices of the Epyc CPUs have increased a lot since
| their first generation until now.
|
| With Zen 1, a server with any Ryzen would not have been
| competitive with a server with Epyc. Meanwhile, the ratio
| between the prices per core of Epyc and Ryzen has increased
| a lot, while the ratio between the performance per core of
| Epyc and Ryzen has decreased a lot, because the clock
| frequency of Ryzens has become much greater while that of
| Epycs has increased only a little.
|
| These two evolutions combined have made that now the
| servers with Ryzen have become preferable to servers with
| Epyc in many cases.
|
| AMD has realized that they no longer have a solution for
| cheap servers, so in theory they have introduced the Siena
| CPUs for this purpose. Nevertheless, those remain somewhat
| too expensive and moreover they are nowhere to be seen.
| loeg wrote:
| The ASRock boards have always supported ECC in their consumer
| tier AMD products (Ryzen).
| snvzz wrote:
| I wish regulators would step in to enforce ECC in consumer
| devices.
|
| Otherwise, organizations such as Microsoft (windows readyness
| logo) or RISC-V (application profiles such as RVA23) could make
| it a requirement.
|
| It is clear that we can't just rely on the hardware vendors to do
| it, and practically nobody should be running computers w/o ECC.
| umanwizard wrote:
| > practically nobody should be running computers w/o ECC
|
| Why? I've done so my whole life with zero problems.
| frognumber wrote:
| I envy you. A life with zero problems.... I think you're
| probably the only person in the universe so fortunate.
|
| Here's the basic set of problems:
|
| 1) ECC is almost free if done at scale (e.g. every computer).
| The only reason it costs extra is that it is only done for
| high-paying customers and in limited volume.
|
| 2) Computer crashes cost lives. If e.g. your doctor's device
| has an issue mid-surgery, you potentially pay.
|
| 3) It's good for the economy. In either case, the cost
| savings of not having a call center rep say "Wait a minute.
| My computer just crashed." far outweighs the costs.
|
| 4) It's critical for resilience. Disasters (solar flares,
| EMP, etc.) are rare, but if every computer destroys data, we
| have a systemic problem.
|
| 5) The issue primarily is one of transparency. Most people
| don't realize it's an issue, and don't shop on it.
|
| 6) That's not to mention dual-market uses. E.g. if we need to
| ramp up military production, medical production, etc. next
| crisis, our infrastructure should be ready for it.
|
| Classical economic theory states government interventions do
| well for disaster resilience, for externalized costs, and for
| lack of transparency. Those are all places where markets
| fail. That means this should be regulated. We should have
| error correction in all storage and in all computation.
|
| Best intervention, though, isn't a ban but a tax. Sales tax
| should be on a sliding scale from 1-10%, with:
|
| - 1% for parts / support / service manuals being available
| for some extended period
|
| - 1% for all software being open-source
|
| - 1% for all hardware being open
|
| - 1% for use of standard, interchangeable parts (e.g.
| batteries, power connectors, etc.)
|
| - 1% for environmental factors (CO2, toxic materials, etc.)
|
| - 1% for reusable infrastructure (e.g. being able to
| repurpose manufacturing equipment for crisis use)
|
| - 1% for workplace factors (e.g reasonable benefits,
| audits/compliance, non-discrimination for target populations
| like ex-convicts, etc.)
|
| ... and so on
|
| Those sorts of gentle incentives accomplish the same thing
| via market mechanisms without limiting freedom. For a
| commodity product, a 10% edge far outweighs costs. If Apple
| wants to make a proprietary magnetic screen connector and
| save 1mm of space, a 1% extra tax won't even slow them down.
| Tuna-Fish wrote:
| > 1) ECC is almost free if done at scale (e.g. every
| computer). The only reason it costs extra is that it is
| only done for high-paying customers and in limited volume.
|
| This is simply not true. ECC means you need to store a
| minimum of 11% more bits, and at scale this means that you
| are paying a minimum of 11% more for memory.
|
| Note that I am very pro ECC, and think that most computers
| should use ECC ram.
| adrian_b wrote:
| As I have also said above, +12.5% paid by the
| manufacturer of DIMM for memory chips may mean +10% in
| the price of DIMMs when they are not marked up
| excessively, like the DDR5 modules today.
|
| That +10% may become +5% after adding CPU, cooler, MB,
| PSU, case and it may become +3% after adding SSDs, a
| high-quality monitor and other usual peripherals.
|
| While at work I have seen plenty of problems with
| computers without ECC, for my desktops and servers and
| even for some laptops, when I could, I have used only ECC
| memory, for the last few decades.
|
| That has never increased the cost as much as I have saved
| by choosing wisely other components in the computers that
| I have built myself.
|
| The main problem has never been the extra cost, but the
| difficulty to find them exactly when I needed them,
| because at retail the availability of ECC modules may be
| sporadic and many distributors appear and disappear, so
| if I upgrade systems after some years I seldom can find
| the same source.
| satellite2 wrote:
| I don't think the price of ram grows linearly with its
| bits. Manufacturing, shipping, distribution etc. are
| major costs that are mostly fixed for a single stick I
| would imagine.
| gruez wrote:
| >1) ECC is almost free if done at scale (e.g. every
| computer). The only reason it costs extra is that it is
| only done for high-paying customers and in limited volume.
|
| A ECC DRAM module contains 9 chips rather than 8. I'm not
| sure how you can claim that's "almost free" unless you have
| a very loose definition of "almost free".
|
| >2) Computer crashes cost lives. If e.g. your doctor's
| device has an issue mid-surgery, you potentially pay.
|
| Sounds like a good reason to regulate medical devices and
| force them to undergo certification as well. Surely you
| wouldn't want the the computer to have ECC memory, but it
| decides to reboot for a 2 hour update?
|
| >3) It's good for the economy. In either case, the cost
| savings of not having a call center rep say "Wait a minute.
| My computer just crashed." far outweighs the costs.
|
| If this is true, why do we need the government to step in?
| Why aren't businesses doing this themselves out of self
| interest?
|
| >4) It's critical for resilience. Disasters (solar flares,
| EMP, etc.) are rare, but if every computer destroys data,
| we have a systemic problem.
|
| ECC protects against solar flares and EMP? That's new.
|
| >6) That's not to mention dual-market uses. E.g. if we need
| to ramp up military production, medical production, etc.
| next crisis, our infrastructure should be ready for it.
|
| 1. DRAM modules aren't made in the US, they're made in
| asian countries with cheap labor. If there's a military
| crisis, we'd be screwed regardless
|
| 2. As mentioned above, ECC DRAM modules are basically the
| same as non-ECC modules but with an extra chip. The only
| work to retool a line from non-ECC to ECC is to change the
| PCBs. Thus the supposed benefits is negligible, unless for
| whatever reason the military needs a massive amount of
| desktops.
| adrian_b wrote:
| Almost free means that the 12.5% extra cost for memory
| chips may mean 10% extra manufacturing cost for the
| memory modules and that may mean less than 5% of the
| total cost of a computer with 64 GB DRAM (not counting
| monitor, SSDs and other peripherals, which would drive
| the percentage to be much lower) and even less for
| computers having less memory.
| frognumber wrote:
| > A ECC DRAM module contains 9 chips rather than 8. I'm
| not sure how you can claim that's "almost free" unless
| you have a very loose definition of "almost free".
|
| I do. Let's do the math here. A typical computer has 8GB
| RAM. That's under $20 at my local retail store. We're
| talking a $2 price difference. That's with the methods
| used today. The overhead can go arbitrarily low if we
| adopt techniques used in nonvolatile storage and work in
| larger blocks of bits.
|
| $2 is less than the cost of a tiny fraction of a crash.
|
| > Surely you wouldn't want the the computer to have ECC
| memory, but it decides to reboot for a 2 hour update?
|
| A broader regulatory regime would certainly be good.
|
| > If this is true, why do we need the government to step
| in? Why aren't businesses doing this themselves out of
| self interest?
|
| Mostly, because businesses can focus on a finite number
| of things. A $20 benefit which costs $2 to implement
| stops making sense if it means your local clinic needs to
| have someone on-staff who can do an ROI computation on
| adopting ECC. It's cheaper to buy something sub-optimal.
|
| Again, that's where centralization and regulation helps.
|
| > ECC protects against solar flares and EMP? That's new.
|
| That's literally what they're there for. ECC primarily
| protects against bit flips, which are the result of
| random radiation. Mild flares will flip a few bits which
| ECC will correct. Severe ones will flip a lot of bits,
| which ECC will detect, and the computer can shut down
| before destroying more data.
|
| > 1. DRAM modules aren't made in the US, they're made in
| asian countries with cheap labor. If there's a military
| crisis, we'd be screwed regardless
|
| I'm not sure what you mean by "modules," but DRAM PCBs
| are easy to assemble. They can be made in the US
| overnight. DRAM chips are primarily made in the US
| (Micron), Korea (Samsung), and Taiwan (TSMC).
|
| https://www.micron.com/manufacturing-expansion
|
| And not all crises are military.
|
| > The only work to retool a line from non-ECC to ECC is
| to change the PCBs. Thus the supposed benefits is
| negligible, unless for whatever reason the military needs
| a massive amount of desktops.
|
| If all you need is memory, that's true. If you also want
| motherboards and processors which will take those chips,
| that's harder. If you don't want bit flips in your
| computation, that's impossible without redesigning all
| CPUs. Academic CPUs can do error-correcting computation
| at nominal cost, but there isn't a market incentive to
| bring that to market.
| thfuran wrote:
| >>2) Computer crashes cost lives. If e.g. your doctor's
| device has an issue mid-surgery, you potentially pay.
|
| >Sounds like a good reason to regulate medical devices
| and force them to undergo certification as well. Surely
| you wouldn't want the the computer to have ECC memory,
| but it decides to reboot for a 2 hour update?
|
| Did you know that medical reports are frequently viewed
| on consumer or consumer-grade hardware? Or that there is
| software that is classified as a class II medical device
| that runs on iOS devices and commodity desktops? Or that
| class III devices like pacemakers now often use phone
| apps for things like managing telemetry? It's all well
| and good to make sure the hardware in the OR is reliable,
| but that's woefully insufficient.
| eptcyka wrote:
| ECC is more costly in terms of the BoM and higher power
| consumption.
| adrian_b wrote:
| The power consumption is higher by an amount that is not
| easily detectable.
|
| The power consumed by the relatively few gates that
| generate and verify the ECC code is dwarfed by the power
| consumption needed to transport bits over the PCB traces.
|
| The extra memory cell arrays for ECC bits add some power
| consumption for refreshing their content and the extra
| PCB traces for the ECC bits add some power consumption,
| but this extra power is in the milliwatt range, so it is
| negligible in comparison with the CPU and GPU power even
| in a small laptop, not only in a desktop.
|
| Most gaming desktops use overclocked memory modules that
| have an extra power consumption many times greater than
| the DDR5-4800 or DDR5-5600 ECC UDIMMs.
|
| The ECC UDIMMs never need heatsinks, while many of the
| commonly used overclocked DIMMs will overheat without
| heatsinks.
|
| So the claim that using ECC will increase the power
| consumption is completely bogus. Something like using a
| power-save profile instead of a high-performance profile,
| for clock management, or using JEDEC timings for memory,
| instead of high-performance timings, would diminish the
| power consumption by orders of magnitude more than not
| using ECC memory.
| frognumber wrote:
| It's actually a bit more bogus than that. If I can
| tolerate occasional bit errors -- which ECC permits -- I
| can reduce my power consumption by far more than the
| overhead of ECC.
|
| Right now, I power rails set to where billions of bits to
| never flip. If I can tolerate a one-in-a-billion bit
| flip, my tolerances go up a lot, and power consumption
| goes down by a similar factor.
|
| Properly implemented, ECC would likely cut power while
| increasing reliability.
|
| "Properly implemented" only happens in volume. That sort
| of optimization makes no sense for the very limited
| volumes ECC has today. Neither do a lot of other
| optimizations.
| eptcyka wrote:
| Fair point, thanks for proving my naive assumptions
| wrong.
| pkaye wrote:
| 2) Computer crashes cost lives. If e.g. your doctor's
| device has an issue mid-> surgery, you potentially pay.
|
| Given that software can also crash, I'd go with a different
| approach: Redundant Array of Inexpensive Computers. For
| most situations, RAIC1 is good enough.
| woadwarrior01 wrote:
| It's all fun and games until your computer starts acting all
| weird, like my 3 year old box did, last week. After being
| befuddled for a couple of hours, I decided to run MemTest86
| off of an Ubuntu installer and found memory errors[1].
|
| The manufacturer has RMA'd the whole set of memory sticks,
| but ECC would've detected this earlier and I'd have saved a
| couple of hours of my time.
|
| [1]: https://imgur.com/a/aRHWKxw
| eptcyka wrote:
| If you've used the computer for rendering things or running
| heavy duty analysis, could you trust your results now, if
| you know that the memory was faulty for some time? ECC
| isn't about making life easy when the memory goes bad, it's
| about allowing one to have more confidence in their system
| operating correctly.
| umanwizard wrote:
| So you think everyone should be forced to accept higher
| prices and lower battery life permanently because of
| something that cost you a few hours, once, that most people
| never encounter?
| loeg wrote:
| The manufacturing costs and power use are not
| meaningfully different.
| kevin_thibedeau wrote:
| The purpose of on chip ECC in DDR5 is to lower costs by
| hiding failures in marginal chips. They can't sell that
| memory as normal ECC.
| throw0101b wrote:
| > _Why? I've done so my whole life with zero problems._
|
| And folks experienced zero file corruption... until they
| started using ZFS checksums and regular scrubbing.
| echoangle wrote:
| Ok, but that's not a reason to mandate it by law. Or are
| you suggesting we should mandate file checksums too?
| almostnormal wrote:
| Devices sold should work to a certain expected level.
| Regulators could define (or even just judges decide in a
| specific case that can be applied more generally) what
| that level is. Losing bits randomly isn't within the
| range of what is expected. The more memory there is, the
| more likely it is going to happen. And at some point
| devices without ECC could simply be returned as faulty
| for a full refund. ECC would be required, without any
| explicit rule.
| umanwizard wrote:
| That is already how it works now... you can already
| return defective ram, ECC or not.
|
| I am happy with the point in the price/reliability/power
| usage tradeoff that non-ECC ram represents. Why force me
| to have fewer choices and use something that is worse for
| my needs in order to solve a theoretical problem that I
| don't care about?
| rvnx wrote:
| To me, it sounds like "if it doesn't work 100.00% of the
| time, we cannot trust it and should forbid it", but the
| reality is that neither the world (and even internet) nor
| electricity nor healthcare, are "100.00%
| available/reliable".
|
| In general don't worry that for very important
| calculations, they are run a couple of times, at least
| just to verify.
| snvzz wrote:
| >In general don't worry that for very important
| calculations, they are run a couple of times, at least
| just to verify.
|
| Citation needed, as this is unheard of in software we
| trust regardless to do banking or sign legally binding
| documents with.
| sliken wrote:
| Well the trick is a process or your entire OS crashed,
| why? Was it the powersupply? Motherboard? CPU? DIMM?
| Which DIMM? Are you going to keep a complete PC of spares
| to replace piece by piece to see if the crash stops?
|
| For such a minimal cost I'd rather have DIMM 3 died with
| a non-correctable error.
| echoangle wrote:
| Not really. Some errors are expected, are you going to
| mandate 2 GPUs because the Memory could fail and it could
| show artifacts on the screen? Electronics failure is
| always a possibility with a small but nonzero
| probability. If your RAM is faulty, you can already
| return it. That does not mean that every device has to
| have ECC to prevent very improbable bitflips.
| rvnx wrote:
| Processors too can have implementation bugs (and had!),
| and bit flips, so even a perfect memory won't solve this
| edge scenario.
| snvzz wrote:
| An excellent example of Whataboutism.
|
| Measures are in the works there, such as the switch to a
| simpler and cleaner ISA.
|
| The value of simplicity is often overlooked; RISC-V will
| lead to higher quality software and hardware.
|
| But I digress, as this has no weight in whether or not
| lack of ECC is acceptable.
| umanwizard wrote:
| Too late to edit my original response but I also want to
| point out: I'd be fine mandating ECC in critical
| applications like medical devices.
| pyb wrote:
| I would assume that the worldwide yearly cost of data
| corruption is orders of magnitude larger than the cost of
| mandating ECC everywhere.
| brookst wrote:
| Big assumption! I'd love to see a model showing this.
| thfuran wrote:
| Are you sure about that?
| mrb wrote:
| Zero problems? Very unlikely. The thing about memory
| corruption is that it tends to create problems that most
| people don't recognize as memory corruption issues. A random
| BSOD or freeze. A corrupted file. A video glitch. A
| website/JS code fragment misbehaving. Etc.
| qwertox wrote:
| There's also the chance that you think you had zero problems
| because you never got notified.
|
| Some years ago I built a small N3150 based home server with
| 16GB RAM. All was fine. But there was this one process that
| would always act weird after running for a couple of days. It
| was a Python server with long-running WebSocket connections
| which I had configured to use `select()` instead of
| `epoll()`, and for some reason it would eat up file handles
| until it crashed or froze, can't remember what it was, very
| odd stuff. I think it was running inside a Docker container
| and that container then behaved very strangely.
|
| Turns out that when I then made a MemTest86 it had faulty
| RAM. All the data which passed through that area of the RAM
| and got written to disk was assumed to be corrupted.
|
| After replacing the RAM (RMA) and doing a MemTest on it for a
| day to ensure that it was OK, that problem never appeared
| again.
|
| But keep in mind, this was just one specific process which
| showed an issue after running for a couple of days, among
| other long-running processes.
| NikolaNovak wrote:
| 1 >> I've done so my whole life with zero problems.
|
| 2 >> There's also the chance that you think you had zero
| problems because you never got notified.
|
| Both are likely true. I.e. Over my 40 years of dealing with
| computers ,it is likely there were _errors_ which did not
| become _problems_.
|
| I have about 20tb of unique files in my personal storage at
| this point (used to run photography business, plus personal
| videos and photos,plus movies and tv shows, plus actually
| important documents which are less than 3% by size).I have
| about 4-8 computers running around my home doing stuff
| (plus dozens of computing devices - phones, googles,
| whatnots).
|
| I understand the overwhelming chance there were many errors
| over time in running computers or raid transfers or usb
| backups over decades. They just haven't manifested
| themselves as problems.
| snvzz wrote:
| The fact that you're dealing with silent corruption does
| not make it any better, but in fact, much worse.
|
| And yes, the corruption is there. DRAM is alarmingly
| unreliable. Because no ECC, we just do not notice it most
| of the time, or misattribute the problems it causes.
| qwertox wrote:
| If you're unlucky you get into mode with ZFS where every
| scrub corrupts more and more data, while ZFS itself
| thinks that all is OK because it computes the checksum on
| the data it just corrupted while scrubbing.
|
| See https://www.truenas.com/community/threads/ecc-vs-non-
| ecc-ram...
| Fire-Dragon-DoL wrote:
| You would be surprised how often the system behaves
| unpredictably and it is because of RAM. i had multiple
| times faulty ram and the symptoms are wildly different.
|
| Also, when you download large files which you can run
| checksun against such as steam games, it is pretty common
| to find minor corruption (this i verified across many
| computers)
| 1letterunixname wrote:
| Anecdotal non-evidence. You've never noticed or had specific
| evidence of silent corruption that probabilistically risked
| silent corruption to quiesced to permanent storage or caused
| unexplained storage. Perhaps you've never heard of domain
| bitsquatting but it was rampant before https:// became
| widespread. All network gear, servers, and client devices
| need ECC or parity on all buses and at higher level storage,
| compute, and networking protocols to ensure integrity. Retail
| consumer electronics manufacturers throw out features
| customers lack the expertise to understand the consequences
| of. ECC RAM would need to be enforced by government mandate
| because that's the only way things will ever change. It's a
| massive, still present security and data integrity hole that
| could be closed today by spending a trivial amount of money
| if it were implemented at large scale.
|
| See also: PDF https://media.defcon.org/DEF%20CON%2021/DEF%20C
| ON%2021%20pre...
|
| https://cr.yp.to/hardware/ecc.html
| ddtaylor wrote:
| I'm very happy this should never happen.
| brookst wrote:
| Not a fan of using regulatory hammer to forbid people from
| having different priorities than I do. Adding across the board
| cost to computers has consequences.
| snvzz wrote:
| People do shopping, taxes and banking on their computers now.
| They use personal certificates to sign legally binding
| documents.
|
| Them not having ECC is insanity.
|
| There are of course niche cases where memory reliability does
| not matter. Exceptions can be made for those few.
|
| But think about the general case first. Got to protect the
| non-technical from non-ECC memory.
| gosub100 wrote:
| I've had way more problems with defective security and bugs
| from Microsoft than I have with ECC bit-flipping errors. I
| would rather have "regulators step in" and nullify the
| disclaimer about "no fitness for any particular purpose"
| clauses in software license agreements. If I buy a windows
| computer and it gets hacked, I should be able to take MSFT to
| court (small claims if need be) and sue for damages (if I can
| show them) due to _their negligence_. They market PCs as these
| great tools for running everything in your life, but first they
| want release for any responsibility on their part. Those terms
| do not agree.
| adrian_b wrote:
| The frequency of software bugs is what has allowed the
| hardware manufacturers to omit ECC from most computer
| products.
|
| Every time when a program crashes or remains stuck or data
| corruption is discovered, the user automatically assumes that
| this is a software bug.
|
| While it is indeed true that software bugs are really much
| more frequent than memory errors before the memory modules
| happen to become defective, in a computer without ECC the
| user can never know whether it has been a software bug or it
| was a hardware memory problem.
|
| So many memory errors remain hidden by the abundant software
| bugs, which in most commercial programs, especially in those
| from Microsoft, almost never generate error messages from
| which the cause of the error can be determined.
| sys_64738 wrote:
| What do people think ECC is going to do for them, and what do
| they think will happen when an ECC event is triggered?
| AshamedCaptain wrote:
| I suppose they'll claim that it's what doesn't trap which is
| important, e.g. what can be (silently) corrected by ECC would
| normally result in silent corruption without ECC.
| CTDOCodebases wrote:
| What I think it will do for me?
|
| Correct single bit errors and alert me to them.
| gambiting wrote:
| Like, in what way - you get a windows popup saying "hey, just
| detected a bit error, but it was corrected, all good"?
| Because if yes, you'd get such a popup daily, and I'm not
| sure what the actual value of it is. And I'm not being funny,
| I literally don't see the value of it(of the notification,
| not the correcting).
| dale_glass wrote:
| You don't get a popup. In Linux, it's logged in dmesg. In
| Windows I imagine it goes in the system log. The system
| keeps on running.
|
| The value of the notification is that if most days you get
| nothing and suddenly 5000 corrected errors pop up, you know
| something is not quite right as of late. Maybe you're
| overdoing the overclocking. Maybe the module isn't sitting
| quite well in the slot. Maybe a solder joint is failing.
|
| If you have a 2 bit memory error, the typical result is a
| BSOD/kernel oops.
|
| Which is a good thing actually. It means you know for sure
| your RAM is bad, and it prevents any faulty data from being
| written anywhere. So rather than possibly propagating the
| error is just stopped.
|
| I believe there have been systems that try to handle it
| more elegantly. Kill the process the memory belongs to if
| it's a non-essential one then permanently blacklist that
| block to ensure it's not used again.
| CTDOCodebases wrote:
| > Like, in what way - you get a windows popup saying "hey,
| just detected a bit error, but it was corrected, all good"?
|
| No I've set it up so I get an email.
|
| > Because if yes, you'd get such a popup daily
|
| I doubt that. There was a study done at Google that
| reported 8% of DIMMs encountered at least one memory error
| per year[0].
|
| > I'm not sure what the actual value of it is.
|
| I like my PC to be as deterministic as possible[1] so if my
| programs crash I don't want to have to turn it off and test
| the memory which takes hours[2].
|
| [0] -
| https://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf
|
| [1] - https://blog.cskr.dev/posts/corrupt-backup/
|
| [2] - https://www.youtube.com/watch?v=0itrM7t4l34
| adrian_b wrote:
| On a computer configured correctly, the errors are logged and
| the user or administrator may be notified when they happen.
|
| In my experience the most valuable benefit of ECC is the
| detection of aged memory modules.
|
| All modern DRAM has a limited lifetime. Hopefully that lifetime
| is more than 5 years, but there are always some modules that
| may reach end-of-life much faster, especially on computers that
| are always on.
|
| While on a new memory module it would be abnormal for memory
| errors to happen more frequently than a few times per year, on
| an aged module errors may happen many times per day, or even
| many times per hour.
|
| I have encountered a few such cases that happened after several
| years of continuous using. Due to ECC I was notified about the
| errors and I could replace the offending modules before them
| becoming so bad as to guarantee data corruption.
|
| Another reason is that even if ECC cannot prevent a RowHammer
| attack, it makes pretty certain that a RowHammer attack will be
| detected immediately (because the attacker cannot control the
| location of the flipped bits, and even if there are many
| errors, so they will be miscorrected, in most cases they will
| still be detected and logged and the administrator should
| always be instantly notified when there are two or more errors
| per day, as this will never happen normally).
| musicale wrote:
| My thought also when I first heard about RowHammer - how do
| you avoid triggering an ECC error? It seemed like you'd need
| multiple bit flips, possibly of specific bits.
| RecycledEle wrote:
| I had a stick of RAM go bad in an old Socket 775 motherboard,
| but since it was one of 6 sticks of RAM, it only affected the
| PC on rare occasions. That PC was flaky for years before I
| pulled out some RAM to upgrade and found the problem had been
| fixed.
|
| When a similar problem occurred on a PC with ECC RAM, I got a
| warning message saying memory errors had been detected that
| repeated a few times, then I got a message saying one stick of
| RAM had been deactivated and that I could keep running. The
| Dell T3500 dropped from 24 GB to 20 GB and did not even need to
| be rebooted.
|
| That is why I love ECC RAM and Xeon processors. I know if there
| is a problem and can wait to fix it.
|
| I can buy used RAM and processors with confidence that the BIOS
| self rests will tell me what is good and what is bad.
| sys42590 wrote:
| Some devices with 13th gen Intel processors support in-band ECC.
| This basically hides a fraction of RAM from the OS and uses it
| for ECC.
|
| Example devices are: LattePanda Sigma [0] and AsRock Industrial
| NUCS BOX-1360P [1].
|
| Unfortunately they are quite expensive and enabling in-band ECC
| lowers performance significantly. So my next server rig will
| likely have an AMD PRO CPU instead.
|
| [0] https://www.servethehome.com/lattepanda-sigma-review-the-
| ras... [1] https://www.anandtech.com/show/18732/asrock-
| industrial-nucs-...
| tpolzer wrote:
| I find it annoying that (a) this seems to be a hardware feature
| that is almost universally permanently disabled in firmware and
| (b) it's almost impossible to find out whether it is supported
| by any given product (neither of the two products you link
| provide any mention on the spec sheet).
| zokier wrote:
| This inertia holding ECC away from desktops is just so dumb. As
| if me doing banking, taxes and everything else on desktop is less
| important than some server hosting cat pictures. Or as if ECC
| would truly represent any meaningful part on higher end system
| total cost. Or as if I would even have the option of simply
| paying some premium to get ECC, when in practice desktop Xeon
| availability and selection are both extremely limited.
|
| Of course the benefit of ECC can be debated. But it seems such a
| waste to bother debating it when Intel could just make the whole
| thing a non-issue by a flick of a switch.
| AshamedCaptain wrote:
| The entire computing industry is based on trading-off the
| probability & effects of failure with cost. Why would you even
| bother with ECC when practically no one except maybe server
| farms with tens of 24h-on servers would notice a thing? Even a
| single micro-cent per device spent on improving testing would
| have a much more felt-through effect on the entire computing
| landscape.
| kiririn wrote:
| For desktop computers most of the value of ECC comes from not
| corrupting your hard disk when a ram stick spontaneously goes
| bad, rather than protecting against once in a blue moon bit
| flips
| AshamedCaptain wrote:
| ECC is kind of useless when a RAM stick "spontaneously goes
| bad", unless precisely you are talking about the situation
| where a single bit goes bad (which is, indeed, less than
| once in a blue moon).
| JonChesterfield wrote:
| This one does pervasive damage to software quality though.
| You get a bug report, it doesn't reproduce easily. The
| thought is always there: "maybe their hardware had a
| transient error?". If it did you have no bug to find. That's
| attractive.
|
| If storage drives reported writes accurately and memory
| didn't occasionally silently corrupt, software falling over
| would be more likely to imply an error the developer is
| empowered to fix.
| AshamedCaptain wrote:
| ECC RAM doesn't prevent _all_ hardware transient errors. It
| doesn't even prevent the majority of them.
|
| But whenever you get a bug report, is really the first
| thing that comes to mind "maybe their hardware had a
| transient error?"? The amount of software vs hardware bugs
| is still like a thousand to one, and that's _even_ for
| companies which receive a disproportionate amount of
| hardware errors (like OS vendors). For regular software,
| I'd bet it's a couple million to one.
| JonChesterfield wrote:
| It's market segmentation. You want to sell expensive chips to
| people with money and cheap ones to people without. That's hard
| to do if the products seem to be equivalent. ECC availability
| is part of the marketing skill with which people buy Xeon
| branded processors.
|
| This isn't inherently evil behaviour.
|
| AMD is interesting in this regard. ECC on desktops. Lots of x86
| chips have some integrated GPU. Lots of PCIe lanes. The idea
| seems to be biased much more towards build a great platform, as
| opposed to build a great profit margin.
| paulmd wrote:
| Ecc is still segmented on the amd apu lineup, including
| desktop models (and for many years these were the only models
| with integrated graphics!), and there literally is no way for
| consumers to even buy some of the parts that AMD has
| segmented these features to. So there is not some moral
| imperative that AMD has undertaken here - AMD backs my
| "right" to ECC right up until I want to use it on a 5700G, or
| buy a 5750G Pro, which they won't even sell to me as a
| consumer!
|
| (just like AMD also segmented based on SMT up through at
| least the ryzen 4000 series despite the fans bashing intel
| for the same, lol)
|
| Moreover, with amd one increasingly has to worry about the
| platform lock, which is an absolutely obscene e-waste stream
| AMD has created to reduce the secondhand sales market. This
| isn't limited to server CPUs or pro CPUs, notionally it
| affects all CPUs that might have ever passed through an oem
| system, including non-pro consumer CPUs etc.
|
| https://youtu.be/bFNJVaO9E-o?t=420
|
| The stated reason is security/antitheft, but the platform
| lock doesn't lock a CPU to a _motherboard_ , it locks it to a
| brand. So if someone has physical access to your server rack
| and wants to steal the CPU, they just have to find a lenovo-
| branded or hp-branded chip to swap it out with... and there
| are rivers of those because locked chips are essentially
| treated as e-waste. What it _does_ do very successfully is
| throw a whole cloud of confusion and pain over the secondhand
| market and drive these to be scrapped etc and push sales of
| new devices instead.
|
| AMD could very easily have implemented a lock/unlock system
| using non-volatile storage or simple e-fuse even/odd counting
| (limited number of locks/unlocks, granted, but it carries
| with the CPU not the mobo, and how many systems is the
| average CPU used in?). They, naturally, chose the one that is
| permanent but bricks the cpu for everyone else, but which
| also doesn't really fully prevent CPU swapping attacks
| either. Plenty of e-fuses to spare for detecting memory
| overclocking though - we all know denying warranties comes
| first ;)
|
| For the type of person who gets wound up about the moral
| imperative of ecc, that's not exactly what you want to see
| from AMD either. First 2 Rs are "reduce" and "re-use",
| followed by "recycle". Like how about we don't tivoize the
| cpu in 2018 or whatever? Secondhand sales are just as much of
| a moral right and AMD is really having the old college try at
| taking that away, and frankly a lot more impactful to the
| average user. I strongly wish the EU would step in and ban
| this practice. This is not a good/healthy path, it would
| actually be really bad if everyone was doing this on their
| products.
|
| Also, Intel has ECC support on the upper half of their range.
| Yes, it requires motherboard support, but you also have to
| pick and choose with AMD because only a handful of
| motherboards actually support/validate it anyway. Plus with
| AMD you have the fTPM issue (and very few of the premium
| creator boards/etc have external TPM headers). This is the
| reality as a homelab owner - you have to pick your hardware
| very carefully regardless of brand, a random B650 board
| probably is not going to report errors and may not even
| detect them. And while it may work on any given BIOS, it may
| stop working at any future point, whether intentionally or
| not.
| JonChesterfield wrote:
| I thought the APU chips also supported ECC though some
| motherboards don't. That could well be wrong though, the
| messaging seems quite muddied. In fairness my NAS is an
| intel i3 with ECC support so it's not a simple Xeon/not
| split there either.
|
| I remember news stories about the platform lock but haven't
| kept up to date, do you know how that played out in
| practice?
|
| My asrock board doesn't seem to have an option to burn out
| fuses to stop it working elsewhere but I am slightly less
| cavalier about changing settings knowing that might be in
| there.
| paulmd wrote:
| ECC support is segmented to Pro APUs only. Which aren't
| even sold to consumers.
|
| Yeah the lack of iGPU support (plex encoding, VMs, etc)
| pushed me to the 7100 last time I did a homeserver build.
| It's been muddled on intel's line forever - for a while
| it was only i3s (since they dropped the idea of a 2C xeon
| there was no reason to keep segmenting it) but then with
| alder lake they flipped it and it's on everything 12500
| and above (except F skus). And while you do need a
| motherboard that supports it, W680 supports
| overclocking/XMP/etc - the tier of boards I'm looking at
| aren't materially different in cost (or actually intel
| often trends cheaper).
|
| The platform lock has actually affected me quite badly
| trying to do a ROMED8-2T build, a massive number of the
| desirable CPUs are platform locked and unless they
| explicitly say unlocked it's always a question whether
| any given chip is actually _unlocked_ or just doesn 't
| say/doesn't know. Which is what I see as the whole point
| - get those machines into the scrapper (zero value means
| nobody bothers reselling) instead of people buying
| perfectly good older cpus and drawing extra life from
| them.
|
| I am not thrilled about the idea of that bleeding into
| the consumer market at all. So far AMD hasn't gotten much
| traction with OEMs but that's changing over time, and I
| see every reason for AMD to incentivize OEM partners to
| use this functionality across their whole lineups - OEMs
| want cheaper chips and AMD can sell them cheaper because
| they know they are junk after the first sale. Why
| wouldn't ThinkStation SFF PCs etc come with locked
| processors at some point? AMD clearly has laid the
| groundwork and all the tooling is there and just is
| boiling the frog very cautiously.
|
| By the time people see the impact in the consumer market
| it will be too late. And there are a lot more
| ThinkStations and Optiplexes sold than DIY PCs.
| pella wrote:
| (2021)
|
| _Torvalds went on his lengthy post to say, "The "modern DRAM is
| so reliable that it doesn't need ECC" was always a bedtime story
| for children that had been dropped on their heads a bit too many
| times. Yes, I'm pissed off about it. You can find me complaining
| about this literally for decades now. I don't want to say "I was
| right". I want this fixed, and I want ECC. And AMD did it. Intel
| didn't."_
|
| (via https://www.phoronix.com/news/Linus-Torvalds-ECC )
| pella wrote:
| (2023) "AMD Ryzen 9 7900X Performance With ECC DDR5 Memory"
|
| https://www.phoronix.com/review/amd-ryzen9-ddr5-ecc
|
| _" When plotting the benchmarks with a statistically significant
| difference, there was just a handful of benchmarks within the
| 2~3% faster range when ECC was disabled. As shown earlier, the
| Graph500 HPC benchmark was the outlier with a ~8% difference
| toggling ECC. PostgreSQL and Hadoop were among the few real-world
| workloads with the 2~3% difference, which is a small price to pay
| for the benefits of ECC. It's great that the desktop AMD Ryzen
| processors continue to support ECC memory while still it's
| ultimately up to the motherboard vendors whether ECC is to be
| supported. We're at least seeing more AMD Ryzen motherboards come
| to market with a server focus which in turn is great for allowing
| more motherboard options with ECC memory support. My testing with
| the ASRock Rack 1U4LW-B650/2L2T 1U server (ASRockRack
| B650D4U-2L2T/BCM motherboard) continues working out very well and
| hasn't yielded any platform troubles with all of my hundreds of
| hours of Linux testing thus far."_
| fourfour3 wrote:
| I have one of these ASRock Rack servers, and I can confirm ECC
| works on it - I have 128GB of ECC DDR5 with a Ryzen 9 7900.
|
| I wish more motherboards bothered to validate ECC working -
| it's a nice benefit in an affordable platform.
| mwpmaybe wrote:
| > I can confirm ECC works on it
|
| "It works" as in it boots, or "it works" as in it checks and
| reports single-bit errors and reports double-bit errors?
|
| What OS? If Linux, are you using rasdaemon? Does it confirm
| that the appropriate EDAC drivers are loaded and working?
|
| Or is that data only available in the BMC's event log?
| fourfour3 wrote:
| Fault injection has been used successfully and people have
| introduced manual errors that do get flagged - eg https://f
| orum.asrock.com/forum_posts.asp?TID=24901&PID=10365...
|
| For Linux support you need a very recent kernel - I forget
| how recent, but some reporting on it:
| https://www.phoronix.com/news/AMD-EDAC-Ryzen-7000-Series
| 486sx33 wrote:
| Running azrock mb with a ryzen 9 3900x and Samsung b die
| ram... it's pile of crap compared to asus mobo with the
| same plugins... ecc, I don't know... what's a proper ecc
| test today and is it worth sacrificing 12% in performance?
| jurgenkesker wrote:
| Normal Asrock Ryzen 5xxx motherboards also supported ECC, I
| used ECC RAM for my homesever with a normal 5900X and a
| Asrock B650M Pro4 motherboard.
|
| The new Asrock Ryzen 7xxx motherboards unfortunately don't
| have that same ECC support.
| sys_64738 wrote:
| How much memory is lost when you enable ECC? That's why consumers
| will disable it.
| adrian_b wrote:
| No memory capacity or memory throughput is lost in consumer
| motherboards when enabling ECC.
|
| You just need to buy ECC UDIMM modules, which, especially for
| DDR5, are more expensive, up to 50% more expensive for DDR5,
| even if this price difference is not justified.
|
| Moreover, the ECC modules are never factory overclocked, like
| most non-ECC modules, therefore ECC UDIMM modules faster than
| DDR5-4800 are hard to find. Nevertheless, that has not stopped
| some users to overclock them manually.
|
| There are some embedded computers for industrial applications,
| with certain models of Intel Atom or Core CPUs, which support
| the so-called in-band ECC, which can be used with non-ECC
| memory, like LPDDR5. Only on those computers enabling ECC
| reduces both the available memory and the performance.
| HankB99 wrote:
| No mention of on die ECC so far so I'll mention it. It's used in
| modern RAM (DDR5, maybe some DDR4) to improve yields. A
| borderline chip may pass tests if occasional bit errors are
| corrected before being read out. This is transparent to the OS as
| there is no mechanism that I'm aware of to report fixed bits or
| unfixable errors.
|
| I'm hopeful that this is going to help fix problems with
| occasional bit flips from cosmic rays and similar. Since it is
| used to keep marginal chips in the product stream, I'm not sure
| if it is really beneficial.
|
| The Raspberry Pi 4B claims DDR4 ECC RAM and I'm pretty sure
| that's on die ECC.
|
| I'd really prefer full blown ECC RAM and would pay a reasonable
| increase for it. I recently upgraded my desktop and sticking with
| ECC RAM would have cost hundreds of dollars more so I skipped it.
| eptcyka wrote:
| ECC without error reporting still won't save one from data
| corruption. Ultimately, I want correctness not continuous but
| faulty operation. On-die ECC helps with the latter.
| mrob wrote:
| Agreed. Reliability is the most important part of
| "reliability, availability and serviceability"[0]. Error
| correction is nice to have, but even simple parity checking
| is much better than nothing. I'd rather halt with a kernel
| panic than continue using corrupted data.
|
| [0] https://en.wikipedia.org/wiki/Reliability,_availability_a
| nd_...
| thfuran wrote:
| And yet many developers insist it's better for the users to
| just swallow exceptions and carry on rather than halting on
| bad state.
| brookst wrote:
| It really depends on the application. Halting rather than
| displaying a slightly corrupted graphics texture is
| probably not what gamers want. Swallowing an exception in
| a finite element analysis is probably not what engineers
| want.
| thfuran wrote:
| It does depend on the application. And there's a whole
| lot of software for which the developers don't have
| complete enough understanding of everything downstream to
| really know the significance of incorrect output.
| jeffbee wrote:
| There are loads of different ways to "have ECC"... it isn't as
| binary as the post implies. Different methods can detect and
| correct different failures.
| sph wrote:
| What's an affordable x86_64 NUC with ECC RAM? I remember getting
| frustrated trying to find one that I just abandoned the idea of
| making a home server altogether.
|
| Please, I don't want to hear "you don't need ECC", I just want
| some recommendations.
___________________________________________________________________
(page generated 2024-02-17 23:02 UTC)