[HN Gopher] What makes Intel Optane stand out (2023)
___________________________________________________________________
What makes Intel Optane stand out (2023)
Author : walterbell
Score : 164 points
Date : 2026-03-15 15:09 UTC (7 hours ago)
(HTM) web link (blog.zuthof.nl)
(TXT) w3m dump (blog.zuthof.nl)
| hbogert wrote:
| It stands out, because it didn't sell. Which is weird because
| there were some pretty big pros about using them. The latency for
| updating 1 byte was crazy good. Some databases or journals for
| something like zfs really benefited from this.
| bombcar wrote:
| It feels like everyone figured out what to do with them and how
| just about when they stopped making them.
| timschmidt wrote:
| Same for the Larabee / Knights architecture. Would sure be
| fun to play around with a 500 core Knights CPU with a couple
| TB of optane for LLM inference.
|
| Intel's got an amazing record of axing projects as soon as
| they've done the hard work of building an ecosystem.
| zozbot234 wrote:
| > 500 core
|
| The newest fully E-core based Xeon CPUs have reached that
| figure by now, at least in dual-socket configs.
| timschmidt wrote:
| Yup. And high end GPU compute now has on-package HBM like
| Knight's had a decade ago, and those new Intel CPUs are
| finally shipping with AVX reliably again. We lost a
| decade for workloads that would benefit from both.
| epistasis wrote:
| When most people are running databases on AWS RDS, or on
| ridiculous EBS drives with insanely low throughput and latency,
| it makes sense to me.
|
| There are very few applications that benefit from such low
| latency, and if one has to go off the standard path of easy,
| but slow and expensive and automatically backup up, people will
| pick the ease.
|
| Having the best technology performance is not enough to have
| product market fit. The execution required from the side of
| executives at Intel is far far beyond their capability. They
| developed a platform and wanted others to do the work of
| building all the applications. Without that starting killer
| app, there's not enough adoption to build an ecosystem.
| amluto wrote:
| > There are very few applications that benefit from such low
| latency
|
| Basically any RDBMS? MySQL and Postgres both benefit from
| high performance storage, but too many customers have moved
| into the cloud where you can't get NVMe-like performance for
| durable storage for anything remotely close to a worthwhile
| price.
| epistasis wrote:
| I'm saying that there are very few downstream applications
| that use databases that benefit from reducing latency
| beyond the slow performance of the cloud. Running your
| database on VMs or baremetal gives better performance, but
| almost no applications built on databases bother to do it.
| p-e-w wrote:
| Optane was a victim of its own hype, such as "entirely new
| physics", or "as fast as RAM, but persistent". The reality felt
| like a failure afterwards even though it was still
| revolutionary, objectively speaking.
| zozbot234 wrote:
| Optane didn't sell because they focused on their weird
| persistent DIMM sticks, which are a nightmare for enterprise
| where for many ordinary purposes you _want_ ephemeral data that
| disappears as soon as you cut power. Thet should have focused
| on making ordinary storage and solving the interconnect
| bandwidth and latency problems differently, such as with more
| up-to-date PCIe standards.
| jauntywundrkind wrote:
| I don't think that would be my main complaint. Sticking
| optane in a dimm was just awkward as hell. You now have
| different bits of memory with very different characteristics,
| & you lose a ton of bandwidth.
|
| If CXL was around at the time it would have been such a nice
| fit, allowing for much lower latency access.
|
| It also seems like in spite of the bad fit, there were enough
| regular options drives, and they were indeed pretty
| incredible. Good endurance, reasonable price (and _cheap as
| dirt_ if you consider that endurance /lifecycle cost!), some
| just fantastic performance figures. My conclusion is that
| alas there just aren't many people in the world who are
| serious about storage performance.
| tayo42 wrote:
| Can Linux differentiate that different dimms are different?
| Or does it see it all as one big memory space still?
| wmf wrote:
| Yes, Linux was aware of the difference via ACPI tables.
| hrmtst93837 wrote:
| PCIe was a bottleneck in consumer boxes, but that wasn't the
| whole problem. Optane's low latency and write endurance
| looked great on paper, yet once you put it behind SSD
| controllers and file systems built around NAND assumptions, a
| lot of the upside got shaved off before users ever saw it.
|
| "Just make it a faster SSD" was never a business. The DIMMs
| were weird, sure, but the bigger issue was that Optane made
| the most sense when software treated storage and memory as
| one tier, and almost nobody was going to rewrite kernels,
| DBs, and apps for a product that cost more than flash and
| solved pain most buyers barely felt.
| PunchyHamster wrote:
| > and file systems built around NAND assumptions, a lot of
| the upside got shaved off before users ever saw it.
|
| What file systems ? Most common one you'd find would be
| ext4 or XFS and neither of them are
| ksec wrote:
| >Which is weird....
|
| It isn't weird at all. I would be surprised if it ever succeed
| in the first place.
|
| Cost was way too high. Intel not sharing the tech with others
| other than Micron. Micron wasn't committed to it either, and
| since unused capacity at the Fab was paid by Intel regardless
| they dont care. No long term solution or strategy to bring cost
| down. Neither Intel or Micron have a vision on this. No one
| wanted another Intel only tech lock in. And despite the high
| price, it barely made any profits per unit compared to NAND and
| DRAM which was at the time making historic high profits. Once
| the NAND and DRAM cycle went down again cost / performance on
| Optane wasn't as attractive. Samsung even made some form of SLC
| NAND that performs similar to Optane but cheaper, and even they
| end up stopped developing for it due to lack of interest.
| jauntywundrkind wrote:
| Cost was fantastically cheap, if you take into account that
| Optane is going to live >>10x longer than a SSD.
|
| For a lot of bulk storage, yes, you don't have frequently
| changing data. But for databases or caches, that are under
| heavy load, optane was not only far faster, but if looking at
| life-cycle costs, way way less.
| wtallis wrote:
| Optane was in the market during a time when the mainstream
| trend in the SSD industry was all about sacrificing
| endurance to get higher capacity. It's been several years,
| and I'm not seeing a lot of regrets from folks who moved to
| TLC and QLC NAND, and those products are more popular than
| ever.
|
| The niche that could actually _make use of_ Optane 's
| endurance was small and shrinking, and Intel had no roadmap
| to significantly improve Optane's $/GB which was
| unquestionably the technology's biggest weakness.
| mapt wrote:
| Write endurance of the drive would be measured in TBW, and
| TLC flash kept adding enough 3D layers to stay cheap
| enough, quickly enough, that Optane never really beat their
| pricing per TBW to make a practical product.
|
| I have to wonder if it isn't usable for some kind of
| specialized AI workflow that would benefit from extremely
| low latency reads but which is isn't written often, at this
| point. Perhaps integrated in a GPU board.
| jauntywundrkind wrote:
| The extra capacity of modern SSD is a good point,
| especially now that we have 100TB+ SSD.
|
| But Optane still offered 100 DWPD (drive writes per day),
| up to 3.2TB. Thats still just so many more DWPD than
| flash ssd. A Kioxia CM8V for example will do 12TB at 3
| DWPD. The net TBW is still 10x apart.
|
| You can get back to high endurance with SLC drives like
| the Solidigm p7-p5810, but you're back down to 1.6TB and
| 50 DWPD, so, 1/4 the Intel P5800X endurance, and worse
| latencies. I highly suspect the drive model here is a
| homage, and in spite of being much newer and very
| expensive, the original is still so much better in so
| many ways. https://www.solidigm.com/content/solidigm/us/e
| n/products/dat...
|
| You also end up paying for what I assume is a circa six
| figure drive, if you are substituting DWPD with more
| capacity than you need. There's something elegant about
| being able to keep using your cells, versus overbuying on
| cells with the intent to be able to rip through them
| relatively quickly.
| zozbot234 wrote:
| Optane practical TBW endurance is _way_ higher than that
| of even TLC flash, never mind QLC or PLC which is the
| current standard for consumer NAND hardware. It even
| seems to go way beyond what 's stated on the spec sheet.
| However, while Optane excels for write-heavy workloads
| (not read-heavy, where NAND actually performs very well)
| these are also power-hungry which is a limitation for
| modern AI workflow.
| PunchyHamster wrote:
| So instead of replacing every 5 years you replace every 5
| years because if you need that level of performance you're
| replacing servers every 5 years anyway
| deepsquirrelnet wrote:
| I worked at Micron in the SSD division when Optane
| (originally called crosspoint "Xpoint") was being made. In my
| mind, there was never a real serious push to productize it.
| But it's not clear to me whether that was due to unattractive
| terms of the joint venture or lack of clear product fit.
|
| There was certainly a time when it seemed they were shopping
| for engineers opinions of what to do with it, but I think
| they quickly determined it would be a much smaller market
| anyway from ssds and didn't end up pushing on it too hard. I
| could be wrong though, it's a big company and my corner was
| manufacturing and not product development.
| chrneu wrote:
| I worked at Intel for a while and might be able to explain
| this.
|
| There were/are often projects that come down from
| management that nobody thinks are worth pursuing. When i
| say nobody, it might not just be engineers but even say 1
| or 2 people in management who just do a shit roll out.
| There are a lot of layers of Intel and if even one layer in
| the Intel Sandwich drag their feet it can kill an entire
| project. I saw it happen a few times in my time there. That
| _one specific node_ that intel dropped the ball on kind of
| came back to 2-3 people in one specific department, as an
| example.
|
| Optane was a minute before I got there, but having been
| excited about it at the time and somewhat following it,
| that's the vibe I get from Optane. It had a lot of
| potential but someone screwed it up and it killed the
| momentum.
| osnium123 wrote:
| Are you referring to the Intel 10nm struggles in your
| reference to 2-3 people?
| empiricus wrote:
| This is actually insane. Do you mean 2-4 people in one
| department basically killed Intel? Roll to disbelief.
| wtallis wrote:
| It's somewhat plausible that a small group of people in
| one department were responsible for the bad bets that
| made their 10nm process a failure. But it was very much a
| _group effort_ for Intel to escalate that problem into
| the prolonged disaster. Management should have stopped
| _believing_ the undeliverable promises coming out of
| their fab side after a year or two, and should have
| started much sooner to design chips targeting fab
| processes that _actually worked_.
| LASR wrote:
| Yes this is pretty common in large enterprise-ey tech
| companies that are successful. There are usually a small
| group of vocal members that have a strong conviction and
| drive to make a vision a reality. This is contrary to
| popular belief that large companies design by committee.
|
| Of course it works exceptionally well when the instinct
| turns out to be right. But can end companies if it isn't.
| rjsw wrote:
| A friend was working at Micron on a rackmount network
| server with a lot of flash memory, I didn't ask at the time
| what kind of flash it used. The project was cancelled when
| nearly finished.
| amluto wrote:
| A ways back, I wrote a sort of database that was memory-
| mapped-file backed (a mistake, but I didn't know that at the
| time), and I would have paid top dollar for even a few GB of
| NVDIMMs that could be put in an ordinary server and could be
| somewhat straightforwardly mounted as a DAX filesystem. I
| even tried to do some of the kernel work. But the hardware
| and firmware was such a mess that it was basically a lost
| cause. And none of the tech ever seemed to turn into an
| actual purchasable product. I'm a bit suspicious that Intel
| never found product-market fit in part because they never had
| a credible product on the NVDIMM side.
|
| Somewhere I still have some actual battery-backed DIMMs (DRAM
| plus FPGA interposer plus awkward little supercapacitor
| bundle) in a drawer. They were not made by Intel, but Intel
| was clearly using them as a stepping stone toward the broader
| NVDIMM ecosystem. They worked on exactly one SuperMicro
| board, kind of, and not at all if you booted using UEFI.
| Rebooting without doing the magic handshake over SMBUS [0]
| first took something like 15 minutes, which was not good for
| those nines of availability.
|
| [0] You can find my SMBUS host driver for exactly this
| purpose on the LKML archives. It was never merged, in part,
| because no one could ever get all the teams involved in the
| Xeon memory controller to reach any sort of agreement as to
| who owned the bus or how the OS was supposed to communicate
| without, say, defeating platform thermal management or
| causing the refresh interval to get out of sync with the DIMM
| temperature, thus causing corruption.
|
| I'm suspicious that everything involved in Optane development
| was like this.
| cogman10 wrote:
| IMO, the reason they didn't sell is the ideal usage for them is
| pairing them with some slow spinning disks. The issue Optane
| had is that SSD capacity grew dramatically while the price
| plummeted. The difference between Optane and SSDs was too
| small. Especially since the M.2 standard proliferated and SSDs
| took advantage of PCI-E performance.
|
| I believe Optane retained a performance advantage (and I think
| even today it's still faster than the best SSDs) but SSDs
| remain good enough and fast enough while being a lot cheaper.
|
| The ideal usage of optane was as a ZIL in ZFS.
| zozbot234 wrote:
| That may have been the ideal usage _back in the day_ , but
| ideal usage now is just for setting up swap. Write-heavy
| workloads are king with Optane, and threshing to swap is the
| prototypical example of something that's so write-heavy it's
| a terrible fit for NAND. Optane might not have been "as fast
| as DRAM" but it was plenty close enough to be fit for
| purpose.
| mort96 wrote:
| That would be fine if I could put it in an M.2 slot. But
| all my computers already have RAM in their RAM slots, and
| even if I had a spare RAM slot, I don't know that I'd trust
| the software stack to treat one RAM slot as a drive...
|
| And their whole deal was making RAM persistent anyway,
| which isn't exactly what I want.
| zozbot234 wrote:
| Optane M.2-format hardware exists.
| mort96 wrote:
| Interesting, all I ever saw advertised was that weird
| persistent kinda slow RAM stick. Does the M.2 version
| just show up as a normal block device or is that too
| trying to be persistent RAM?
| saxonww wrote:
| Iirc it wasn't great because higher power == more heat
| though
| zozbot234 wrote:
| That could be addressed with a small NVMe heatsink.
| They're available and their use is advised already for
| NAND PCIe 4.0 and 5.0 hardware, but they would fit the
| Optane use just as well.
| bushbaba wrote:
| Not just capacity but SSD speeds also improved to the point
| it was good enough for many high memory workloads.
| exmadscientist wrote:
| > The ideal usage of optane was as a ZIL in ZFS.
|
| It was also the best boot drive money could buy. Still is, I
| think, though other comments in the thread ask how it
| compares against today's best, which I'd also love to see.
| gozzoo wrote:
| This concept was very popular back in the days when
| computers used to boot from HDD, but now it doesn't make
| much sense. I wouldn't notice If my laptop boots for 5 sec
| instead of 10.
| exmadscientist wrote:
| At the time of their introduction Optane drives were
| noticeably faster to boot your machine than even the
| fastest available Flash SSD. So in a workstation with
| multiple hard drives installed anyway, buying one to boot
| off of made decent sense.
|
| If they had been cheaper, I think they'd have been
| really, really popular.
| amluto wrote:
| Intel did a spectacularly poor job with the ecosystem around
| the memory cells. They made two plays, and both were flops.
|
| 1. "Optane" in DIMM form factor. This targeted (I think) two
| markets. First, use as slower but cheaper and higher density
| volatile RAM. There was actual demand -- various caching
| workloads, for example, wanted hundreds of GB or even multiple
| TB in one server, and Optane was a route to get there. But the
| machines and DIMMs never really became available. Then there
| was the idea of using Optane DIMMs as persistent storage. This
| was always tricky because the DDR interface wasn't meant for
| this, and Intel also seems to have a lot of legacy tech in the
| way (their caching system and memory controller) and, for
| whatever reason, they seem to be barely capable of improving
| their own technology. They had multiple serious false starts in
| the space (a power-supply-early-warning scheme using NMI or MCE
| to idle the system, a horrible platform-specific register to
| poke to ask the memory controller to kindly flush itself, and
| the stillborn PCOMMIT instruction).
|
| 2. Very nice NVMe devices. I think this was more of a failure
| of marketing. If they had marketed a line of SSDs that, coupled
| with an appropriate filesystem, could give 99% fsync latency of
| 5 microseconds _and they had marketed this_ , I bet people
| would have paid. But they did nothing of the sort -- instead
| they just threw around the term "Optane" inconsistently.
|
| These days one could build a PCM-backed CXL-connected memory
| mapped drive, and the performance might be awesome. Heck, I bet
| it wouldn't be too hard to get a GPU to stream weights directly
| off such a device at NVLink-like speeds. Maybe Intel should try
| it.
| orion138 wrote:
| One of the many problems was trying to limit the use of
| Optane to Intel devices. They should have manufactured and
| sold Optane memory and let other players build on top of it
| at a low level.
| amluto wrote:
| > Optane memory
|
| Which "Optane memory"? The NVMe product always worked on
| non-Intel. The NVDIMM products that I played with only ever
| worked on a very small set of rather specialized Intel
| platforms. I bet AMD could have supported them about as
| easily as Intel, and Intel barely ever managed to support
| them.
| wtallis wrote:
| The consumer "Optane memory" products were a combination
| of NVMe and Intel's proprietary caching software, the
| latter of which _was_ locked to Intel 's platforms. They
| also did two generations of hybrid Optane+QLC drives that
| only worked on certain Intel platforms, because they ran
| a PCIe x2+x2 pair of links over a slot normally used for
| a single X2 or x4 link.
|
| Yes, the pure-Optane consumer "Optane memory" products
| were at a hardware level just small, fast NVMe drives
| that could be use anywhere, but they were never marketed
| that way.
| amluto wrote:
| I feel like this is proving my point. You can't read
| "Optane" and have any real idea of what you're buying.
|
| Also... were those weird hybrid SSDs even implemented by
| actual hardware, or were they part of the giant series of
| massive kludges in the "Rapid Storage" family where some
| secret sauce in the PCIe host lied to the OS about what
| was actually connected so an Intel driver could replace
| the OS's native storage driver (NVMe, AHCI, or perhaps
| something worse depending on generation) to implement all
| the actual logic in software?
|
| It didn't help Intel that some major storage companies
| started selling very, very nice flash SSDs in the mean
| time.
| wtallis wrote:
| > were those weird hybrid SSDs even implemented by actual
| hardware, or were they part of the giant series of
| massive kludges
|
| They were definitely part of the series of massive
| kludges. But aside from the Intel platforms they were
| marketed for, I never found a PCIe host that could see
| both of the NVMe devices on the drive. Some hosts would
| bring up the x2 link to the Optane half of the drive,
| some hosts would bring up the x2 link to the QLC half of
| the drive, but I couldn't find any way to get both links
| active even when the drive was connected downstream of a
| PCIe switch that definitely had hardware support for
| bifurcation down to x2 links. I suspect that with
| appropriate firmware hacking on the host side, it may
| have been possible to get those drives fully operational
| on a non-Intel host.
| myself248 wrote:
| Exactly. I happen to have all AMD sitting around here,
| and buying my first Optane devices was a gamble, because
| I had no idea if they'd work. Only reason I ever did, is
| they got cheap at one point and I could afford the
| gamble.
|
| That uncertainty couldn't have done the market any
| favors.
| thesz wrote:
| In "databases and journals" you rarely update just one byte,
| you do a transaction that updates data, several indexes and
| metadata. All of that needs to be atomic.
|
| Power failure can happen in between any of "1 byte updates with
| crazy latencies." However small latency is, power failure is
| still faster. Usually, there is a write ahead or some other log
| that alleviates the problem, this log is usually written in
| streaming fashion.
|
| What is good, though, is that "blast radius" [1] of failure is
| smaller than usual - failed one byte write rarely corrupts more
| that one byte or cache line. SQLite has to deal with 512 (and
| even more) bytes long possible corruptions on most disks, with
| Optane it is not necessarily so. So, less data to copy, scan,
| etc.
|
| [1] https://sqlite.org/psow.html
| PunchyHamster wrote:
| It's not. You won't be writing one byte, ever (even if you
| had layers that actually supported less-than-block writes),
| because the overhead of instruction would be massive and
| you'd be murdering both latency and bandwidth for anything
| non-trivial
| mort96 wrote:
| I never understood what they're meant to do. Intel seemed to
| picture some future where RAM is persistent; but they were
| never close to fast enough to replace RAM, and the option to
| reboot in order to fix some weird state your system has gotten
| itself into is a _feature_ of computers, not a problem to work
| around.
| ashvardanian wrote:
| I don't have the inside scoop on Intel's current mess, but they
| definitely have a habit of killing off their coolest projects.
| brcmthrowaway wrote:
| Realsense too
| walterbell wrote:
| Related: _" High-bandwidth flash progress and future"_ (15
| comments), https://news.ycombinator.com/item?id=46700384
|
| In an era of RAM shortages and quarterly price increases, Optane
| remains viable for swap and CPU/GPU cache.
| trollbridge wrote:
| Yeah, I've wondered if we might see a revival of this kind of
| technology.
| newsclues wrote:
| in an era of shortages, if there was an optane factory today
| ready to print money...
| walterbell wrote:
| Secondary market surplus pricing (~$1/GB) value accrues to
| the buyer..
| zozbot234 wrote:
| > (~$1/GB)
|
| Isn't that actually crazy good, even insane value for the
| performance and DWPD you get with Optane, especially with
| DRAM being ~$15/GB or so? I don't think ~$1/GB NAND is
| anywhere that good on durability, even if the raw
| performance is quite possibly higher.
| Weryj wrote:
| I've been considering buying 8x64g models and setting them as
| equal priority swap disks (to mitigate the low throughput) for
| this exact reason.
| MrDrMcCoy wrote:
| Can confirm doing so is awesome. Get some slightly bigger
| ones and partition them for additional use as zil. They're
| extremely satisfying to use, and depressing to remember that
| we'll never see their like again.
| amelius wrote:
| For a good technical explanation at the physical level of a
| memory cell:
|
| https://pcper.com/2017/06/how-3d-xpoint-phase-change-memory-...
| FpUser wrote:
| I feel sorry about the situation. From my perspective Optane was
| a godsend for databases. I was contemplating building a system.
| Could've been a pinnacle of vertical scalability for cheap.
| ece wrote:
| Fabs are expensive and all, but maybe running a right-sized fab
| could have still been profitable at making optane for low-latency
| work that it was so good at. Even moreso with RAM prices as they
| are.
| downrightmike wrote:
| PE popped already, now Private credit is already popping,
| having been in too many bubbles, Datacenter bubble popped them.
| The whole system just isn't talking about it because the media
| isn't doing their job to report. There are no more good loans
| to make, the ROI is nil and the interest rate is spot plus
| 5.5%, and businesses don't want to pay 10% interest on a loan.
| We're ina zombie cycle
| dangoodmanUT wrote:
| Optane was crazy good tech, it way just too expensive at the time
| for mass adoption, but the benefits were so good.
|
| Looking at those charts, besides the DWPD it feels like normal
| NVMe has mostly caught up. I occassionally wonder where a gen
| 7/8(?) optane would be today if it caught on, it'd probably be
| nuts.
| zozbot234 wrote:
| > besides the DWPD it feels like normal NVMe has mostly caught
| up.
|
| So what you mean is that on the most important metric of them
| all for many workloads, Flash-based NVMe has _not_ caught up at
| all. When you run a write heavy workload on storage with a
| limited DWPD (including heavy swapping from RAM) higher
| performance actually hurts your durability.
| exmadscientist wrote:
| The actual strength of Optane was on mixed workloads. It's hard
| to write a flash cell (read-erase-write cycle, higher program
| voltage, settling time, et cetera). Optane didn't have any of
| that baggage.
|
| This showed up as _amazing_ numbers on a 50%-read, 50%-write
| mix. Which, guess what, a lot of real workloads have, but
| benchmarks don 't often cover well. This is why it's a great OS
| boot drive: there's so much cruddy logging going on (writes) at
| the same time as reads to actually load the OS. So Optane was
| king there.
| lvl155 wrote:
| It's the best OS drive especially p5800x.
| gozzoo wrote:
| Maybe we can also mention the HP Memristor here.
| jamiek88 wrote:
| Oh I was so excited for that. I _devoured_ any news or blogs or
| rumours about that immediately!
| readitalready wrote:
| These are absolute beasts for database servers, and definitely
| needs to make a comeback.
|
| They suck for large sequential file access, but incredible for
| small random access: databases.
| gigatexal wrote:
| I'm still sad they discontinued them. What's the alternative now
| does anything come close?
| walterbell wrote:
| Small sizes are on secondary market for ~$1/GB.
| zozbot234 wrote:
| Which is a bargain compared to what DRAM costs today. If you
| just include the bare minimum of DRAM for a successful boot
| and immediately set up the entire "small" Optane drive as
| swap, that's a viable workstation-class system for
| comparative peanuts. You can't do this with NAND because the
| write workload of swap kills the media (I suppose it becomes
| viable if you monitor SMART wearout indicators and heavily
| overprovision the storage to leverage the drive's pSLC mode,
| but you're still treating $~0.10/GB hardware as a consumable
| and that will cost you) and of course you can't do it with
| spinning rust because the media is too slow.
| rkagerer wrote:
| My understanding is Optane is still unbeaten when it comes to
| latency. Has anyone examined its use as an OS volume, compared to
| today's leading SSD's? I know the throughput won't be as high,
| but in my experience that's not as important to how responsive
| your machine feels as latency.
| rkagerer wrote:
| Before people claim it doesn't matter due to OS write
| buffering, I should point out a) today's bloated software and
| the many-layered, abstracted I/O stack it's built on tends to
| issue lots of unnecessary flushes, b) read latency is just as
| important as write (if not moreso) to how responsive your OS
| feels, particularly if the whole thing doesn't fit in (or
| preload to) memory.
| speedgoose wrote:
| I configured a hetzner ax101 bare metal server with a 480GB 3d
| xpoint ssd some years ago. It's used as the boot volume and it
| seems fast despite the server being heavily over provisioned,
| but I can't really compare because I don't have a baseline
| without.
| hamdingers wrote:
| > Has anyone examined its use as an OS volume, compared to
| today's leading SSD's?
|
| Late last year I switched from a 1.5tb Optane 905P to a 4tb WD
| Blue SN5000 NVMe drive in a gaming machine and saw improved
| load times, which makes sense given the read and write speeds
| are ~double. No observable difference otherwise.
|
| I'm sure that's not the use case you were looking for. I could
| probably tease out the difference in latency with benchmarks
| but that's not how I use the computer.
|
| The 905P is now in service as an SSD cache for a large media
| server and that came with a big performance boost but the
| baseline I'm comparing to is just spinning drives.
| rkagerer wrote:
| Thanks, that's helpful real-world feedback (not that I
| wouldn't also be interested in some synthetic benchmark
| comparisons from someone else).
| aggieNick02 wrote:
| We benchmarked three of the popular Optane NVMe SSDs about
| three years ago. There was a short window when they were on
| clearance and a popular choice as a cache SSD in TrueNAS.
|
| https://pcpartpicker.com/forums/topic/425127-benchmarking-
| op...
|
| You can compare their benchmarks with the other almost 400
| SSDs we've benchmarked. Most impressive is that three years
| later they are still the top random read QD1 performers,
| with no traditional flash SSD coming anywhere close:
|
| https://pcpartpicker.com/products/internal-hard-
| drive/benchm...
|
| They are amazing for how consistent and boring their
| performance is. Bit level access means no need for TRIM or
| garbage collection, performance doesn't degrade over time,
| latency is great, and random IO is not problematic.
| exmadscientist wrote:
| Unfortunately a gaming machine workload is so read-heavy that
| I wouldn't expect Optane to square up well. Gaming is all
| about read speed and overall capacity. You need that heavy
| I/O mix, especially with low latency deadlines, to see gains
| from Optane. That limited target use case, coupled with
| ignorant benchmarking, always limited them.
| aaronmdjones wrote:
| I have a 16 GiB Optane NVMe M.2 drive in my router as a boot
| drive, running OpenWRT.
|
| It's so incredibly fast and responsive that the LuCI interface
| _completely_ loads the moment I hit enter on the login form.
| dmayle wrote:
| I run two 1.5TB Optanes in raid-0 with XFS (I picked them up
| for $300 each on sale about two years ago). These are limited
| to PCIE 3.0 x4 (about 4GB/s max each). I also have a 64GB
| optane drive I use as my boot drive.
|
| It's hard to tell you, because it's subjective, I don't swap
| back and forth between an SSD and the optane drives. I have my
| old system, which has a 2TB Samsung 980 Pro NVME drive (PCIE
| 4.0 x4, or 8GB/s max) as root, and a Sabrent rocket 4 plus 4TB
| drive secondary (also PCIE 4.0), so I ran sysbench on both
| systems, so I could share the differences. (Old system 5950X,
| new system 9950X3D).
|
| It feels snappier, especially when doing compilations...
|
| Sequential reads: I started with a 150GB fileset, but it was
| being served by the kernel cache on my newer system (256GB RAM
| vs 128GB on the old), so I switched to use 300GB of data, and
| the optanes gave me 5000 MiB/s for sequential read as opposed
| to 2800 MiB/s for the 980 Pro, and 4340 MiB/s for the Rocket 4
| Plus.
|
| Random writes alone (no read workload) The optane system gets
| 2184 MiB/s, the 980 Pro gets 32 MiB/s, and the Rocket 4 Plus
| gets 53 MiB/s.
|
| Mixed workload (random read/write) The optanes get 725/483 as
| opposed to 9/6 for the 980 Pro, and 42/28 for the Rocket 4
| Plus.
|
| 2x1.5TB Optane Raid0: Prep time: `sysbench fileio --file-total-
| size=150G prepare` 161061273600 bytes written in 50.41 seconds
| (3047.27 MiB/sec). Benchmark:
| `sysbench fileio --file-total-size=150G --file-test-mode=rndrw
| --max-time=60 --max-requests=0 run` WARNING: --max-time
| is deprecated, use --time instead sysbench 1.0.20
| (using system LuaJIT 2.1.1741730670) Running the
| test with following options: Number of threads: 1
| Initializing random number generator from current time
| Extra file open flags: (none) 128 files, 1.1719GiB each
| 150GiB total file size Block size 16KiB Number
| of IO requests: 0 Read/Write ratio for combined random
| IO test: 1.50 Periodic FSYNC enabled, calling fsync()
| each 100 requests. Calling fsync() at the end of test,
| Enabled. Using synchronous I/O mode Doing
| random r/w test Initializing worker threads...
| Threads started! File operations:
| reads/s: 46421.95 writes/s:
| 30947.96 fsyncs/s: 99034.84
| Throughput: read, MiB/s: 725.34
| written, MiB/s: 483.56 General
| statistics: total time:
| 60.0005s total number of events:
| 10584397 Latency (ms): min:
| 0.00 avg:
| 0.01 max:
| 1.32 95th percentile:
| 0.03 sum:
| 58687.09 Threads fairness: events
| (avg/stddev): 10584397.0000/0.00
| execution time (avg/stddev): 58.6871/0.00
|
| 2TB Nand Samsung 980 Pro: Prep time: `sysbench fileio --file-
| total-size=150G prepare` 161061273600 bytes written in 87.15
| seconds (1762.53 MiB/sec). Benchmark:
| `sysbench fileio --file-total-size=150G --file-test-mode=rndrw
| --max-time=60 --max-requests=0 run` WARNING: --max-time
| is deprecated, use --time instead sysbench 1.0.20
| (using system LuaJIT 2.1.1741730670) Running the
| test with following options: Number of threads: 1
| Initializing random number generator from current time
| Extra file open flags: (none) 128 files, 1.1719GiB each
| 150GiB total file size Block size 16KiB Number
| of IO requests: 0 Read/Write ratio for combined random
| IO test: 1.50 Periodic FSYNC enabled, calling fsync()
| each 100 requests. Calling fsync() at the end of test,
| Enabled. Using synchronous I/O mode Doing
| random r/w test Initializing worker threads...
| Threads started! File operations:
| reads/s: 594.34 writes/s:
| 396.23 fsyncs/s: 1268.87
| Throughput: read, MiB/s: 9.29
| written, MiB/s: 6.19 General
| statistics: total time:
| 60.0662s total number of events:
| 135589 Latency (ms): min:
| 0.00 avg:
| 0.44 max:
| 15.35 95th percentile:
| 1.73 sum:
| 59972.76 Threads fairness: events
| (avg/stddev): 135589.0000/0.00 execution
| time (avg/stddev): 59.9728/0.00
|
| 4TB Sabrent Rocket 4 Plus: Prep time: `sysbench fileio --file-
| total-size=300G prepare` 322122547200 bytes written in 152.39
| seconds (2015.92 MiB/sec). Benchmark:
| `sysbench fileio --file-total-size=300G --file-test-mode=rndrw
| --max-time=60 --max-requests=0 run` WARNING: --max-time
| is deprecated, use --time instead sysbench 1.0.20
| (using system LuaJIT 2.1.1741730670) Running the
| test with following options: Number of threads: 1
| Initializing random number generator from current time
| Extra file open flags: (none) 128 files, 2.3438GiB each
| 300GiB total file size Block size 16KiB Number
| of IO requests: 0 Read/Write ratio for combined random
| IO test: 1.50 Periodic FSYNC enabled, calling fsync()
| each 100 requests. Calling fsync() at the end of test,
| Enabled. Using synchronous I/O mode Doing
| random r/w test Initializing worker threads...
| Threads started! File operations:
| reads/s: 2690.28 writes/s:
| 1793.52 fsyncs/s: 5740.92
| Throughput: read, MiB/s: 42.04
| written, MiB/s: 28.02 General
| statistics: total time:
| 60.0155s total number of events:
| 613520 Latency (ms): min:
| 0.00 avg:
| 0.10 max:
| 8.22 95th percentile:
| 0.32 sum:
| 59887.69 Threads fairness: events
| (avg/stddev): 613520.0000/0.00 execution
| time (avg/stddev): 59.8877/0.00
| pgwalsh wrote:
| Sure, they were expensive but they have great endurance and
| sustained read and write speeds. I use one in my car for camera
| recordings. I had gone through several other drives but this one
| has been going on 3 or 4 years now without issue. I have a couple
| more in use too. It's a shame this tech is going away because
| it's excellent.
| exmadscientist wrote:
| Around the time of Optane's discontinuation, the rumor mill was
| saying that the real reason it got the axe was that it couldn't
| be shrunk any, so its costs would never go down. Does anyone know
| if that's true? I never heard anything solid, but it made a _lot_
| of sense given what we know about Optane 's fab process.
|
| And if no shrink was possible, is that because it was (a)
| possible but too hard; (b) known blocks to a die shrink; or (c)
| execs didn't want to pay to find out?
| zozbot234 wrote:
| That's at least physically half-plausible, but it would be a
| terrible reason if true. 3.5 in. format hard drives can't be
| shrunk any, and their costs are correspondingly high, but they
| still sell - newer versions of NVMe even provide support for
| them. Same for LTO tape cartridges. Perhaps they expected other
| persistent-memory technologies to ultimately do better, but we
| haven't really seen this.
|
| Worth noting though that Optane is also power-hungry for writes
| compared to NAND. Even when it was current, people noticed
| this. It's a blocker for many otherwise-plausible use cases,
| especially re: modern large-scale AI where power is a key
| consideration.
| wtallis wrote:
| > 3.5 in. format hard drives can't be shrunk any,
|
| You're looking at the entirely wrong kind of shrinking. Hard
| drives are still (gradually) improving storage density: the
| physical size of a byte on a platter _does_ go down over
| time.
|
| Optane's _memory cells_ had little or no room for shrinking,
| and Optane lacked 3D NAND 's ability to add more layers with
| only a small cost increase.
| hedora wrote:
| I think it was killed primarily because the DIMM version had a
| terrible programming API. There was no way to pin a cache line,
| update it and flush, so no existing database buffer pool
| algorithms were compatible with it. Some academic work tried to
| address this, but I don't know of any products.
|
| The SSD form factor wasn't any faster at writes than NAND +
| capacitor-backed power loss protection. The read path was
| faster, but only in time to first byte. NAND had comparable /
| better throughput. I forget where the cutoff was, but I think
| it was less than 4-16KB, which are typical database read sizes.
|
| So, the DIMMs were unprogrammable, and the SSDs had a
| "sometimes faster, but it depends" performance story.
| myself248 wrote:
| It sounds like they didn't do a good job of putting the DIMM
| version in the hands of folks who'd write the drivers just
| for fun.
|
| The read path is sort of a wash, but writes are still
| unequalled. NAND writes feel like you're mailing a letter to
| the floating gate...
| zozbot234 wrote:
| Isn't this addressed by newer PCIe standards? Of course,
| even the "new" Optane media reviewed in OP is stuck on PCIe
| 4.0...
| exmadscientist wrote:
| The DIMMs were their own shitshow and I don't know how they
| even made it as far as they did.
|
| The SSDs were never going to be dominant at straight read or
| write workloads, but they were absolutely king of the hill at
| mixed workloads because, as you note, time to first byte was
| so low that they switched between read and write faster than
| anything short of DRAM. This was really, really useful for a
| lot of workloads, but benchmarkers rarely bothered to look at
| this corner... despite it being, say, the exact workload of
| an OS boot drive.
|
| For years there was nothing that could touch them in that
| corner (OS drive, swap drive, etc) and to this day it's
| unclear if the best modern drives _still_ can or can 't
| compete.
| georgeburdell wrote:
| Flash has the same shrink problem. And the solution for Optane
| was the same: go 3D
| exmadscientist wrote:
| I don't think the shrink problem is at all the same for the
| two technologies. There are some really weird materials and
| production steps in Optane that are simply not present when
| making Flash cells.
| PunchyHamster wrote:
| durability drops quickly with shrinking flash, we won't see
| much smaller cells, the growth has been MLC-TLC-> QLC and
| stacking
| rkagerer wrote:
| Did anyone ever see retention issues like this guy reported on
| one of his older models?
|
| https://goughlui.com/2024/07/28/tech-flashback-intel-optane-...
| zozbot234 wrote:
| That's data retention issues on the very first read-through of
| the media after sitting in cold storage for many years, with
| subsequent performance returning to normal. It's definitely
| something to be aware of (and kudos to the blog poster for
| running that experiment) but worn-out NAND will behave a lot
| worse than that.
| userbinator wrote:
| Retention issues with _completely new_ Optane is not great
| compared to SLC NAND, which Optane was supposed to beat. I
| have USB drives with the latter, probably a few dozen cycles
| at most, yet they have had no problems keeping data intact
| for over a decade.
| zozbot234 wrote:
| That's not _completely new_ , it's just new old stock.
| That's the whole point of why the blogger ran that
| experiment.
| twotwotwo wrote:
| One potential application I briefly had hope for was really good
| power loss protection in front of a conventional Flash SSD. You
| only need a little compared to the overall SSD capacity to be
| able to correctly report the write was persisted, and it's always
| running, so there's less of a 'will PLP work when we really need
| it?' question. (Maybe there's some use as a read cache too? Host
| RAM's probably better for that, though.) It's going to be
| rewritten lots of times, but it's supposed to be ready for that.
|
| It seems like there's a very small window, commercially, for new
| persistent memories. Flash throughput scales really cost-
| efficiently, and a lot is already built around dealing with the
| tens-of-microseconds latencies (or worse--networked block
| storage!). Read latencies you can cache your way out of, and
| writers can either accept commit latency or play it a little fast
| and loose (count a replicated write as safe enough or...just not
| be safe). You have to improve on Flash by enough to make it worth
| the leap while remaining cheaper than other approaches to the
| same problem, and you have to be confident enough in pulling it
| off to invest a ton up front. Not easy!
| zozbot234 wrote:
| > It seems like there's a very small window, commercially, for
| new persistent memories. Flash throughput scales really cost-
| efficiently
|
| Flash is no bueno for write-heavy workloads, and the random-
| access R/W performance is meh compared to Optane. MLC and SLC
| have better durability and performance, but still very mid.
| hedora wrote:
| Any decent SSD has capacitor (enterprise) or battery backed
| (phones) DRAM. Therefore, a sync write is just "copy the data
| to an I/O buffer over PCIe".
|
| For databases, where you do lots of small scattered writes, and
| lots of small overwrites to the tail of the log, modern SSDs
| coalesce writes in that buffer, greatly reducing write wear,
| and allowing the effective write bandwidth to exceed the media
| write bandwidth.
|
| These schemes are much less expensive than optane.
| olavgg wrote:
| I have tried multiple enterprise SSD's, for sync writes.
| Nothing comes close to Optane Dimm, even Optane NVMe is 10x
| slower than PDIMMS.
|
| https://forums.servethehome.com/index.php?threads/so-i-
| teste...
| wtallis wrote:
| > One potential application I briefly had hope for was really
| good power loss protection in front of a conventional Flash
| SSD.
|
| That was _never_ going to work out. Adding an entirely new kind
| of memory to your storage stack was never going to be easier or
| cheaper than adding a few large capacitors to the drive so it
| could save the contents of the DRAM that the SSD still needed
| whether or not there was Optane in the picture.
| myself248 wrote:
| My kingdom for a MicroSD card with Optane inside. My dashcam
| wants it soooo badly.
| bluedino wrote:
| Now do Intel's HBM/CPU Max
| ritcgab wrote:
| All those nice numbers are just beaten by the unit cost. And the
| ecosystem is a mess.
| Havoc wrote:
| I've got a bunch of them in my homelab. They're good if applied
| to tasks that leverage their attributes.
___________________________________________________________________
(page generated 2026-03-15 23:00 UTC)