[HN Gopher] Linux Achieves 5.1M IOPS Per-Core with AMD Zen 3 and...
___________________________________________________________________
Linux Achieves 5.1M IOPS Per-Core with AMD Zen 3 and Intel Optane
Author : ksec
Score : 113 points
Date : 2021-09-30 14:30 UTC (8 hours ago)
(HTM) web link (www.phoronix.com)
(TXT) w3m dump (www.phoronix.com)
| thepenis wrote:
| Wow! This is OFF TOPIC.
|
| Hackernews is _only_ for covid and anal sex conversations.
|
| Anything to do with computing is off-topic.
| ThinkBeat wrote:
| I need some context.
|
| Linux developers have made improvements to specific parts of the
| IO sub system. These changes will give Linux better IOPS
| performance.
|
| Prior test are quoted at 3,8 IOPS. They ran the old code on a MD
| Ryzen Threadripper 3970X at 3,8 IOPS.
|
| Now he got a better / fast CPU AMD Ryzen 9 5950X and it achieves
| an impressive 5.1M
|
| It seems to me that the new test does not make it clear how much
| code changes helped and how much a new CPU helped?
|
| Running the old version on the new CPU or the other way around
| would make it easier to compare?
| wtallis wrote:
| My understanding from Jens Axboe's tweets is that he ran the
| same software test on his old hardware and his new hardware.
| Separately, he's been using this test for quite a while to
| track the progress of software optimizations.
| willis936 wrote:
| I genuinely want to know more. What specific model Optane drive
| was used? A lot of the newer stuff is greatly hampered by the QLC
| backend.
| wtallis wrote:
| There are no products that mix second-generation Optane with
| QLC. The only two products that mix first-generation Optane
| with QLC are consumer drives that are don't work on AMD
| platforms. Jens is testing using the second-generation _server_
| Optane drives (P5800X), and getting performance that 's
| completely impossible with any of the consumer Optane products,
| because all of those are still limited to PCIe gen3 speeds.
| COGlory wrote:
| Wendell from Level1Techs has been beating his drum about pushing
| IOPS on these newer hardware platforms, and has some great videos
| about his journey and what is fundamentally broken under the hood
| with how software interacts with and fails to anticipate hardware
| as fast as the new Optane drives.
|
| https://www.youtube.com/watch?v=RfrMnVpPuVw - The initial video
| attempting to clear 20 million IOPS on Threadripper
|
| https://www.youtube.com/watch?v=WEMszbMUj68 - 15 million IOPS on
| Xeon 8380s (you can see how this is going)
|
| https://www.youtube.com/watch?v=9Gu_rT8N0-U - Discussion with
| Allyn Malventano of Intel
|
| My broad interpretation of this is that the software layer was
| not ready for things like 64+ core systems on a single socket,
| and PCI-E 4 drives moving data this fast. Nice to see it's
| starting to catch up.
| wtallis wrote:
| I think the most fundamental way in which the software
| ecosystem is unprepared doesn't require Optane storage or 64+
| core systems to demonstrate.
|
| When CPU core counts first started growing, storage was still
| effectively single-threaded: a hard drive can only work on
| servicing one request at a time (though you can get a small
| throughput increase by keeping several requests in the queue),
| and scaling up to arrays of many drives was much more effective
| at increasing sequential throughput than random IO performance.
|
| With flash-based storage, the SSD now has far more parallelism
| than your CPU. Having one pending IO operation per CPU core
| isn't enough to keep your SSDs busy. Having one pending IO per
| software thread and doing lots of context switches to run more
| threads than you have cores _still_ won 't keep your drives
| busy, unless you're only doing IO in large chunks. So
| synchronous, blocking IO APIs are unable to make full use of
| your storage performance. Optane's low latency doesn't save you
| from this problem; it just highlights that context switches are
| also too expensive.
| eptcyka wrote:
| Funny that, some people argue that the future of I/O is
| synchronous memory-mapped devices.
| wtallis wrote:
| That can potentially work, but only by getting the
| filesystem/block storage/PCIe layers of the IO stack out of
| the way, and even then it's only plausible for Optane or
| similarly low-latency storage. It'll never work for flash,
| because the latency of flash will never be low enough.
| jeffbee wrote:
| It is not only the software layer that was not prepared for so
| many. Some NVMe devices support only 64 I/O queues (this is how
| many you need for Windows compatibility certification). Some
| NICs only have 32 receive queues. It's now trivial to build a
| box with 256 threads, so these numbers don't look great.
| wtallis wrote:
| > Some NVMe devices support only 64 I/O queues (this is how
| many you need for Windows compatibility certification).
|
| You're talking specifically about server environments there,
| right? Because consumer NVMe devices started out with far
| lower queue counts (eg. 7) and even the top consumer SSD
| product lines have struggled to keep pace with growing
| Threadripper core counts. 64 queue support is still something
| of a luxury among consumer NVMe devices.
| jeffbee wrote:
| That sucks. I can't recall having personally seen one that
| didn't support at least 64 submission queues.
| wtallis wrote:
| Samsung's current flagship 980 PRO consumer drive
| supports 128 queues, but the previous generation (970
| PRO/EVO/EVO Plus) only supported 32. Their first two
| generations were limited to 8 and 7 queues. I wouldn't be
| surprised if these limits also applied to their entry-
| level enterprise SSDs that used the same controllers.
| MR4D wrote:
| How can we find this information out easily (not just for
| Samsung, but for others)?
|
| Even Anandtech only tested 32 queues on the 980 Pro [0].
|
| [0] - https://www.anandtech.com/show/16087/the-
| samsung-980-pro-pci...
| wtallis wrote:
| Number of queues is not the same as the number of
| operations in a queue.
|
| Supporting more queues means that you need less
| communication and synchronization between CPU cores;
| ideally, you have at least one queue per core so that
| each CPU core can submit commands to the SSD
| independently.
|
| I didn't test queue depths higher than 32 when I reviewed
| the 980 PRO, because it's a consumer drive and consumer
| workloads hardly ever produce double-digit queue depths.
| Even going all the way up to QD32 is fairly unrealistic.
|
| Server workloads can be quite different, and some of them
| genuinely can result in a large number of simultaneous IO
| requests. Consumer workloads have barely even started to
| evolve to take advantage of SSDs, and hard drives aren't
| quite entirely gone from the consumer PC market.
|
| As for how to check how many queues a NVMe drive
| supports: the easy way is to install the drive in a
| system with a very large number of CPU cores, and see how
| many queues the Linux NVMe driver creates (check the
| drive's sysfs directory). My understanding from reading
| the NVMe spec is that you can't necessarily query the
| drive itself for the number of supported queues after the
| operating system's NVMe driver has already done that and
| set up its queues.
| antongribok wrote:
| Is this not the correct way to do it? $
| sudo nvme get-feature /dev/nvme0 -f 7 get-
| feature:0x7 (Number of Queues), Current value:0x7f007f
| wtallis wrote:
| That's the number of queues _requested_ by the host
| system. If you ran that on a system with four CPU cores
| (and no SMT), you 'd likely see that same drive return 4
| instead of 7.
| [deleted]
| MR4D wrote:
| Linux dominance in the datacenter is about to grow even more.
| Awesome news!
___________________________________________________________________
(page generated 2021-09-30 23:02 UTC)