[HN Gopher] How much memory bandwidth do large Amazon instances ...
___________________________________________________________________
How much memory bandwidth do large Amazon instances offer?
Author : ingve
Score : 66 points
Date : 2024-01-19 14:47 UTC (8 hours ago)
(HTM) web link (lemire.me)
(TXT) w3m dump (lemire.me)
| jakebsky wrote:
| Running this code on an "on-prem" AMD EPYC 9454P with 12 x 64 GB
| DIMMs: 1 37.4 2 73.3 3 107.3 4
| 141.4 5 171.6 6 199.5 7 226.0 8 251.1
| 9 235.4 10 243.0 11 264.5 12 281.9 13
| 303.7 14 323.0 15 339.6 16 354.4 17 299.0
| 18 286.3 19 300.9 20 310.6 21 325.7 22
| 339.2 23 352.6 24 364.3 25 305.8 26 309.0
| 27 319.6 28 326.5 29 335.4 30 345.5 31
| 356.7 32 364.9
|
| And then it settles around there.
| bgirard wrote:
| What are the units? MB/s, GB/s
| deadmutex wrote:
| looks like GB/s
| buzzdenver wrote:
| So around 3x of what AWS has? I guess this is more about what
| motherboard they have, and not an on-prem vs cloud difference.
| champtar wrote:
| Number of DIMM is important, I know the standard BOM at my
| company has way more memory that we use just to have more
| bandwidth.
| adrian_b wrote:
| 1.5 times greater speed comes from 768-bit vs. 512-bit width
| of the memory interface.
|
| Another 1.5 times greater speed comes from DDR5-4800 vs.
| DDR4-3200.
|
| The rest may be from virtualization and other overheads.
| tekla wrote:
| You could also probably just ask your TAM for this info.
| dpkirchner wrote:
| It'd be interesting to compare the results.
| nostrebored wrote:
| As a former AWS employee -- answer will vary wildly!
| danburgo wrote:
| Do you have more information on the results obtained? any
| information on how to enable NUMA-aware code ?
|
| The r6i.metal instance says up to 50 Gbps on Network bandwidth:
| https://aws.amazon.com/ec2/instance-types/r6i/
| swiftcoder wrote:
| 50 Gbps is only ~6 GB/s, so you are unlikely to saturate memory
| bandwidth via network traffic alone
| vlovich123 wrote:
| Typically there are NUMA-aware memory allocators. I don't
| believe glibc is but mimalloc and the non gperftools tcmalloc
| should be. The Linux kernel is NUMA-aware and will try to avoid
| shifting work to a different NUMA node but it may depending on
| what else is happening - it's a generic algorithm trying to
| balance CPU utilization, latency, and memory bandwidth. If
| you're using a higher level language like Go or Java then there
| may be more involved. Java has a NUMA aware allocator while
| Go's design requires it to also have a NUMA aware scheduler
| which I don't believe it does.
|
| You could go extreme and start pinning your threads to specific
| CPUs to tune the code manually instead of relying on the kernel
| if you know memory bandwidth is extremely important and you
| won't have much CPU contention to worry about in terms of
| getting work scheduled in a timely manner.
|
| All that being said, you typically also need to design your
| application from the ground up to be NUMA aware to take full
| advantage so that you can set up your allocations to happen on
| the right zone & whatnot.
| EarthLaunch wrote:
| For the curious like me, here's the script running on consumer
| gaming Intel hardware (i9 14900K, 2x 48GB DDR5 6600MHz CL32 XMP,
| Windows 11): 1 27.6 2 50.5 3 62.5
| 4 68.3 5 75.6 6 82.6 7 87.4 8 90.7
| 9 93.0 10 94.3 ...repeating
|
| I expected a bit more, my guess would have been around 120 GB/s.
| I've been playing with LLMs and this hardware is about as fast as
| memory gets on consumer Intel without overclocking.
|
| Screenshot showing usage: https://i.imgur.com/okdqgG9.png
| lowbloodsugar wrote:
| >You can double this amount of bandwidth with NUMA-aware code,
| but it will require further engineering.
|
| So is this benchmark really useful? Seems like it's engineered
| _enough_ that it probably isn 't relevant for most use cases
| (like running java, go or python services), and _not_ engineered
| enough so it 's not relevant for performant systems like
| databases. (That said NUMA simply isn't that hard, even with Java
| which is also NUMA aware).
| shivam543 wrote:
| I changed data_volume to 8GB and ran on my 4 year old laptop (Amd
| Ryzen 4800H + 2 x 8GB 3200MHz DDR4 running NixOS with Linux
| 6.1.71) I get around 36 GB/s 1 20.9 2 33.4
| 3 35.7 4 36.2 5 35.2 6 35.4 7 34.7
| 8 34.9 9 34.0 10 34.2 11 34.4 12 34.3
| 13 33.8 14 33.6 15 33.4 16 32.9
|
| I expected 51.2 GB/s (2 x 3200 x 1e6 Transfers/s of 8 bytes each)
| adrian_b wrote:
| Either because it is a laptop CPU or (less likely) because it
| is an older Zen 2 might prevent it to reach the DRAM
| throughput.
|
| Running the test on a desktop Zen 3 (5900X) with slower ECC
| DDR4-2666, i.e. with a maximum throughput of 42.7 GB/s,
| provides for 2 or more threads a throughput around 39 GB/s,
| with a maximum of 39.5 GB/s at 4 threads.
| andrewgross wrote:
| 7950x3d w/ 128GB at stock timings (~3200MT/s?). Showed a high
| base but no increase with threads, need to investigate what is
| happening. 1 54.7 2 50.6 3 49.6
| 4 48.2 5 47.9 6 47.4 7 47.1 8 46.6
| 9 46.5 10 46.2 11 46.1 12 45.9 13 45.8
| 14 45.7 15 45.7 16 45.7 17 45.7 18 45.8
| 19 45.9 20 45.8 21 45.8 22 45.6 23 45.6
| 24 45.5 25 45.5 26 45.5 27 45.5 28 45.4
| 29 45.4 30 45.4 31 45.4 32 45.4
| shivam543 wrote:
| For 2 x 3200 MHz, 51.2 GB/s is the theoretical limit. I guess
| your single core is good enough to use all bandwidth. How many
| RAM sticks do you have?
| andrewgross wrote:
| 4 x 32GB for now. I need to investigate manual OC as EXPO
| doesn't work with all 4 slots populated. Another option is to
| try the 192GB 4x48GB Corsair kits at 5200.
| seiferteric wrote:
| 7950x with 32GB @ 6000MT/s: 1 52.6 2
| 78.1 3 71.0 4 74.3 5 71.3 6
| 72.5 7 70.0 8 69.6 9 68.4 10
| 68.7 11 68.5 12 68.3 13 68.3 14
| 68.0 15 67.8
| Stagnant wrote:
| 7950x with 4x32GB @ 3800MHz here. Getting similar results
| except my single thread performance is the worst (around 40),
| otherwise it looks almost identical. I assume my poor single
| thread perf is due to having capped max CPU core clock at
| 4.5Ghz. Getting 4 sticks to boot was a pain, haven't really
| bothered to change anything once I finally got it running
| stable.
| acdha wrote:
| For reference, this is an M1 Pro - the first run is consistently
| half speed so I'm guessing that's some kind of scheduler
| adjustment: 1 59.9 2 109.1 3
| 108.4 4 109.1 5 108.0 6 109.3 7
| 109.4 8 110.5 9 119.3 10 123.9
| dd_xplore wrote:
| This is why M series chips "feels" faster
| sapiogram wrote:
| No it's not.
| jmole wrote:
| M1 Pro Macbook 32GB: 1 51.1 2 99.4
| 3 108.2 4 121.3 5 116.4 6 122.9 7
| 122.0 8 123.2 9 125.6 10 124.9
|
| Xeon E-2288G (aka i9-9900K) 128GB DDR4-2666: $
| ./a.out 1 19.7 2 27.8 3 30.0 4
| 31.7 5 33.6 6 34.4 7 33.9 8 33.2
| 9 32.2 10 32.2 11 32.3 12 32.1 13
| 31.9 14 31.5 15 31.2 16 30.9
| skywhopper wrote:
| Interesting stuff, although I'll admit when I read the headline I
| took it to mean the literal `large` instance types, which are
| actually quite small as AWS lingo goes. Whereas the author means
| _actual_ LARGE instances, ie, an entire box with no hypervisor
| layer.
| gavinray wrote:
| Note that you can get a significant speedup by using "restrict"
| on the pointer in the "sum()" function to tell the compiler that
| there is no mutation or aliasing happening during the call's
| lifetime.
|
| On Godbolt, with a size of 100MB (due to limits), this improves
| the speed using LLVM 17 (https://godbolt.org/z/6dW1h8aev) from:
| 1 15.1 2 15.9
|
| To (https://godbolt.org/z/sh3489Mxv) 1 12.8
| 2 13.7
| scode2 wrote:
| Isn't higher better?
| d0mine wrote:
| yes. The result is max bandwidth in GB/s
| https://github.com/lemire/Code-used-on-Daniel-Lemire-s-
| blog/...
| secondcoming wrote:
| GCP VM, 40GB: Model name: Intel(R)
| Xeon(R) CPU @ 2.20GHz CPU family: 6
| Model: 79 Thread(s) per core: 2
| Core(s) per socket: 8 1 11.4 2 21.3
| 3 30.8 4 39.5 5 48.2 6 56.8 7
| 64.3 8 71.0 9 61.2 10 64.1 11
| 67.7 12 72.4 13 74.3 14 78.3
| 15 81.8 16 84.9
| speedgoose wrote:
| Running on an 2xARM Cavium ThunderX2 CN9980 with 1TB of ram and
| 256 threads (Simula Ex3): ~120.
|
| Running on an 2xAMD EPYC 7763 64-Core Processor, with 256GB of
| ram and 256 threads (EuroHPC LUMI standard CPU node): ~38.
|
| Running on my Macbook Pro M1 16GB on battery power, but with 8GB
| of data to not swap as I have other apps running: ~59.
| dd_xplore wrote:
| I changed data_volume to 8GB and ran on my desktop (Amd Ryzen5
| 7600X + 2x8GB DDR5-4800 running @ 5000 Mhz)(Win11) I get around
| 55 GB/s PS E:\> .\bandwidth.exe 1 54.0
| 2 56.7 3 56.1 4 55.3 5 54.6 6
| 54.3 7 53.9 8 53.1 9 53.4 10 53.1
| 11 52.3 12 52.8
|
| 55 is kinda low for DDR5 I guess..
|
| On S23 Ultra (Snapdragon 8 Gen 2+, 12GB LPDDR5X @4200MHz)
| u0_a339@localhost ~> ./bandwidth 1 34.4 GB/s 2 38.7
| GB/s 3 42.0 GB/s 4 42.3 GB/s 5 42.4 GB/s
| 6 40.2 GB/s 7 41.0 GB/s 8 40.3 GB/s
|
| On a NanoPi-R6C (RockChip RK3588s @2.4Ghz + 8GB LPDDR4X)
| pi@nanopi ~> ./bandwidth 1 22.1 GB/s 2 26.2 GB/s
| 3 27.5 GB/s 4 27.7 GB/s 5 27.1 GB/s 6 27.1 GB/s
| 7 27.0 GB/s 8 26.9 GB/s
| CraigJPerry wrote:
| Raspberry pi5 8Gb, with data volume set to 4Gb
| 1 12.9 2 12.4 3 12.1 4 11.6
|
| E5-1620 v2 @ 3.70GHz, 32Gb 1 16.9 2
| 29.5 3 39.8 4 37.8 5 35.3 6
| 35.3 7 34.9 8 34.7
| timzaman wrote:
| > You can double this amount of bandwidth with NUMA-aware code,
| but it will require further engineering.
|
| If you have non uniform memory, you may be shooting yourself in
| the foot. Author used a dual socket system. If you are not
| careful, and your code is not NUMA aware, you may end up with
| _less_ memory bandwidth for your application, than a single-
| socket would have. Can be avoided by prefixing `numactl
| --cpubind=0` to your command
___________________________________________________________________
(page generated 2024-01-19 23:01 UTC)