Max swap size on amd64 = 64GB? reddit / freebsd ---------------------------------------------------------------------- heathenskwerl · 2022-03-25 18:28 I had to recreate my root zpool after moving to smaller disks, so I increased the swap partition size in the process (my root zpool is basically just a giant /tmp as almost everything is on other zpools). However I'm seeing the following in my dmesg after I increased the size of my swap partition: GEOM_ELI: Device mirror/swap.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: software WARNING: reducing swap size to maximum of 65536MB per unit 'swapinfo' shows the following: Device 1G-blocks Used Avail Capacity /dev/mirror/swap.eli 64 0 64 0% However 'gmirror list' shows that the size is 128GB: Geom name: swap State: COMPLETE Components: 2 Balance: round-robin Slice: 4096 Flags: NONE GenID: 0 SyncID: 1 ID: 1874776346 Type: AUTOMATIC Providers: 1. Name: mirror/swap Mediasize: 137438952960 (128G) Sectorsize: 512 Mode: r1w1e1 Consumers: 1. Name: ada2p3 Mediasize: 137438953472 (128G) 2. Name: ada3p3 Mediasize: 137438953472 (128G) Any thoughts as to how to convince it to use the entire 128GB? I have 128GB ram, as shown by dmesg: real memory = 137438953472 (131072 MB) avail memory = 133822480384 (127623 MB) ---------------------------------------------------------------------- nktje · 2022-03-25 19:07 How about using multiple 64G swaps? ---------------------------------------------------------------------- heathenskwerl · 2022-03-25 20:57 I'd do that, except the swap partition isn't on the end of the drive and I don't know how to fix that without going through the whole process again (which I'd very much like to avoid). => 40 3907029088 ada2 GPT (1.8T) 40 409600 1 efi (200M) 409640 1024 2 freebsd-boot (512K) 410664 268435456 3 freebsd-swap (128G) 268846120 3638183008 4 freebsd-zfs (1.7T) ---------------------------------------------------------------------- FUZxxl · 2022-03-26 00:20 64 GB is the maximum size of a swap partition. You can add multiple swap partitions to exceed this limit. ---------------------------------------------------------------------- FUZxxl · 2022-03-26 00:23 Split the swap partition into two: swapoff /dev/ada2p3 gpart delete -i 3 ada2 gpart add -t freebsd-swap -s 64G -i 3 ada2 gpart add -t freebsd-swap -s 64G -i 5 ada2 swapon /dev/ada2p3 /dev/ada2p5 Partitions do not need to be on the disk in ascending order. ---------------------------------------------------------------------- 1s44c · 2022-03-26 02:27 Did you just want to show off that you have 128GB of ram? Not sure what you are running but swap doesn't have to equal physical ram. ---------------------------------------------------------------------- theamigan · 2022-03-26 02:33 Lol, imagine actually filling even 32GB of swap. Holy working set batman. The only use I can think of is for a giant swap-backed md(4). ---------------------------------------------------------------------- heathenskwerl · 2022-03-26 03:15 It's actually used swap several times before, even with 128GB of RAM, and it's physically maxed out (cannot support more RAM). Swap may not have to equal physical RAM, but I wasn't using this disk space for anything else--2TB is completely overkill for the OS drive but I don't have anything smaller that still *works*. As to what I'm running, it's a large rack-mount ZFS-based file server (SuperMicro 846). ---------------------------------------------------------------------- heathenskwerl · 2022-03-26 03:18 I haven't filled the original 64GB of swap but I've used close to 20GB before. Just don't have anything else useful to do with the disk space so figured why not. Edit: Sorry, I realized later this was incorrect--I have used enough swap that I had about 20GB of swap space *left*, not 20GB of swap space *in use*. ---------------------------------------------------------------------- theamigan · 2022-03-26 03:19 Wow, how are you swapping so much just on a file server? ARC obviously won't swap since the data is already backed by disk. If it isn't a fileserver, I would expect the thrashing to be unbearable. ---------------------------------------------------------------------- heathenskwerl · 2022-03-26 03:20 Did *not* know that they didn't need to be in ascending order. Thanks, that's really useful information, I'll go ahead and set it up that way. Edit: Everything working fine. Swap's in the fstab and all 128GB is being used. Thanks for your help! Device 1G-blocks Used Avail Capacity /dev/mirror/swapa.eli 64 0 64 0% /dev/mirror/swapb.eli 64 0 64 0% Total 128 0 128 0% ---------------------------------------------------------------------- heathenskwerl · 2022-03-26 03:20 Is there a reason for this that you know of? The kernel can clearly address more memory than 64GB. ---------------------------------------------------------------------- heathenskwerl · 2022-03-26 03:26 It is a rack-mount file server (SuperMicro 846) but it also gets used as a general server in a pinch, as it's the machine I have with the greatest amount of RAM (even allowing for the ARC memory usage). As one example, I use it for the creation of solid 7zip archives of really large files. This uses a truly ridiculous amount of RAM. ---------------------------------------------------------------------- theamigan · 2022-03-26 03:27 Ahhh, yes, the archiving use case seals it. Makes perfect sense. Though I stand by saying that if you fill >64GB of swap, you're probably better off just waiting for the kernel to OOM kill rather than waiting for such a working set to page in and out. :) ---------------------------------------------------------------------- FUZxxl · 2022-03-26 09:42 Apparently the internal data structures use 32 bit indices and thus have this sort of limit. ---------------------------------------------------------------------- FUZxxl · 2022-03-26 09:43 It is very unlikely that you will be using this much swap. If the disk in question is fast, consider using most of the space as an L2ARC for your zpools instead. ---------------------------------------------------------------------- antiduh · 2022-03-26 14:12 If you're running a file server with 128 GB of ram and you're using any more than a trivial amount of swap, then something is very wrong. The memory demands of such a system should be allocated to the following: * Running samba or nfs or whatever for exporting the file systems. Neither of these should use significant amounta of ram, even for serving many connections. * Caching zfs data in ram. * Running the rest of the kernel. Are you sure it's operating and configured correctly? ---------------------------------------------------------------------- heathenskwerl · 2022-03-26 14:39 The system itself doesn't use much RAM, outside of ZFS. It's currently reporting 17GB in use for the ARC and 14MB in use for--well, pretty much everything else. This is after a fresh reboot. It is a dual CPU system (2x Xeon E5-2630s), so if I'm understanding the architecture correctly, it's essentially 64GB per CPU. There's some other oddities with memory performance with this architecture as well, in that it has higher performance with all the slots filled, even if it's not using all the RAM (and it came to me with 1/2 of the slots filled with 64GB total RAM). Under heavy load the ARC will climb to about 64GB and level off. I have always understood that to be the default maximum (1/2 system RAM). The rest of the system usually doesn't use more than 1GB that I've seen. That changes when I use the system to actually do memory-intensive operations. As mentioned above, one of the things this system is used for is creating solid 7zip archives of really large files. This uses quite a bit of RAM, and is usually when I see it swapping (if the file is large enough, 7z can consume upwards of 70GB+ RAM). Oddly the system performance doesn't seem to degrade appreciably when this is happening. The archiving slows down quite a bit, but completion is more important than speed for that. Because 7z isn't multithreaded when compressing solid archives, it may only consume a single CPU, which may be why the system is still responsive during that time. ---------------------------------------------------------------------- heathenskwerl · 2022-03-26 14:44 They're old WD 2TB gold drives. I don't know that they're appreciably faster than the drives in the zpools (which vary from 4TB 5900 RPM Seagates to 12TB WD white labels to 16TB Seagate EXOS drives). I was under the impression that L2ARC was only really useful if the drives in question were SSDs. Even if they were, I'm not convinced I'd get much benefit from an L2ARC. I'm limited to GigE, and the existing disks have zero trouble saturating that link. The only time I see them even close to fully utilized is when the system is doing something internal to itself (scrub, resilver, zfs send/recv). ---------------------------------------------------------------------- FUZxxl · 2022-03-26 14:48 You are right with your assessment. ---------------------------------------------------------------------- grahamperrin · 2022-03-26 15:45 > … the ARC will climb … and level off. I have always understood that to be the default maximum (1/2 system RAM). … From [zfs.4 — OpenZFS documentation](https://openzfs.github.io/openzfs- docs/man/4/zfs.4.html): >> … the larger of **all_system_memory - 1GB** and **5/8 × all_system_memory** … ---------------------------------------------------------------------- heathenskwerl · 2022-03-26 20:55 I'm on FreeBSD 12.3-RELEASE-p1--when first set up it was 12.2-RELEASE. I'm fairly certain it's still the old legacy ZFS. Edit: Though it looks like you are correct. sysctl is showing me 1GB less than system RAM. vfs.zfs.arc_max: 132820705280 Wonder why I never see it use more than 64GB? ---------------------------------------------------------------------- edthesmokebeard · 2022-03-27 03:08 What's your workload such that it requires this much swap? ---------------------------------------------------------------------- grahamperrin · 2022-03-27 18:07 there's an example, might help. ---------------------------------------------------------------------- heathenskwerl · 2022-03-27 18:41 The memory usage when archiving is actually kind of strange. The system swaps out a large amount of RAM while still having 70GB-100GB tied up with 7z. I can only assume it's swapping out part of 7z's working set, because the system remains responsive enough on the command line to use, and smb still serves up files with only minor performance degradation. 7z slows down quite a bit once it starts swapping, but it DOES complete, and it doesn't really matter to me if it takes 20-60 minutes to do so (it's usually compressing a bunch of files in a loop overnight anyway). ---------------------------------------------------------------------- heathenskwerl · 2022-03-27 18:45 Never used the entire 64GB of swap, but I've used well more than half (40-50GB). Could I get by on 64GB still? Almost certainly. Is it a waste of disk space? Highly likely, but I'm using 18.6GB of a mirrored pair of 2TB drives (1.67TB free), so if I don't use it for swap it's just not likely to get used. Will it hurt anything to have more? I'm not sure how, but if it *will* actually hurt something, by all means let me know. ---------------------------------------------------------------------- theamigan · 2022-03-27 19:11 Yes, the system is probably swapping 7z. You can check with `top -w` and ``procstat -v `pgrep 7z` `` which should show swap usage and details on each allocation. ---------------------------------------------------------------------- heathenskwerl · 2022-03-27 19:31 Helpful! I'll check that out next time, never knew about the -w switch for top. Thanks! ---------------------------------------------------------------------- antiduh · 2022-03-27 20:55 FYI, 7z can do multi threaded compression: -mt=on. You might need to tell it to use lzma2 to support that. Also, there's really no need to use the 7z options that use that much ram (ultra settings). The marginal benefit is absolutely tiny. ---------------------------------------------------------------------- edthesmokebeard · 2022-03-28 01:17 Im genuinely curious what memory intensive workloads you're running. ---------------------------------------------------------------------- grahamperrin · 2022-03-28 16:10 Did you see the linked example? ---------------------------------------------------------------------- heathenskwerl · 2022-03-28 18:27 I don't believe it can multithread when using solid compression, but I'm willing to be corrected. The marginal benefit absolutely is tiny for archives whose size is measured in MB. For the archives I'm compressing, it's not tiny. It's not huge either, but I'm only doing this once, so every little bit helps.