Post Ax0elyFWkJndC8z52O by zekjur@mas.to
 (DIR) More posts by zekjur@mas.to
 (DIR) Post #Ax0elyFWkJndC8z52O by zekjur@mas.to
       2025-08-09T15:17:25Z
       
       0 likes, 1 repeats
       
       I cleaned up and optimized the #Debian Code Search index merging code over the last week after noticing that it kept over 20 GB of memory (RSS) alive at peak. I didn’t even notice that on my old Hetzner server which had 128 GB of RAM, but these days, you only get 64 GB of RAM.The key insight was to tell the kernel that it doesn’t need to keep the mmap'ed file content mapped (in process), but rather return them to the page cache (in kernel):  https://github.com/Debian/dcs/commit/8c48c0464f92f4dc87cc7e759ca14671ab167e3d
       
 (DIR) Post #Ax0em47wtPIpQNU73g by zekjur@mas.to
       2025-08-09T15:19:53Z
       
       0 likes, 0 repeats
       
       For context, when you mmap a file on Linux, the contents are not actually counted towards your RSS, but when you read the file, they are! So, with a sequential access pattern, process RSS grows and approaches the total size of the working set (= all individual index files) over time. Linux can reclaim such pages when the system is under memory pressure, but at that point it’s already too late for my taste. With the madvise, RSS stays low throughout the merge :) Much happier server overall!