Post ASzT1lbiYGJjymGeRM by mikemol@pony.social
(DIR) More posts by mikemol@pony.social
(DIR) Post #ASzQOeFNmcU3wx2W5g by niconiconi@mk.absturztau.be
2023-02-24T01:54:43.768Z
2 likes, 0 repeats
In the days of spinning rust, conventional wisdom says it's a good idea to set the number of build jobs equal to the CPU threads plus 1. The plus 1 is here to keep the HDD busy to hide I/O latency from the CPU. I wonder if it's ever proven by anyone with real benchmarks...
(DIR) Post #ASzRGitKjQ8lzXVkjg by lanodan@queer.hacktivis.me
2023-02-24T02:01:31.033641Z
1 likes, 0 repeats
@niconiconi I still follow the +1 rule except when fucking C++ happens.
(DIR) Post #ASzRQ0de7NqDdgbMFk by niconiconi@mk.absturztau.be
2023-02-24T02:06:11.794Z
0 likes, 0 repeats
@lanodan@queer.hacktivis.me I stopped following it after 4C/8T hyperthread became the norm. It feels like the hyperthreads are enough to keep the disk busy enough.
(DIR) Post #ASzRiN2pYEAMLL8XVA by lanodan@queer.hacktivis.me
2023-02-24T02:09:08.554677Z
1 likes, 0 repeats
@niconiconi Well for my 16 core machine I did try +2 instead but honestly… not sure how I could do a significant benchmark without recompiling a shitton of things.
(DIR) Post #ASzSmFlie2iKtDY3TE by niconiconi@mk.absturztau.be
2023-02-24T02:21:26.134Z
0 likes, 0 repeats
@lanodan@queer.hacktivis.me And it would be quite tedious to create an accurate benchmark. The low-hanging fruit to criticize is the difference in disk fragmentation after each build. So you need to nuke and recreate the filesystem in each run... (And now most things are on SSDs so it's not like anyone cares about HDD code compile performance anymore.) I have to respect all the PC hardware reviewers, it's not challenging in terms of raw difficulties, but it's still honest hard work...
(DIR) Post #ASzT1lbiYGJjymGeRM by mikemol@pony.social
2023-02-24T02:23:26Z
0 likes, 1 repeats
@niconiconi I've played with it. As with all things, it depends.Most importantly, it depends on how much you're building; if you're doing daily updates, you're only rarely building enough for it to be measurable. If you're doing a `-e` build, it could make a difference. But, really, not reliably; builds can't always be parallelized.Personally, I target a load average of $cpu, but with an upper limit of less than 2x $cpu. I used to use tmpfs for builds, but beyond a certain amount of RAM, it's not as fast as NVMe or a quick raid array; the data structures that make filesystems fast can help them outperform tmpfs.What I'd *like* is something that hooks into both emerge and make and blocks spawns until conditions are right. (Those being, for me, tasks in an R state being <= $CPU, and I/O utilization being below 90%.) Maybe it's doable with LD_PRELOAD?
(DIR) Post #ASzTC1bLwNvSTozLAe by niconiconi@mk.absturztau.be
2023-02-24T02:26:04.281Z
0 likes, 0 repeats
@mikemol@pony.social Yep. Both ./configure and the final linking are huge "serialization" bottlenecks. For a small codebase it can easily mask the small gain of compile speed... Supporting multiple instances of emerge in Gentoo has definitely been a good idea in this aspect.
(DIR) Post #ASzTEeH77laX0Flykq by lanodan@queer.hacktivis.me
2023-02-24T02:23:20.781783Z
1 likes, 0 repeats
@niconiconi I think a lot of hardware reviewers just let it run a bunch of times, which is why sometimes you can see them talking about lack of cooling.
(DIR) Post #ASzTav0Bc9c7FfTSbI by niconiconi@mk.absturztau.be
2023-02-24T02:30:34.592Z
0 likes, 0 repeats
@lanodan@queer.hacktivis.me The Phoronix Test Suite even has a feature to decide if the number of test runs is enough to be "statistical significant"
(DIR) Post #ASzThBGE4szFPlxkTA by lanodan@queer.hacktivis.me
2023-02-24T02:31:19.350331Z
1 likes, 0 repeats
@niconiconi Well https://metacpan.org/release/Dumbbench does that too :)
(DIR) Post #ASzTnNDgWl7WF5j8oy by mikemol@pony.social
2023-02-24T02:31:53Z
1 likes, 0 repeats
@niconiconi I put my flags in both MAKEOPTS and EMERGE_DEFAULT_OPTS...but I really do hate that the two mechanisms don't coordinate. And then there's ninja. _Really_ thinking about that LD_PRELOAD approach, now. Maybe if I hook into fork() and exec(). (Or whatever the variations work out to under the hood).
(DIR) Post #ASzU1IwtCf3j3CVvay by niconiconi@mk.absturztau.be
2023-02-24T02:35:22.071Z
0 likes, 0 repeats
@mikemol@pony.social At the end of the day, Amdahl's law is no joke...
(DIR) Post #ASzUPBr5EFOo04Bhey by lanodan@queer.hacktivis.me
2023-02-24T02:39:09.722303Z
0 likes, 0 repeats
@mikemol @niconiconi Well ninja can limit itself via load average by default, -l option.And I think GNU Make should add that option.And why would you go with LD_PRELOAD when /etc/portage/patches exists and wouldn’t have side-effects everywhere? (specially as gentoo’s sandbox is also LD_PRELOAD based)
(DIR) Post #ASzWP5y7JZXFeSYn9U by mikemol@pony.social
2023-02-24T02:42:12Z
1 likes, 0 repeats
@niconiconi next, you'll have me fighting the halting problem, wondering if the path I'm pausing means I'm stranding CPU or I/O.(Actually, it probably doesn't matter if I'm oversubscribed on I/O, as long as the CPU stays hot, but isn't swapping or losing too much time to context switching. So exciting!)
(DIR) Post #ASzWdGzx6UI1uhBAY4 by qasar@snowdin.town
2023-02-24T03:04:00.943090Z
1 likes, 0 repeats
@lanodan @niconiconi @mikemol why would you /etc/portage/patches for that though? you could just put things in post_pkg_configure(){ } via /etc/portage/env and hook something in it instead of patching a package, i think it'd be better that way imo
(DIR) Post #ASzWo351BxFvvqgeBc by lanodan@queer.hacktivis.me
2023-02-24T03:06:07.818943Z
0 likes, 0 repeats
@qasar @niconiconi @mikemol What would you even put in there?
(DIR) Post #ASzXXBpBWghw5Ho1bc by a1ba@suya.place
2023-02-24T03:13:59.683942Z
0 likes, 0 repeats
@lanodan @niconiconi I even do -1 today to let CPU do other stuff while it's compiling.1 SMT core won't do much for compiling but at least give some time for other tasks.
(DIR) Post #ASzY0DFHK3VwtAqlXc by lanodan@queer.hacktivis.me
2023-02-24T03:19:30.204801Z
0 likes, 0 repeats
@a1ba @niconiconi I've yet to have compiling significantly impact other things.And I think I'd rather try to tweak scheduling/nice/… levels in make.conf if that would happen.
(DIR) Post #ASzYFy2HdgdULnHgQa by a1ba@suya.place
2023-02-24T03:22:50.079838Z
0 likes, 0 repeats
@lanodan @niconiconi I had weird issue with CPU scheduler on Ryzen on early kernels when parallel compiling could stop pulse or pipewire, regardless of their niceness values.With newer kernels it's much better but still happens rarely.
(DIR) Post #ASzZEZX1MMZC15YmRM by koakuma@uwu.social
2023-02-24T03:31:42Z
0 likes, 0 repeats
@lanodan @niconiconi @mikemol GNU make seems to have -l too:> You can use the `-l` option to tell make to limit the number of jobs to run at once, based on the load average. https://www.gnu.org/software/make/manual/html_node/Parallel.html
(DIR) Post #ASzZEf7iZLtLMLlepk by lanodan@queer.hacktivis.me
2023-02-24T03:33:19.024415Z
0 likes, 0 repeats
@koakuma @niconiconi @mikemol Oh, that's interesting, somehow I skipped it in the manual.(And it's been there for a while, like was already there in 2012)
(DIR) Post #ASzZpFI5ZrnFxI3xsO by koakuma@uwu.social
2023-02-24T03:33:10Z
1 likes, 0 repeats
@niconiconi @mikemol > Supporting multiple instances of emerge in Gentoo has definitely been a good idea in this aspect.I honestly still struggle to get emerge to build multiple packages at once even when passing -j to it; there's usually That One Package that is depended by a lot of others and then bam! package parallelization is back to 1 again...
(DIR) Post #ASzZpFqpUhaBh2Ji1g by mikemol@pony.social
2023-02-24T03:37:32Z
1 likes, 0 repeats
@koakuma @niconiconi yup. That's honestly why I use --load-avg=; it's the only way to get emerge and make to back off when the other is driving load. But it's not perfect, and needs -j as a cap, or you'll get an oscillating explosion of concurrency, and either slow everything down on inefficient I/O, have so many tasks cycle through each core that you spend a ton of time in sys CPU, or you straight up OOM when you get a couple notable linkers running at the same time. (Actually, that last is still a risk. If I could limit linker concurrency system-wide to just 1, that'd be great...)
(DIR) Post #ASzjQWL23cdp84HwwK by acsawdey@fosstodon.org
2023-02-24T04:56:56Z
1 likes, 0 repeats
@niconiconi On servers with large numbers of processors, you can run into trouble because modern compilers use lots of memory, so N+1 but not so many as to cause paging …