[HN Gopher] Show HN: Forkrun - NUMA-aware shell parallelizer (50...
___________________________________________________________________
Show HN: Forkrun - NUMA-aware shell parallelizer (50x-400x faster
than parallel)
forkrun is the culmination of a 10-year-long journey focused on
"how to make shell parallelization fast". What started as a
standard "fork jobs in a loop" has turned into a lock-free, CAS-
retry-loop-free, SIMD-accelerated, self-tuning, NUMA aware shell-
based stream parallelization engine that is (mostly) a drop-in
replacement for xargs -P and GNU parallel. On my 14-core/28-thread
i9-7940x, forkrun achieves: * 200,000+ batch dispatches/sec (vs
~500 for GNU Parallel) * ~95-99% CPU utilization across all 28
logical cores, even when the workload is non-existant (bash no-ops
/ `:`) (vs ~6% for GNU Parallel). These benchmarks are
intentionally worst-case (near-zero work per task) because they
measure the capability of the parallelization framework itself, not
how much work an external tool can do. * Typically 50x-400x faster
on real high-frequency low-latency workloads (vs GNU Parallel) A
few of the techniques that make this possible: * Born-local NUMA:
stdin is splice()'d into a shared memfd, then pages are placed on
the target NUMA node via set_mempolicy(MPOL_BIND) before any worker
touches them, making the memfd NUMA-spliced. Each numa node only
claims work that is _already_ born-local on its node. Stealing from
other nodes is permitted under some conditions when no local work
exists. * SIMD scanning: per-node indexers/scanners use AVX2/NEON
to find line boundaries (delimiters) at speeds approaching memory
bandwidth, and publish byte-offsets and line-counts into per-node
lock-free rings. * Lock-free claiming: workers claim batches with
a single atomic_fetch_add -- no locks, no CAS retry loops;
contention is reduced to a single atomic on one cache line. *
Memory management: a background thread uses fallocate(PUNCH_HOLE)
to reclaim space without breaking the logical offset system.
...and that's just the surface. The implementation uses many
additional systems-level techniques (phase-aware tail handling,
adaptive batching, early-flush detection, etc.) to eliminate
overhead, increase throughput and reduce latency at every stage.
In its fastest (-b) mode (fixed-size batches, minimal processing),
it can exceed 1B lines/sec. forkrun ships as a single bash file
with an embedded, self-extracting C extension -- no Perl, no
Python, no install, full native support for parallelizing arbitrary
shell functions. The binary is built in public GitHub Actions so
you can trace it back to CI (see the GitHub "Blame" on the line
containing the base64 embeddings). Trying it is literally two
commands: . frun.bash frun
shell_func_or_cmd < inputs For benchmarking scripts and
results, see the BENCHMARKS dir in the GitHub repo For an
architecture deep-dive, see the DOCS dir in the GitHub repo Happy
to answer questions.
Author : jkool702
Score : 91 points
Date : 2026-03-27 12:12 UTC (4 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| jkool702 wrote:
| Hi HN,
|
| Have you ever run GNU Parallel on a powerful machine just to find
| one core pegged at 100% while the rest sit mostly idle?
|
| I hit that wall...so I built forkrun.
|
| forkrun is a self-tuning, drop-in replacement for GNU Parallel
| (and xargs -P) designed for high-frequency, low-latency shell
| workloads on modern and NUMA hardware (e.g., log processing, text
| transforms, HPC data prep pipelines).
|
| On my 14-core/28-thread i9-7940x it achieves:
|
| - 200,000+ batch dispatches/sec (vs ~500 for GNU Parallel)
|
| - ~95-99% CPU utilization across all 28 logical cores (vs ~6% for
| GNU Parallel)
|
| - Typically 50x-400x faster on real high-frequency low-latency
| workloads (vs GNU Parallel)
|
| These benchmarks are intentionally worst-case (near-zero work per
| task), where dispatch overhead dominates. This is exactly the
| regime where GNU Parallel and similar tools struggle -- and where
| forkrun is designed to perform.
|
| A few of the techniques that make this possible:
|
| - Born-local NUMA: stdin is splice()'d into a shared memfd, then
| pages are placed on the target NUMA node via
| set_mempolicy(MPOL_BIND) before any worker touches them, making
| the memfd NUMA-spliced.
|
| - SIMD scanning: per-node indexers use AVX2/NEON to find line
| boundaries at memory bandwidth and publish byte-offsets and line-
| counts into per-node lock-free rings.
|
| - Lock-free claiming: workers claim batches with a single
| atomic_fetch_add -- no locks, no CAS retry loops; contention is
| reduced to a single atomic on one cache line.
|
| - Memory management: a background thread uses
| fallocate(PUNCH_HOLE) to reclaim space without breaking the
| logical offset system.
|
| ...and that's just the surface. The implementation uses many
| additional systems-level techniques (phase-aware tail handling,
| adaptive batching, early-flush detection, etc.) to eliminate
| overhead at every stage.
|
| In its fastest (-b) mode (fixed-size batches, minimal
| processing), it can exceed 1B lines/sec. In typical streaming
| workloads it's often 50x-400x faster than GNU Parallel.
|
| forkrun ships as a single bash file with an embedded, self-
| extracting C extension -- no Perl, no Python, no install, full
| native support for parallelizing arbitrary shell functions. The
| binary is built in public GitHub Actions so you can trace it back
| to CI (see the GitHub "Blame" on the line containing the base64
| embeddings).
|
| - Benchmarking scripts and raw results:
| https://github.com/jkool702/forkrun/blob/main/BENCHMARKS
|
| - Architecture deep-dive:
| https://github.com/jkool702/forkrun/blob/main/DOCS
|
| - Repo: https://github.com/jkool702/forkrun
|
| Trying it is literally two commands: .
| frun.bash # OR `. <(curl https://raw.githubusercontent.com/jk
| ool702/forkrun/main/frun.bash)` frun shell_func_or_cmd <
| inputs
|
| Happy to answer questions.
| esafak wrote:
| Please don't support only curl for installation. There are many
| package registries you can use; e.g.,
| https://github.com/aquaproj/aqua-registry
| jkool702 wrote:
| Theres no "install" - you just need to source the `frun.bash`
| file. Downloading frun.bash and sourcing it works just fine.
| directly sourcing a curl stream that grabs frun.bash from the
| git repo is just an alternate approach. It is not "required"
| by any means.
| brightmood wrote:
| Why the hell do you curl ? Additionally, why do you advertise
| it when you just had uploaded it? Nobody should install
| something that new...
| jkool702 wrote:
| curl isnt required - you just need to source the `frun.bash`
| file. Downloading frun.bash and sourcing it works just fine.
| directly sourcing a curl stream that grabs frun.bash from the
| github repo is just an alternate approach. It is not
| "required" by any means.
| DetroitThrow wrote:
| >Have you ever run GNU Parallel on a powerful machine just to
| find one core pegged at 100% while the rest sit mostly idle?
|
| Yes, to my extreme frustration. Thank you, I'm installing this
| right now while I read the rest of your comment.
| jkool702 wrote:
| How did it work for you?
| brightmood wrote:
| I am using a 9950x3D processor and didn't see any slow-down nor
| cpu sitting idle, I suggest you read the man-pages more clearly
| :P
| DetroitThrow wrote:
| I know this is obviously sarcasm and it made me laugh but I'm
| pretty sad HN couldn't catch it.
| brightmood wrote:
| No. I was being earnest! Works for me TM
| wood_spirit wrote:
| Thanks for making and thanks for sharing :)
|
| I'm not a parallels kind of user but I can appreciate your
| craft and know how rewarding these odysseys can be :)
|
| What was the biggest "aha" moment when you worked how things
| interlock or you needed to make both change an and b at the
| same time, as either on their own slowed it down? Etc. And what
| is the single biggest impacting design choice?
|
| And if you're objective, what could be done to other tools to
| make them competitive?
| jkool702 wrote:
| So, in forkruns development there have been a few "AHA!"
| moments. Most of them were accompanied by a full re-write
| (current forkrun is v3).
|
| The 1st AHA, and the basis for the original forkrun, was that
| you could eliminate a HUGE amount of the overhead of
| parallelizing things in shell in you use persistent workers
| and have them run things for you in a loop and distribute
| data to them. This is why the project is called "forkrun" -
| its short for "first you FORK, then you RUN".
|
| The 2nd AHA, which spawned forkrun v2, was that you could
| distribute work without a central coordinator thread (which
| inevitably becomes the bottleneck). forkrun v2 did this by
| having 1 process dump data into a tmpfile on a ramdisk, then
| all the workers read from this file using a shared file
| descriptor and a lightweight pipe-based lock: write a newline
| into a shared anonymous pipe, read from pipe to acquire lock,
| write newline back to pipe to release it. FIFO naturally
| queues up waiters. This version actually worked really well,
| but it was a "serial read, parallel execute" design.
| Furthermore, the time it took to acquire and release a lock
| meant the design topped out at ~7 million lines per second.
| Nothing would make it faster, since that was the locking
| overhead.
|
| The 3rd AHA was that I could make a very fast (SIMD-
| accellerated) delimiter scanner, post the byte offsets where
| lines (or batches of lines) started in the global data file,
| and then workers could claim batches and read data in
| parallel, making the design fully "parallel read + parallel
| execute"
|
| The 4th AHA was regarding NUMA. it was "instead of reactively
| re-shuffling data between nodes, just put it on the right
| node to begin with". Furthermore, determine the "right node"
| using real-time backpressure from the nodes with a 3 chunk
| buffer to ensure the nodes are always fed with data. This one
| didn't need a rewrite, but is why forkrun scales SO WELL with
| NUMA.
| nasretdinov wrote:
| Generally when I want to run something with so much parallelism I
| just write a small Go program instead, and let Go's runtime
| handle the scheduling. It works remarkably well and there's no
| execve() overhead too
| beanjuiceII wrote:
| dang and u did all that without a 10 year journey
| jkool702 wrote:
| 10 years represents going from
| maxJobs=$(nprocs) while read -r nn; do
| code_to_parallelize "$nn" (( $(jobs -p | wc -l) >
| maxJobs )) && wait -n done < inputs
|
| to a NUMA-Aware Contention-Free Dynamically-Auto-Tuning Bash-
| Native Streaming Parallelization Engine. I dare say 10 years
| is about the norm for going from "beginner" to "PhD-level"
| work.
| cbsmith wrote:
| AFAIK, the Go runtime is pretty NUMA-oblivious. The mcache
| helps a bit with locality of small allocations, but otherwise,
| you aren't going to get the same benefits (though I absolutely
| here you about avoiding execve overhead).
| jkool702 wrote:
| So...yes, the execve overhead is real. BUT there's still a
| lot you can accomplish with pure bash builtins (which don't
| have the execve overhead). And, if you're open to rewriting
| things (which would probably be required to some extent if
| you were to make something intended for shell to run in Go)
| you can port whatever you need to run into a bash builtin and
| bypass the execve overhead that way. In fact, doing that is
| EXACTLY what forkrun does, and is a big part of why it is so
| fast.
| jkool702 wrote:
| So, there are a few reasons why forkrun _might_ work better
| than this, depending on the situation:
|
| 1. if what you want to run is built to be called from a shell
| (including multi-step shell functions) and not Go. This is the
| main appeal of forkrun in my opinion - extreme performance
| without needing to rewrite anything. 2. if you are running on
| NUMA hardware. Forkrun deals with NUMA hardware remarkably well
| - it distributes work between nodes almost perfectly with
| almost 0 cross-node traffic.
| tombert wrote:
| I guess I've never really used parallel for anything that was
| bound by the dispatch speed of parallel itself. I've always use
| parallel for running stuff like ffmpeg in a folder of 200+
| videos, and the speed in which parallel decides to queue up the
| jobs is going to be very thoroughly eaten by the cost of ffmpeg
| itself.
|
| Still, worth a shot.
|
| I have to ask, was this vibe-coded though? I ask because I see
| multiple em dashes in your description here, and a lot of no X,
| no Y... notation that Codex seems to be fond of.
|
| ETA: Not vibe coded, I see stuff from four years ago...my
| mistake!
| jkool702 wrote:
| > I ask because I see multiple em dashes in your description
| here, and a lot of no X, no Y... notation that Codex seems to
| be fond of.
|
| I asked a few LLM's for tips on writing the HN post. The post
| is my own words, but their style may have rubbed off on me a
| little bit. I'm admittedly better at the technical aspects than
| I am at "writing good catchy posts that dont turn into 20 pages
| of technical writing", so...
| HackerThemAll wrote:
| I like it, and I hope it's soon going to be available in various
| Linux distributions, along with other modern tools such as fd
| instead of find, ripgrep instead of grep, and fzf, for instance.
___________________________________________________________________
(page generated 2026-03-31 23:00 UTC)