[HN Gopher] Bpftune uses BPF to auto-tune Linux systems
___________________________________________________________________
Bpftune uses BPF to auto-tune Linux systems
Author : BSDobelix
Score : 186 points
Date : 2024-11-17 11:38 UTC (11 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| bloopernova wrote:
| Fascinating!
|
| I'd like to hear from people who are running this. Is it
| effective? Worth the setup time?
| usr1106 wrote:
| Interesting. But if tuning parameters to their best values were
| easy, shouldn't the kernel just do that in the first place?
| nitinreddy88 wrote:
| It depends on workload. This tool generates recommended config
| for that specific machine workload. App Nodes can have
| completely different recommendations vs Database Nodes. It will
| be completely different for Workstation.
| usr1106 wrote:
| Sure, but the kernel could just do the same. Of course the
| kernel is already too big. Is BPF the right level to make it
| more modular? Just thinking, I don't think I have the answer.
| onetoo wrote:
| This doesn't necessarily find the _best_ parameters, and it
| doesn 't necessarily do it _easily_. From my reading, it will
| converge on a local optimum, and it may take some time to do
| that.
|
| In theory, I don't see why the kernel couldn't have a
| parameter-auto-tune similar to this. In practice, I think the
| kernel has to work in so many different domains, it'd be
| impossible to land on a "globally good enough" set of tuning
| heuristics.
|
| I'm far from a kernel developer, so I'm ready to be corrected
| here.
|
| IMO if we ever see something like this deployed widely, it will
| be because a popular distribution decided to install it by
| default.
| RandomThoughts3 wrote:
| I would reverse the question: if it can be done by a BPF
| module, why should it be in the kernel?
|
| Distributions turning it on by default is another story. Maybe
| it deserves to be shiped on all the time but that's not the
| same thing as being part of the kernel.
| jiehong wrote:
| Indeed!
|
| The kernel might already be too monolithic.
|
| This kernel parameters optimisation reminds me of PGO
| compilation in programs.
|
| Yet, perhaps the kernel could come with multiple defaults
| config files, each being a good base for different workloads:
| server, embedded, laptop, mobile, database, router, etc.
| 6SixTy wrote:
| I second the different profiles for server, laptop, and so
| on. Though I know the kernel already comes with default
| configs, so I think there could be room for specialized
| kernel config options in addition to what's already there.
|
| Though in my opinion, there's already kind of too much
| segmentation between the different use cases. A server is
| just a role for a computer, and embedded could literally
| mean anything. Quite a few WiFi access points come with a
| USB port on them so you can plug in a USB drive and start a
| SMB server.
| sgarland wrote:
| I'd rather the kernel present a good-enough but extremely
| stable set of configs. If I'm using a distro like Arch or
| Gentoo, then sure, maybe run wild (though both of those would
| probably assume I'm tuning them anyway), but CentOS, Debian, et
| al.? Stable and boring. If you change something, you'd better
| know what it is, and why you're doing it.
| LinuxBender wrote:
| HP-UX and to a lesser degree AIX had some of this. Instead of
| just static tunable values there were some tunable formulas
| exposed to the admin. If one knew what they were doing they
| could change the formula as needed without having to recompile
| the kernel. I would not mind something like this being exposed
| from the Linux kernel as tunable options as opposed to just
| static values. Such options could reduce the number of
| companies that need to recompile the kernel.
| robinhoodexe wrote:
| Is tuning the TCP buffer size for instance worth it?
| viraptor wrote:
| It depends. At home - probably not. On a fleet of 2000 machines
| where you want to keep network utilisation close to 100% with
| maximal throughput, and the non-optional settings translate to
| a not-trivial value in $ - yes.
| londons_explore wrote:
| TCP parameters are a classic example of where an autotuner
| might bite you in the ass...
|
| Imagine your tuner keeps making the congestion control more
| aggressive, filling network links up to 99.99% to get more
| data through...
|
| But then any other users of the network see super high
| latency and packet loss and fail because the tuner isn't
| aware of anything it isn't specifically measuring - and it's
| just been told to make this one application run as fast as
| possible.
| withinboredom wrote:
| It literally covers this exact scenario in the readme and
| explains how it prevents that.
| crest wrote:
| It depends mostly on the bandwidth-delay-product and packet
| loss you expect on each connection. A there is a vast
| difference between a local interactive SSH session and
| downloading a large VM image from across an ocean.
| zymhan wrote:
| There's not much cost to doing it, so yes.
| toast0 wrote:
| In my experience running big servers, tuning TCP buffers is
| definitely worth it, because different kinds of servers have
| different needs. It doesn't often work miracles, but tuning
| buffers is low cost, so the potential for a small positive
| impact is often worth the time to try.
|
| If your servers communicate at high datarates with a handful of
| other servers, some of which are far away, but all of which
| have large potential throughput, you want big buffers. Big
| buffers allow you to have a large amount of data in flight to
| remote systems, which lets you maintain throughput regardless
| of where your servers are. You'd know to look at making buffers
| bigger if your throughput to far away servers is poor.
|
| If you're providing large numbers of large downloads to public
| clients that are worldwide from servers in the US only, you
| probably want smaller buffers. Larger buffers would help with
| throughput to far away clients, but slow, far away clients will
| use a lot of buffer space and limit your concurrency. Clients
| that disappear mid download will tie up buffers until the
| connection is torn down and it's nice if that's less memory for
| each instance. You'd know to look at making buffers smaller if
| you're using more memory than you think is appropriate for
| network buffers... a prereq is monitoring memory use by type.
|
| If you're serving dynamic web pages, you want your tcp buffers
| to be at least as big as your largest page, so that your
| dynamic generation never has to block for a slow client. You'd
| know to look at this if you see a lot of servers blocked on
| sending to clients, and/or if you see divergent server measured
| response times for things that should be consistent. This is
| one case where getting buffer sizes right can enable miracles;
| Apache pre-fork+mod_PHP can scale amazingly well or amazingly
| poorly; it scales well when you can use an accept filter so
| apache doesn't get a socket until the request is ready to be
| read, and PHP/apache can send the whole response to the tcp
| buffer without waiting, then closes the socket; letting the
| kernel deal with it from there. Keep-alive and TLS make this a
| bit harder, but the basic idea of having enough room to buffer
| the whole page still fits.
| BSDobelix wrote:
| BTW one can use it out of the box with CachyOS.
|
| After installation -> CachyOS Hello -> Apps/Tweaks
| bastloing wrote:
| It's great how it grew out of simple packet filtering into
| tracing and monitoring. It's one of those great tools most should
| know. Been using it for years.
| orbital-decay wrote:
| The entire modern ML and most of the HPC grew out of very
| simple programmable shaders in GeForce 3.
| gmuslera wrote:
| Two words: "feedback loop".
|
| That was the first idea that jumped in when thinking in what
| could go wrong, not because the Linux kernel, or BPF or this
| program, just for how it is intended to work. There might be no
| risk of that happening, there may be controls around that, or if
| they happen they might only converge to an estable state, but
| still it is something to have in the map.
| marcosdumay wrote:
| > or if they happen they might only converge to an stable state
|
| That one will always be dependent on the usage patterns. So the
| auto-tuner can't guarantee it.
|
| Also, I imagine the risk of the feedback turning positive is
| related to the system load (not CPU, but the usage of the
| resources you are optimizing). If so, it will make your
| computer less able to manage load. But this can still be useful
| for optimizing for latency.
| nevon wrote:
| I wonder how effective this would be in multi-tenant environments
| like shared k8s clusters. On the one hand, each application
| running will have a different purpose and will move around
| between nodes over time, but on the other hand there are likely
| broad similarities between most applications.
| mrbluecoat wrote:
| > bpftune is designed to be zero configuration; there are no
| options
|
| On behalf of every junior administrator, overworked IT admin, and
| security-concerned "cattle" wrangler, thank you.
|
| Having to learn a thousand+ knobs & dials means most will never
| be touched. I for one welcome automated assistance in this area,
| even if the results are imperfect.
| sgarland wrote:
| I think it's still important to know what those dials and knobs
| do, otherwise (as the currently top-voted comment says) when
| things break, you'll be lost.
| zymhan wrote:
| That's exactly why it's such a burden.
| sgarland wrote:
| I continue to be amazed and frustrated that people will
| simultaneously believe that deeply understanding a
| programming language is a noble pursuit, but that deeply
| understanding the software that allows their code to run is
| somehow burdensome.
|
| Ops remains extremely important and extremely real. You can
| abstract it away in exchange for higher costs and less
| control, but ultimately someone at some level has studied
| these parameters, and decided what is best for your code.
| gausswho wrote:
| With this tool I am wary that I'll encounter system issues that
| are dramatically more difficult to diagnose and troubleshoot
| because I'll have drifted from a standard distro configuration.
| And in ways I'm unaware of. Is this a reasonable hesitation?
| pbhjpbhj wrote:
| >"bpftune logs to syslog so /var/log/messages will contain
| details of any tuning carried out." (from OP GitHub readme)
|
| The rmem example seems to allay fears that it will make changes
| one can't reverse.
| admax88qqq wrote:
| It's not a questions of being able to reverse. It's a
| question of being able to diagnose that one of these changes
| even was the problem and if so which one.
| nehal3m wrote:
| If they can be reversed individually you can simply deduce
| by rolling back changes one by one, no?
| jstanley wrote:
| Only if you already suspect that this tool caused the
| problem.
| spenczar5 wrote:
| Suppose you run a fleet of a thousand machines. They all
| autotune. They are, lets say, serving cached video, or
| something.
|
| You notice that your aggregate error rate been drifting
| upwards since using bpftune. It turns out, in reality,
| there is some complex interaction between the tuning and
| your routers, or your TOR switches, or whatever - there
| is feedback that causes oscillations in a tuned value,
| swinging between too high and too low.
|
| Can you see how this is not a matter of simple deduction
| and rollbacks?
|
| This scenario is plausible. Autotuning generally has
| issues with feedback, since the _overall_ system lacks
| control theoretic structure. And the premise here is that
| you use this to tune a large number of machines where
| individual admin is infeasible.
| pbhjpbhj wrote:
| >not only can we observe the system and tune
| appropriately, we can also observe the effect of that
| tuning and re-tune if necessary. //
|
| Does sound like a potential way to implement literal
| chaos.
|
| Surely it's like anything else, you do pre-release
| testing and balance the benefits for you against the
| risks?
| Modified3019 wrote:
| Sounds like you have your answer of "don't use it" then.
| pstuart wrote:
| In that scenario you could run it on a couple servers,
| compare and contrast, and then apply globally via
| whatever management tool you use.
| KennyBlanken wrote:
| Presumably one would use autotune to find optimized
| parameters, and then roll those out via change control,
| either one parameter at a time, or a mix of parameters
| across the systems.
|
| Alternatively: if you have a fleet of thousands of
| machines you can very easily do a binary search with them
| to a)establish the problem with the auto-tuner and then
| b)which of the changes it settled on are causing your
| problems.
|
| I get the impression you've never actually managed a
| "fleet" of systems, because these techniques would have
| immediately occurred to you.
| spenczar5 wrote:
| Certainly when we managed Twitch's ~10,000 boxes of video
| servers, neither of the tasks you describe would have
| been simple. We underinvested in tools, for sure. Even
| so, I don't think you can really argue that dynamically
| changing configs like this are going to make life easier!
| toast0 wrote:
| When you have a thousand machines, you can usually get
| feedback pretty quick, in my experience.
|
| Run the tune on one machine. Looks good? Put it on ten.
| Looks good? Put it on one hundred. Looks good? Put it on
| everyone.
|
| Find an issue a week later, and want to dig into it? Run
| 100 machines back on the old tune, and 100 machines with
| half the difference. See what happens.
| pbhjpbhj wrote:
| I focused primarily on guesswho's "in ways I am unaware
| of".
|
| Your issue appears to be true for any system change.
| Although, risk will of course vary.
| sgarland wrote:
| Yes, it is. IMO, except for learning (which should not be done
| in prod), you shouldn't make changes that you don't understand.
|
| The tools seems to mostly tweak various networking settings.
| You could set up a test instance with monitoring, throw load at
| it, and change the parameters the tool modifies (one at a
| time!) to see how it reacts.
| nine_k wrote:
| I'd run such a tool on prod in "advice mode". It should
| _suggest_ the tweaks, explaining the reasoning behind them,
| and listing the actions necessary to implement them.
|
| Then humans would decide if they want to implement that as
| is, partly, modified, or not at all.
| sgarland wrote:
| Fair point, though I didn't see any such option with this
| tool.
| nine_k wrote:
| It's developed in the open; we can create Github issue.
|
| Actually https://github.com/oracle/bpftune/issues/99
| westurner wrote:
| In the existing issue, we can link to the code and docs
| that would need to be understood and changed:
|
| usage, main() https://github.com/oracle/bpftune/blob/6a50
| f5ff619caeea6f04d...
|
| - [ ] CLI opts: --pretend-allow <tuner> or --log-only-
| allow <tuner> or [...]
|
| Probably relevant function headers in libbpftune.c:
|
| bpftune_sysctl_write(
|
| bpftuner_tunable_sysctl_write(
|
| bpftune_module_load(
|
| static void bpftuner_scenario_log(struct bpftuner *tuner,
| unsigned int tunable, ; https://github.com/oracle/bpftune
| /blob/6a50f5ff619caeea6f04d... https://github.com/oracle/
| bpftune/blob/6a50f5ff619caeea6f04d...
| bink wrote:
| I agree. There's very little need to have a tool
| automatically changing complex kernel settings on the fly
| unless your infrastructure is undergoing dramatic changes
| in load and behavior on a daily basis, which seems unlikely
| for a modern server.
| trelliscoded wrote:
| If your staging doesn't do capacity checks in excess of what
| production sees, yes.
| Twirrim wrote:
| Disclaimer: I work for Oracle, who publish this tool, though I
| have nothing to do with the org or engineers that created it
|
| I've been running this for a while on my laptop. So far yet to
| see any particular weirdness, but also I don't know that I can
| state with any confidence it has a positive impact either. I've
| not carried out any benchmarks in either direction.
|
| It logs all changes that it's going to make including what they
| were on before. Here's an example from my logs:
| bpftune[1852994]: Scenario 'need to increase TCP buffer
| size(s)' occurred for tunable 'net.ipv4.tcp_rmem' in global ns.
| Need to increase buffer size(s) to maximize throughput
| bpftune[1852994]: Due to need to increase max buffer size to
| maximize throughput change net.ipv4.tcp_rmem(min default max)
| from (4096 131072 7864320) -> (4096 131072 9830400)
| klysm wrote:
| This seems to step into control theory, which I think is somewhat
| underapplied in software engineering.
| 6SixTy wrote:
| As an insane Gentoo user, sign me the F*** up
___________________________________________________________________
(page generated 2024-11-17 23:00 UTC)