[HN Gopher] Client-side GPU load balancing with Redis and Lua
___________________________________________________________________
Client-side GPU load balancing with Redis and Lua
Author : lneiman
Score : 46 points
Date : 2025-12-02 17:39 UTC (6 days ago)
(HTM) web link (galileo.ai)
(TXT) w3m dump (galileo.ai)
| lneiman wrote:
| Author here. We were hitting tail latency and low GPU utilization
| issues serving SLMs via Triton.
|
| I built a scrappy client-side router using Redis and Lua to track
| real-time GPU load. It boosted utilization by ~40% and improved
| latencies.
|
| Happy to hear feedback on the implementation or thoughts on
| better ways to do this!
| pbrumm wrote:
| Have you tried switching it to a job queue where the GPU
| instances try to keep themselves busy. That way you can auto
| scale the gpus based on utilization. I find it easier to tune
| and you can monitor latency and backlogs easier. It does
| require some async mechanisms to the client but I have found it
| easier to maintain
| artyom wrote:
| If I understand the article correctly, any sufficiently capable
| attacker can:
|
| - Know the global state of your GPU cluster via the client.
|
| - Target the most struggling GPU instances specifically since the
| client decides which one to hit.
|
| You offer a free tier which means anyone can get an account and
| try to do it (e.g. you can have one "harmless, mostly inactive"
| free account with the only purpose of retrieving GPU cluster
| status, and a bunch of burner accounts to overload struggling
| instances).
|
| I may be completely wrong, but this sounds like DDoS served on a
| silver plate to me.
| singron wrote:
| They run these clients themselves and the redis instance isn't
| publically exposed.
|
| It would indeed be very strange to hope your random users
| coordinate with your client side load balancer. You wouldn't
| even have to send real traffic. You could just manipulate redis
| directly to force all the real traffic to go to a single node.
| DoSing redis itself is also pretty easy.
| artyom wrote:
| I don't think the article implied that the client was for
| some sort of internal server-to-server communication, or that
| the Redis instance was _directly_ exposed to the internet.
|
| So no, I don't think they run these clients themselves. If
| the code runs out there, it's open to inspection.
| tpurves wrote:
| Either way, you are right to point out that it important to
| only a try a pattern like this if your clients are highly
| trusted (or/and have additional compensating controls
| against DDOS threats). It would be beneficial if the OP
| made more explicit what their client/server relationships
| and also flagged the risk you mentioned for general
| audiences not to go implementing such a solution in the
| wrong places.
| PunchyHamster wrote:
| I'm gonna guess just switching from round-robin to leastconn
| (most balancers offer that option) would solve that just fine.
| You can then go to dynamically tune server weights if you have
| servers of unequal size or some other issues.
| gorkish wrote:
| Yeah I really don't understand why they went this direction as
| it builds considerable additional complexity directly into the
| application to solve a problem with an external component
|
| I would have probably approached this by implementing a fix for
| the misbehaving part of k8s, though since there _isnt_ a
| default LoadBalancer in k8s, I can 't really can't speculate
| further as to the root cause of the initial problem. But most
| CNI or cloud providers that implement LB do have a way to take
| feedback from an external metric. I'd be curious why doing it
| this way wasn't considered, at least.
| bnr4u wrote:
| Very cool work! Did you investigate using the Power of two random
| choices method for your load balancing algorithm ?
|
| https://brooker.co.za/blog/2012/01/17/two-random.html
| https://medium.com/the-intuition-project/load-balancing-the-...
___________________________________________________________________
(page generated 2025-12-08 23:01 UTC)