[HN Gopher] HAProxy: How to temporary disable a back end server ...
___________________________________________________________________
HAProxy: How to temporary disable a back end server using the
command line
Author : Napsty
Score : 98 points
Date : 2022-09-13 08:03 UTC (14 hours ago)
(HTM) web link (www.claudiokuenzler.com)
(TXT) w3m dump (www.claudiokuenzler.com)
| [deleted]
| vinay_ys wrote:
| Haproxy is definitely under-appreciated for the amount of heavy
| lifting it does in a high-scale deployment.
| NDizzle wrote:
| A lot of just don't realize how and where it's used. I'm almost
| 100% positive that Microsoft uses HAProxy (open source or
| commercial license I couldn't tell you) to power App Services
| on Azure. Yet you'll get friction from CTOs at "Microsoft
| shops" for choosing HAProxy to use yourself.
| jabart wrote:
| HAProxy also has the data plane api, which is another process
| that exposes an API that lets you do the same thing but remote
| and over a rest api.
| manishsharan wrote:
| Dataplane API is pretty sweet :
|
| https://www.haproxy.com/documentation/dataplaneapi/community...
| ElevenLathe wrote:
| Doesn't "Dataplane API" actually constitute part of the
| control plane?
| lukeqsee wrote:
| We built a fantastic CDN on top of HAProxy, and all these nifty
| tricks are only scratching the surface of its potential.
|
| The ease with which you can make real-time reactions to threats
| and conditions is really second to none, and with the ability to
| truly "hitless" reload (reload without affecting in-flight
| requests and connections) it makes those have immediate impact
| and no customer impact.
|
| Sometimes the capabilities make the config a bit inscrutable, but
| other than that, I can't recommend it enough.
| xani_ wrote:
| Newer versions also have few more tricks up the sleeve, like
| you can probe and act upon client's TCP RTT and RTT variance
| (because apparently newer kernels just track that info and it
| can be accessed from userspace).
|
| We ended up using HAPRoxy as front for pretty much everything
| (even if _techncially_ slapping just Nginx would be simpler at
| first), just because it often happens that this and that needs
| to be added (stuff like adding proper headers when devs don 't
| want to figure out how to make their '00s CMS emi the '20s
| security headers).
|
| It can also talk directly via unix socket to the backends so
| you don't even need to have a bunch of ports open if you want
| to front something.
| pull_my_finger wrote:
| Have you used the ngx_lua module (or OpenResty) with Nginx at
| all? I'm curious if there was something HAProxy does that
| can't be done with OpenResty, because that is crazy full-
| featured. Basically the main reason I haven't played around
| with HAProxy is just because I haven't ever felt like I was
| missing anything with OpenResty/Nginx
| lukeqsee wrote:
| HAProxy also has Lua support.
|
| The general answer here, is no, I don't think you can do
| much in HAProxy that you can't do with OpenResty, but with
| HAProxy you generally don't have to pay the overhead of Lua
| (and HAProxy tends to be very well optimized, so the
| difference can be significant at high throughput of
| requests / bytes).
| smw wrote:
| One notable thing is that OpenResty, unlike HAProxy, uses
| LuaJIT, which is dramatically faster.
| bigoldie wrote:
| Or use HATop :-)
| mtmail wrote:
| Last updated 2010. I'm all for software that doesn't need new
| features but no changed in a decade make me not want to install
| it on a production server http://feurix.org/projects/hatop/
| crizzlenizzle wrote:
| Correct link: https://www.claudiokuenzler.com/blog/1240/haproxy-
| how-to-dis...
| Napsty wrote:
| thx mate! just added the correct link as a new submission, too.
| dang wrote:
| Fixed now. Thanks!
| bobek wrote:
| HAproxy is awesome. We have used the socket for ingress rate
| limiter for reasonably successful VoD service.
| elforce002 wrote:
| Wow. Nice. I'm working on a similar project atm. Do you have
| any info on how to configure the socket?
| mihaigalos wrote:
| I prefer commenting out the entry referencing the server/node
| directly in the haproxy.cfg or even better, the code used to
| generate it und version control (Config-as-Code).
| linsomniac wrote:
| Seems a little heavy weight for my "remove from LB, update
| code, add back to LB" Ansible script. :-)
| hayst4ck wrote:
| An alternative I like a bit more than, `remove, update, add
| back` is to set a signal handler to close the listening
| socket, wait for current requests to finish, and then exit.
|
| Assuming you initialize every service you depend on at
| program start and not in the critical path and that health
| checks only function after this setup work has been done it
| simplifies operations a bit and decreases the liklhihood of
| configuration errors (a drained server in a load balancer)
| from persisting.
| Napsty wrote:
| Yes but a reload can still cut a KEEP ALIVE established
| connection (observed in older HAProxy 1.6). Probably solved in
| more recent versions though.
| endre wrote:
| also, old haproxy process (with opened connections) still
| does checks which might be undesirable at scale.
|
| as such we try to avoid reloading haproxy as much as
| possible. you can even renew a cert on-the-fly uploading the
| new cert via the admin socket.
| bsagdiyev wrote:
| This may be resolved in newer versions but definitely bit me
| before in older ones. I just have a script that drains off
| the backends we need before putting them in maintenance mode.
| Makes working on individual hosts much easier.
| Hackerpronoov wrote:
| philliphaydon wrote:
| There's other ways than using telnet??? I thought telnet was the
| recommended way.
| xani_ wrote:
| you can also add stats admin
|
| to the stats backend to enable ability to do that from statistics
| UI address. Just make sure it is at the very least behind a
| password
| simonjgreen wrote:
| hatop is a great ncurses interface to this from cli.
| https://github.com/feurix/hatop
|
| Can also be done remotely https://github.com/Wirehive/haproxy-
| remote
| tomputer wrote:
| Indeed! HATop is great to show the status, traffic, HTTP codes,
| errors or number of connections. It is also very simple to
| enable/disable HAProxy backends, for example:
| hatop -s /var/run/haproxy/example.sock
|
| Use the arrows to select a backend server and press:
| F9 - Enable a backend server (Status: UP) F10 - Disable a
| backend server (Status: MAINTENANCE)
|
| For Apple keyboards it is fn+F9 and fn+F10.
| darkwater wrote:
| The irony:
|
| ```
|
| 403 Forbidden
|
| nginx
|
| ```
| taf2 wrote:
| IMO not really - a valid architecture is nginx - haproxy - app
| servers
| darkwater wrote:
| IME haproxy and nginx can overlap a lot as
| balancers/rewriter; having both of them adds no real world
| benefits, unless you have some special case where you need
| one special, unique feature from both of them.
| hayst4ck wrote:
| > having both of them adds no real world benefits
|
| This depends a great deal on the architecture of your
| application server and traffic load. There are a lot of
| reasons that it is nicer to use a queue in front of your
| load balancer, rather than use queues in your application
| server, or worse, use the operating system queue.
|
| You must have SSL termination, logging, load balancing, and
| queuing. You probably want to modify headers and interpret
| cookies as well.
|
| HAProxy is a queue that feeds into a load balancer. Nginx
| is a buffer that feeds into a load balancer (IIRC).
|
| One useful property that HAProxy can ensure is no more than
| 1 request going to a server at a time. I believe (but
| potentially incorrectly) that nginx will fail requests,
| rather than "queue" them, if it tries to ensure this
| property. HAProxy also historically had health checks for
| backend servers, while for nginx it was a paid premium
| feature.
|
| I could be wrong about nginx not being able to queue. It
| has been a while.
|
| For my mental model, nginx is the right choice for SSL
| termination, logging, request mangling, interpretation of
| cookies and loadbalancing based on request information (for
| example choosing haproxy instances based on a domain name).
| HAProxy is the right tool for queuing and load balancing to
| servers that will actually fulfill the request.
| TimWolla wrote:
| Disclosure: Community contributor to HAProxy, I help
| maintain HAProxy's issue tracker.
|
| > For my mental model, nginx is the right choice for SSL
| termination, logging, request mangling, interpretation of
| cookies and loadbalancing based on request information
| (for example choosing haproxy instances based on a domain
| name).
|
| HAProxy can do all that and IMO it also does it better.
|
| Personally I chain HAProxy and nginx in reverse order:
| HAProxy exposed to the Internet, doing all the heavy
| lifting. (Multiple) nginx with minimal config as a static
| file server and FastCGI gateway behind HAProxy.
|
| see also: https://news.ycombinator.com/item?id=27253579
| gregmac wrote:
| I've done a similar architecture as well a couple of
| times. Both setups use multiple haproxy instances (auto-
| scaling group), multiple backend app servers, and each
| haproxy server has a local nginx instance.
|
| Nginx is used for serving some static error pages that
| live outside of the backend applications haproxy talks to
| (eg: "Customer domain not recognized"). There's some
| hacks to kind of make this work in haproxy but it's much
| simpler with an actual HTTP content server.
|
| Nginx also works well as a caching proxy, and what's
| really cool with haproxy is it's easy to have most
| requests go directly to the application server (avoiding
| an unnecessary proxy hop), but just pattern match certain
| types -- eg regex match `.jpg$` -- to go through the
| nginx cache.
|
| I've also used a similar technique to have per-domain
| custom images overriding specific URL paths (eg
| `/logo.png`), which was a really quick way to allow some
| customization of a multi-tenant application without
| having to make major code modifications to support it
| (only used for a tiny handful of customers).
| yamtaddle wrote:
| Having both makes perfect sense if you like haproxy for
| what it's good at but also want a web server.
| sposeray wrote:
| [deleted]
| Amfy wrote:
| Error 403. Did he disable the last back end server in the pool?
|
| See info below, correct link:
| https://www.claudiokuenzler.com/blog/1240/haproxy-how-to-dis...
| Napsty wrote:
| I did (I am the owner of the blog). The blog article is on
| www.claudiokuenzler.com - not on this kakoku URL. I have no
| idea why Hacker News would change the link to another domain.
| Maybe someone knows this?
| Amfy wrote:
| email dang (hn@ycombinator.com). He and his team will be able
| to fix this in no time.
| Napsty wrote:
| Thanks for the hint! It may have been my error as well as I
| just discovered that copy cat URL before and blocked it.
| May have posted the copy-cat URL myself, who knows. Anyway
| I just posted a new submission with the correct link.
| dang wrote:
| Changed now from
| https://gby.kakoku.online/blog/1240/haproxy-how-to-
| disable-e.... Thanks to both of you!
|
| Our software does change links sometimes (it follows
| redirects and also uses the canonical URL when it finds
| one), but from the logs it doesn't look like any of that
| happened here.
| Napsty wrote:
| thx, appreciate it! still need to figure out what this
| okaku domain does though...
| rglover wrote:
| HAProxy has blown my mind. I'm using it to build a deployment
| tool for Joystick [1] and it's a real treat to work with. I
| utilize this specific feature (enabling/disabling servers on the
| fly) and it worked on the first swing without issue. Made scaling
| a cluster of servers fairly effortless.
|
| [1] https://github.com/cheatcode/joystick
___________________________________________________________________
(page generated 2022-09-13 23:02 UTC)