[HN Gopher] Signal apps DDoS'ed their own server
___________________________________________________________________
Signal apps DDoS'ed their own server
Author : Daniel_sk
Score : 174 points
Date : 2021-01-16 15:56 UTC (7 hours ago)
(HTM) web link (twitter.com)
(TXT) w3m dump (twitter.com)
| mwcampbell wrote:
| Having gotten client-side retry logic wrong before, while also
| being the sysadmin responsible for keeping the server side up
| (back in the days of using a single dedicated server), I'm
| sympathetic to the Signal staff right now. We should go easy on
| them.
| Daniel_sk wrote:
| I have done the same mistake with my own app a few years ago.
| Fortunately I had a spend limit set on Google App Engine and it
| was set quite low, it could have bankrupted me :-). I believe
| that not many mobile developers actually think about this
| "retry hug" problem until it bites them. It's easy to just
| write few lines of automatic retries and think that the server
| will be fine.
| [deleted]
| cloudking wrote:
| This is why you need to implement exponential backoff on retries
| for anything at scale
| https://en.wikipedia.org/wiki/Exponential_backoff
| hnarn wrote:
| Is this what's commonly seen as "Retrying connection in:" with
| a number of seconds increasing like 5 -> 10 -> 20 -> 40 -> 80
| etc?
| whycombagator wrote:
| Could be. If implementing exponential backoff it's always a
| good idea to add some jitter (randomness) to the backoff,
| otherwise you can end up with multiple processes retrying all
| at the same time, all backing off again, and then all
| retrying at the same time etc etc
| manojlds wrote:
| Called Thundering Herd problem.
| whycombagator wrote:
| Looks like they had exponential backoff but they didn't have
| any jitter (among other things)
|
| Helpful comment from this thread:
| https://news.ycombinator.com/item?id=25803251
| nabla9 wrote:
| Exponential backoff retry time slots can be purchased using
| bitcoin.
| fitblipper wrote:
| This is why you need some form of network analytics to figure
| out if the new version you are rolling out is breaking anything
| or even seeing of something _is_ breaking. I get the "we don't
| want to compromise security even an inch", but how secure is a
| messaging app that cannot send messages?
|
| Even more critical since anyone can write clients that function
| with the server and thus a malicious actor could attach them
| without them knowing how to identify it or stop it.
| zamadatix wrote:
| I'm not sure it had anything to do with "a new version
| rolling out", the issue has always been there it just never
| had a trigger (massive enough load) up until this point.
|
| I suppose the app that can't send messages is the most secure
| of all! All joking aside if their focus is security over
| anything else then this is acceptable. Unfortunately I don't
| think most of the users have the same order of priorities so
| it's bound to create some tension over time in multiple ways
| such as testing analytics, features that don't get added, or
| just general friction due to always requiring "most secure".
|
| The last part I agree is probably the most concerning though,
| it seems like Signal's centralized services aren't ready to
| be battle tested from attacks bound to come due to it's
| popularity growth. I'm not sure matrix is the silver bullet
| to that problem either... email isn't resilient to DoS
| because it's distributed it's resilient because the key
| centralized players like Gmail can withstand constant attack
| without affecting service due to their scale. Neither Signal
| nor Matrix are ready for the attention that comes with
| serving billions from that perspective. It's something I
| think will come with growing pains though.
| Exuma wrote:
| What's up with Elon's tweet about Signal... wtf?
| approxim8ion wrote:
| He's just trying to be part of the conversation in a "how do
| you do, fellow kids" way.
| hyuen wrote:
| First rule of network programming: conservative sending, liberal
| receiving
| ncmncm wrote:
| That has turned out to be a really terrible rule.
|
| It makes it very hard, sometimes impossible to upgrade a
| protocol, because some peer somewhere will try to interpret
| your new thing if it were a broken old thing; so security holes
| get enshrined. It is much better to fail fast, and make it easy
| to see why. The only people who benefit from enabling crap
| implementations, in the end, are makers of crap
| implementations. Their stuff should fail as soon as possible,
| so they have to fix it.
|
| In some cases, it is acceptable have a "permit crap" mode that
| can be turned on in cases where the other end is not
| negotiable. But ideally the crap fails to work the first time
| it is plugged in, and returned to the manufacturer. Stuff that
| causes no end of trouble gets replaced eventually, but somebody
| has to expose the trouble, and it is best to have lots of
| company so the culprit is blamed.
| tyingq wrote:
| I remember a problem like this I ran into with a website where a
| short outage occured, then an avalanche of demand as end users
| reloaded or retried the page.
|
| We settled on throttling demand based on the incoming IP at the
| firewall...dropping incoming http packets for 3/4 of the addr
| space, then later 1/2, then 1/4, etc.
|
| That was long ago, though, long enough that it was an NetScape
| webserver.
| rvz wrote:
| The general case to use Erlang for your reliability and
| scalability issues. Plenty of suggestions and time to switch. [0]
|
| Now after the fact of its popularity from the source of it all
| [1].
|
| My response: Use Erlang.
|
| Downvoters: So for a chat app that is still down for several
| hours and needs to be reliable to handle the scale of new users,
| what should they (Signal) have used instead? Java?
|
| [0] https://mobile.twitter.com/ejpcmac/status/977192665489035265
|
| [1]
| https://mobile.twitter.com/anildigital/status/13502749007375...
| tyrust wrote:
| This post is about the app not gracefully handling server
| errors. Server errors are inevitable no matter what your stack
| is.
| vips7L wrote:
| How does erlang fix the problem of not having enough physical
| servers?
| rvz wrote:
| First of all, they're on AWS. They don't 'have' physical
| servers.
|
| Secondly, the point of Erlang is that you do not need that
| many physical/virtual servers to scale to these millions of
| users. Perhaps just a few or the same number of servers they
| have, but running Erlang instead. The BEAM VM and languages
| based on this is quite frankly designed for this scale.
|
| Still working well for WhatsApp and Discord.
| lsllc wrote:
| I think what GP is referring to is that WhatsApp's Erlang
| based system was able to handle 2M connections per server
| allowing you to do more with less [0] (although I think I
| read since the FB acquisition, WhatsApp has moved off from
| Erlang).
|
| [0] http://highscalability.com/blog/2014/2/26/the-whatsapp-
| archi...
| darksaints wrote:
| There are lots of languages that can do this, and it
| requires deliberate engineering effort in all of them,
| including erlang. I mean, you have to have all of your
| ports open, with multiple IP addresses allocated to a
| single server, and the process has to have efficient
| routing algorithms to do so. Erlang doesn't make this job
| more efficient than other languages. Erlang has two killer
| use cases: fault tolerance and horizontal scalability.
| Computational resource efficiency has never been a goal and
| it shows.
|
| The grandparent comment is nothing more than a cargo cult.
| "If we use the same language as WhatsApp, we'll magically
| be just as scalable as WhatsApp". It's horrendously naive.
| toast0 wrote:
| > I mean, you have to have all of your ports open, with
| multiple IP addresses allocated to a single server
|
| Not sure what this section is about. IP connections are
| keyed on a 5-tuple of
|
| {PeerAIP, PeerBIP, Protocol, PeerAPort, PeerBPort}
|
| If you hold PeerAIP and PeerAPort and protocol fixed
| (single server IP, single port, TCP), you can accept
| 65535 connections from any other IP (so long as the
| system managing that IP can use all the ports; if it's
| CGNAT, it probably can; if it's a traditional OS, it will
| probably have soft limits, but easily thousands).
| darksaints wrote:
| 65,535 is substantially less than 2,000,000. At a
| minimum, you'd need 31 IP addresses allocated to a single
| box in order to have 2,000,000 active connections on that
| box. Unless you're arguing that the chat clients would be
| opening dozens of connections...but that's ridiculous.
| Whatsapp was built for actual usage, not research
| projects, and more than 1-2 connections per client is
| overkill.
| toast0 wrote:
| Let me be more concrete.
|
| If your server is 192.0.2.23, and it listens on port
| 5222.
|
| You can accept 65535 connections from 198.51.100.1.
|
| 198.51.100.1:X to 192.0.2.23:5222 for X in {1:65535}
| (port zero is weird)
|
| You can also accept 65535 connections from 198.51.100.2
|
| 198.51.100.2:X to 192.0.2.23:5222 for X in {1:65535}
|
| Pick any X, like say 12345; a connection between
| 198.51.100.1:12345 and 192.0.2.23:5222 does not preclude
| a connection between 198.51.100.2:12345 and
| 192.0.2.23:5222.
|
| Thus, this only gets to be an issue if you have a large
| number of potential clients behind a single (public) IP
| address; which is generally not the case (except when
| you're testing from a load generator)
| rvz wrote:
| So in my comment, it's somewhat 'naive' to use Erlang for
| distributed systems like a messaging system and is
| 'nothing more than a cargo cult' all because I said
| WhatsApp is using it? Ha.
|
| In fact it is the other way round and especially for
| Signal's use case.
|
| Signal is already seeking for Erlang expertise in their
| server developer roles so I'm sure they're already
| looking into this.
|
| Perhaps using languages that are not suitable or designed
| for distributed systems like Java sounds more like a
| cargo-cult than using Erlang.
|
| They bought into the JVM hype until their server capsized
| on high traffic (again) and it is still down.
| KaiserPro wrote:
| A language is but just a brush in which to paint your
| logic.
|
| Granted its a lot more difficult to have 2m open
| connections in python. It is possible though.
|
| You're also assuming that Signal uses the same connection
| scheme that whatsapp does, which I highly doubt.
| darksaints wrote:
| No, you're cargo culting because you think their choice
| of language caused this...as opposed to the code that
| they wrote. There is nothing wrong with using Erlang
| here, it's a perfectly capable language for systems like
| this. But there is nothing magical about erlang either. I
| have first hand experience seeing erlang systems choke
| and croak on 30k connections, and I've seen jvm systems
| and C++ systems and go systems capably handling north of
| 1M connections. The difference isn't the language, it is
| the architecture of the app.
|
| If you don't believe me, I dare you to create an
| alternative. I'd easily take bets that Signal's current
| system could easily outperform your cargo cult
| alternative written in erlang.
| rvz wrote:
| > I have first hand experience seeing erlang systems
| choke and croak on 30k connections, and I've seen jvm
| systems and C++ systems and go systems capably handling
| north of 1M connections.
|
| Maybe you should also apply that _'experience'_ again by
| applying to their jobs posting here [0] as it seems you
| easily fit their requirements unless you doubt your own
| claims.
|
| > If you don't believe me, I dare you to create an
| alternative. I'd easily take bets that Signal's current
| system could easily outperform your cargo cult
| alternative written in erlang.
|
| So you're expecting someone here to create an open-source
| alternative to Signal using Erlang for free? With that
| $60M in funding, Signal can easily hire someone who has
| _'first hand experience'_ just like you to rearchitect
| their systems to be more reliable and fault tolerant.
|
| The job [0] is still there. I can also bet you won't help
| them out, even with your _'experience'_.
|
| [0] https://jobs.lever.co/signal/2a5fee8b-5875-46d4-a41d-
| 773a28a...
| KaiserPro wrote:
| Problem: Client retries every n seconds when it receives a 5xx
| error. This causes outage amplification
|
| Proposed fix: Rebuild the entire backend in a different
| language.
|
| estimated time to outage mitigation: 6-12 months.
|
| Now colour me "inexperienced" but I'd personally push a fix for
| exponential backoff to the client.
| vaduz wrote:
| Ah, but they are using Java [0] - and obviously they need a
| solution that works with the stack they currently have -
| proposing Erland at this time is asinine.
|
| Erlang would have been solution if they have predicted the
| brain-dead actions of Whatsapp/Facebook, and if they had ample
| time to prepare - and it would still have some hard limits on a
| number of messages it can handle.
|
| [0] https://github.com/signalapp/Signal-Server
| suyash wrote:
| Java is much capable of handling any sort of challenges, it's
| not so much about programming language from the looks of it,
| it's about architecting their backend and networking issues.
| rvz wrote:
| > Ah, but they are using Java
|
| Exactly my point. It's still down. Their Java servers doesn't
| seem to handle this scale well and aren't reliable for this.
|
| The fact that they have to "Add more servers" to handle this
| traffic tells you exactly why Erlang / Elixir or BEAM VM
| languages are better suited for this (And also can save them
| money instead of provisioning more servers on AWS).
|
| > obviously they need a solution that works with the stack
| they currently have
|
| It was down before even when they had less users. One user
| made this suggestion 3 years ago, and Signal looked at this.
| Plenty of time to switch.
|
| > proposing Erland at this time is asinine.
|
| You mean 3 years ago with their $60M in funding? Plenty of
| time and money to hire an Erlang / Elixir consultant.
| [deleted]
| darksaints wrote:
| > The fact that they have to "Add more servers" to handle
| this traffic tells you exactly why Erlang / Elixir or BEAM
| VM languages are better suited for this (And also can save
| them money instead of provisioning more servers on AWS).
|
| You do realize that the major reason BeamVM is popular for
| this use case is _because it makes it easy to add more
| servers_ , right? Erlang doesn't magically make it so you
| never have to add more servers. Actually, it's the
| opposite. Erlang is pretty computationally inefficient
| compared to most statically typed languages like Java. If
| there is any VM out there preventing your from needing more
| servers, it's the JVM. Generally speaking you can do more
| work with fewer resources with the JVM than you can with
| the BeamVM...it's just that the BeamVM makes it easier to
| add more resources.
| aaomidi wrote:
| You realize erlang isn't a silver bullet right?
| bilal4hmed wrote:
| So does this mean every message I sent till the update isn't
| rolled out to the client is still DDoSing them ?
| nimbius wrote:
| >I think the @signalapp apps DDoS'ed the server.
|
| and so the chickens come home to roost. Moxies vehement rejection
| of a distributed design seems less and less tenable each outage.
| Last time it was what...verification numbers that werent getting
| sent?
|
| and FWIW the divinations from the community are exceedingly
| helpful in a time when not even the signal website seems to
| confirm or deny any sort of outage. Signups are still being taken
| and the outage page at signal.org is still as static as ever.
| Even the twitter hasnt seen an update in nearly a day. Is anyone
| at Signal foundation at the controls?
|
| As a signal user myself I know this is going to sound rude but at
| this point other than the endorsements from musk and dorsey, why
| would _any_ new user consider this service at all if its been
| down for nearly two days? theres never any postmortem, and
| communication is generally evangelical or solicitous in nature
| for either installs or donations.
| LurkersWillLurk wrote:
| Respectfully, how exactly would federation have helped with
| this outage?
|
| Signal's own official clients failed to properly back off from
| spamming the server with incessant requests. I'm not entirely
| sure how _more_ third party clients would have helped with this
| issue.
| Matumio wrote:
| It would not have prevented the outage. But there would have
| been other servers where people could register and
| communicate during the outage. At least with Mastodon this
| seems to be working in practice. Over-populated instances
| usually switch to invite-only registration.
| xorcist wrote:
| "Oh no! _Ten_ _million_ people set up their own web page
| yesterday!
|
| The web broke. We just have wait a bit while the Elders of
| the Internet sort this one out."
|
| Said no one ever.
| bilal4hmed wrote:
| My question exactly, if majority of the users have an account
| on the @matrix instance and if that fails, wont the same
| issue happen ?
| rglullis wrote:
| The _instance_ dies, not the whole system.
|
| Synapse (Matrix reference server implementation) had for a
| long time severe performance issues when dealing with large
| rooms. Clients would try to enter a room and would take
| literally forever to be able to get messages _for that
| room_.
|
| Also, the natural reaction from people upon seeing how some
| rooms from bigger instances were causing problem was to
| _create_ their own instances and their own rooms.
|
| Federated protocols are anti-fragile.
| Youden wrote:
| Federation would, technically, decentralize the network and
| ensure there is no single point of failure. Federation would
| mean Matrix is about as likely to have a network-wide outage
| as email (or, for a more apt comparison, Mastodon).
|
| Realistically, I understand that the Matrix.org homeserver is
| an enormous part of the network and an outage there would be
| comparable to this one.
|
| However I believe that to be an orthogonal problem that also
| should be solved.
|
| I'm actually curious, why does Matrix promote the Matrix.org
| homeserver so strongly? Why isn't there an easy link to
| community-run homeservers?
|
| If you look at Mastodon for example [0], their onboarding
| process directs you to a registry of community servers [1],
| which actively promotes diversity in the network and
| therefore network resilience and reliability.
|
| Matrix meanwhile, just links you to element.io, which creates
| an account on the matrix.org homeserver.
|
| [0]: https://joinmastodon.org/
|
| [1]: https://joinmastodon.org/communities
| rglullis wrote:
| In some ways, it seems like we are watching a warp-speed
| demonstration of how evolutionary processes work. Environmental
| pressure, lots of contenders coming with slightly mutations and
| all of them passing through some fitness filter.
|
| It's just too bad that those arguing for decentralized systems
| fail to spread the meme more effectively. We federalists are
| like cockroaches.
| hayst4ck wrote:
| You seem to make a lot of statements which imply you have more
| knowledge than I would expect. Yesterday you seemed to have
| insider details on how they were managing (or not) their
| capacity. Where are you getting your information on the inner
| workings of signal?
|
| I am definitely getting strong signal that you are the type of
| person who has no perspective on what they are talking about,
| but I'm open to the idea I'm wrong and would love to be proven
| so. I think your expectations are out of line for the size and
| maturity of the company.
| rglullis wrote:
| FWIW, (pre-Facebook acquisition) WhatsApp's systems were
| serving tens if not hundreds of millions of people and they
| never failed so catastrophically.
|
| There was a difference in approach, though. WhatsApp was
| using battle-tested technology (ejabberd, XMPP) and
| _removing_ things that didn 't make sense to them
| (federation). Moxie had one trick (end-to-end encryption) and
| wanted to build a whole messaging platform around that.
| Instead of standing in shoulders of giants, he chose to
| ignore all the lessons from the past and do everything from
| scratch out of the belief that he can do better.
|
| At some point, these systems break. In Signal's case, this
| point has been reached.
| dboreham wrote:
| Thermal runaway. Easily done and I suppose hard to see as a
| potential problem until you do it to yourself.
| nullify88 wrote:
| Explains why DNS was resolving to 127.0.0.1 during the outage.
| Just didn't think it could be caused by their own app.
| alexandrerond wrote:
| To keep clients from DDoSing the server. Probably they set a %
| of requests to resolve to localhost and work with less load.
| It's a pretty good kill switch.
| lrossi wrote:
| It's easy to judge, but distributed systems are hard to program
| correctly, and hard to test. One would need to run failure
| injection tests at scale to detect this.
|
| This is made worse by mobile device virtualization being
| difficult to achieve as well. Is there a system I could use to
| spin up 10k emulated iPhones, to run a test in a CI job, for
| example?
| jackcosgrove wrote:
| It is easy to judge.
|
| At an old job of mine we had devices we controlled that could
| be rebooted. The initialization sequence after a reboot is much
| more network heavy than normal operation. We tried to push a
| critical security patch to all devices at once, meaning they
| all rebooted at once. Whoops self-DDOS.
| Vinnl wrote:
| They're asking people to share their debug logs to help diagnose
| this potential issue: https://community.signalusers.org/t/help-
| needed-please-send-...
| Bucephalus355 wrote:
| FYI the debug logs are a very curious part of all this. Signal
| streams them usually to a domain they keep their ownership of
| somewhat hidden. The domain is: debuglogs.org and the endpoint
| is just api.debuglogs.org.
|
| It appears to ultimately just front for AWS S3 backend so a
| very common architectural pattern.
| zamadatix wrote:
| I'm not sure I understood what the curious part about the
| debug logs was.
| faitswulff wrote:
| Came here to post this. Extra important since they don't
| collect analytics!
| ffggvv wrote:
| that's not a ddos. it's just called not being able to handle a
| spike in traffic.
| m-p-3 wrote:
| Which is basically what a DDoS will do with its botnet. Nthis
| case it was an involuntary one without a C&C.
| uncledave wrote:
| USDOS - Unexpected Success Denial Of Service.
|
| I woke up one Monday morning to crashed servers and angry phone
| calls once in that situation. Unfortunately it wasn't success
| on my part but what I had done the Friday before was so crap it
| couldn't handle a hundred users. But we blamed it loudly on
| unexpected success, quickly patched the mistake and everyone
| was happy. YMMV :)
| nitrogen wrote:
| Lesson learned: never ship on Friday
| Daniel_sk wrote:
| The application has a retry mechanism that will keep trying
| until a connection succeeds (with an exponential backoff), but
| it doesn't handle a case where the server is basically
| rejecting the messages on purpose (due to capacity issues).
| Suddenly all applications will start retrying the connection at
| once and there is no way to turn if off. So it will make the
| problem even much worse than it already is. They have added a
| server feature flag yesterday to change the maximum backoff
| time and they also added the handling of HTTP 508 error
| response.
| Daniel_sk wrote:
| Commits from yesterday:
|
| Feature flag for maximum retry backoff time:
| https://github.com/signalapp/Signal-
| Android/commit/93e9dd642...
|
| Add jitter to backoff time:
| https://github.com/signalapp/Signal-
| Android/commit/8f7fe5c3e...
|
| Handle ServerRejectedException (HTTP 508):
| https://github.com/signalapp/Signal-
| Android/commit/c95f0fce6...
|
| Feature flag for automatic session reset:
| https://github.com/signalapp/Signal-
| Android/commit/a3c7e7e55...
| LurkersWillLurk wrote:
| Can somebody clarify if I'm correct in believing that
| feature flags are client features that can be activated
| server-side (as opposed to an app update)?
| Daniel_sk wrote:
| These can be set by the server, you don't need to update
| the app.
|
| "A location for flags that can be set locally and
| remotely. These flags can guard features that are not yet
| ready to be activated."
|
| Here is the file with javadoc on top:
|
| https://github.com/signalapp/Signal-
| Android/blob/2c1c6fab356...
| jimsparkman wrote:
| That is correct. Typically used to enable or modify a
| feature in the client on your own time table vs. App
| Store approval.
| erdeszt wrote:
| Problem: whatever
|
| Solution: TRY-CATCH
| fctorial wrote:
| WHILE TRY CATCH
| ericbarrett wrote:
| The technical term for this is "thundering herd"
| williamdclt wrote:
| I'm wondering if another way to handle that would be the
| server accepting the request, but not actually servicing it.
| Basically let it timeout so that the client doesn't retry (at
| least, does not retry before timeout)
| LurkersWillLurk wrote:
| Unfortunately, I believe this is correct. I received over 1,000
| messages from one contact of mine that had the message of "secure
| session reset". It seems his phone tried to reset the encrypted
| connection with me over and over. Considering that I have 3
| devices in total, that's thousands of messages just from one user
| alone.
|
| I'm sure millions of devices doing the same thing probably bogged
| them down.
| spurgu wrote:
| I had this with one contact as well, the message on Android was
| _Bad encrypted message_ and on desktop it was _Error handling
| incoming message_.
| fareesh wrote:
| I've noticed that on Android, WhatsApp and Gmail sync their data
| on a periodic basis (probably using WorkManager or something like
| that).
|
| What is an efficient architecture for something like this?
|
| Are millions of devices hitting some gateway->load balancer
| everytime the Workmanager wakes up and polling for whether new
| messages have arrived?
|
| For example, if my phone does a "sync" 3 minutes from now, as do
| millions of others around the world, do we all get routed to our
| respective box that's keeping a queue of our pending messages in
| memory? If we happen to receive these messages in between the
| poll intervals via persistent socket or push notification, then
| the system wipes the in-memory queue.
|
| Am I imagining this correctly or is there a more optimal
| architecture for something like this?
| maxpert wrote:
| IDK why mobile dev folks do blind retries thinking server is some
| kind of mage. I've had debates on how much retry makes sense for
| a login service I built. People have the tendency of hey this
| endpoint used to work with these parameters didn't work? Fine I
| will keep retrying! The only difference here I think is in
| company paid job you step up immediately put in a server side fix
| to fix outage and then make these changes to roll them out. All
| of this is achieved in matter of hour or so rather than 12+
| hours.
|
| Circuit breaking, exponential backoff, bulkheading should be key
| pillars and no dev should be allowed to write a client without
| these key skills.
|
| Edit: Please add Jitter to the list of items. Any good library
| like https://resilience4j.readme.io/ will give you all of this
| for free out of box!
| champtar wrote:
| I remember some years ago iPhone Activesync client was retrying
| 5 time the same login password on authentication failure (http
| 401), and our AD was locking people out after 5 attempts ... We
| ended up writing a small proxy to fix it server side.
| adkadskhj wrote:
| I'm unfamiliar with "circuit breaking" and "bulkheading" in
| this context. Could you provide a bit more description so i can
| research them and make sure i know what i should know? :)
| cnasc wrote:
| Circuit breaking: also called "kill switch". It's having a
| way to shut down some feature if it becomes problematic.
|
| "Bulkheading" is making sure that failures in one area don't
| cascade into causing failures elsewhere analogous to how
| bulkheads on a ship prevent a single breach from sinking the
| whole ship.
| hayst4ck wrote:
| The context is somewhat important.
|
| While 508s were being sent, there was also a significant
| number of 503s (IIRC). A 503 is an absolute hallmark of
| something reaching max utilization. Sometimes it's a bad
| code push that results in memory bloat and then swap or
| significantly increased request handling time on a poorly
| threaded server (think a code push that makes a blocking
| linear request in a loop), but the vast majority of time an
| upstream dependency (specifically a data store) has been
| overloaded.
|
| So for whatever reason a data store gets slower. What
| happens upstream when this happens? The number of incoming
| requests is constant, but the time each individual thread
| spends attempting to talk to the data store is constant (or
| worsening). This means each request blocks longer
| (resulting in potential thread starvation) or there are
| more concurrent requests (load) to the data store.
|
| This creates a feedback loop of doom: as a data store slows
| down, its load (the number of requests it's handling at
| once) increases until complete failure.
|
| The only way to stop this behavior is by "failing fast."
| This is how a circuit breaker works. When the data store
| starts responding slowly, it's important not to hammer it
| with even more load, so your client watches the number of
| load related failures (or response times) and automatically
| fails requests immediately without sending it to the data
| store (circuit breaker). This allows the data store to
| become unloaded and get out of the doomed feedback loop.
| dtech wrote:
| The difference between a circuit breaker and a kill switch
| is that a circuit breaker, like a real electrical one,
| _automatically trips_ and stops requests going through for
| some time after a certain error threshold to the remote
| server has passes.
| ignoramous wrote:
| As far as I know, the original source for these patterns is
| Michael Nygard's book _Release It_ :
| https://www.amazon.com/gp/product/0978739213
| kileywm wrote:
| "Circuit breaking" is a threshold (examples: elapsed time,
| attempts, etc.) that triggers a discontinuation of the
| current action (examples: HTTP request, HTTP response, etc.).
|
| "Bulkheading" is about one part not taking down a whole,
| which is more thoroughly described here:
| https://docs.microsoft.com/en-
| us/azure/architecture/patterns...
| rektide wrote:
| Bulkhead pattern is one of my favorites.
|
| Even with a monolith, often one can spin up multiple
| copies, and assign different routes in a front end http
| proxy to different copies of the monolith.
|
| It makes it extremely clear which endpoints are consuming
| how much resources. And if one copy goes down, the others
| are ok.
|
| Absolutely one of the best things you can do, not just for
| protecting against faults, buy also for understanding the
| cost of different endpoints.
| ignoramous wrote:
| See also from AWS:
|
| https://aws.amazon.com/builders-library/making-retries-
| safe-...
|
| https://aws.amazon.com/builders-library/timeouts-retries-
| and...
|
| https://aws.amazon.com/builders-library/avoiding-overload-
| in...
| hayst4ck wrote:
| Another (big company strategy) is deadlines:
|
| https://cloud.google.com/appengine/docs/standard/python/how-.
| ..
|
| The idea is that at the time a request is made, you have a
| pretty good idea of the maximum amount of time it can take,
| so you attach that as metadata with the request.
|
| If any service anywhere in infrastructure sees that a request
| has taken more than its budgeted time, it has permission to
| immediately stop handling that request and return an error.
| wtmt wrote:
| > Circuit breaking, exponential backoff, bulkheading should be
| key pillars and no dev should be allowed to write a client
| without these key skills.
|
| This is all the more critical for client applications that
| depend on app stores to review and approve updates and end
| users who may not immediately update to a newer version because
| they don't have auto update turned on.
|
| Edit: I can't seem to find it, but there was a comment recently
| about servers using flags to tell clients to shut up for a
| specific duration and avoid exacerbating problems with retries.
| I think it was about some specific design of Dropbox.
| amluto wrote:
| In an emergency, one could rig up a server to drop TCP
| connections without sending RST. The result would be a client
| that thinks it's connected without using server resources.
| Bonus points for some code on the server that responds to
| some fraction of incoming segments with ACKs to keep
| stringing clients along.
| salawat wrote:
| Computing isn't free. Energy is being produced/generated
| somewhere to prop up the demand created by the system you
| put in place, and your network congestion adversely effects
| everyone else. Don't let the "transparency" of the network
| lull you into Tragedy of the Commons behavioral patterns.
| Act with intention. If you find your thing is overwhelming
| your own capacity to handle traffic, either implement rate
| modulation capability, or restructure your program to
| mitigate whatever is creating the problem.
|
| Just because there are selfish. a-holes who think the
| internet is an infinite resource intended solely for their
| own enjoyment, convenience and exploitation, does not mean
| it is okay for everyone to act that way. Not that anyone in
| this case is one of those types of person mind, but it's
| worth pointing out. This seems more like unexpected growth
| far outstripping provisioned resources.
| toast0 wrote:
| It's pretty effective to drop a percent of incoming SYNs.
|
| But if you're behind a load balancer, and the load balancer
| doesn't know to do that, you can't do it on the origins,
| because the load balancer will fail the origins and that's
| not good.
| eli wrote:
| There's a whole HTTP status for "you are connecting too fast"
| but that assumes the server is kinda working.
| https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429
| mey wrote:
| Most client code I have ever seen unfortunately only ever
| looks for status code != 200
| ju-st wrote:
| Signal started to handle 404 7 months ago and the 508
| code 13 hours ago. (https://github.com/signalapp/Signal-
| Android/blame/master/lib... strg+f 508)
| maxpert wrote:
| Agreed server should have a rate-limiting on each account
| to prevent DDoSing in future.
| tyingq wrote:
| Has the "batteries included" aspect of things like exponential
| backoff improved? Last I was in the details, most of the
| popular client libraries offered little, and you had to
| implement it yourself.
| maxpert wrote:
| Resilience4J has improved quite a lot. I have been using for
| over two years now, never missed out on anything.
| treeman79 wrote:
| Dozen Years ago I built a backend Ruby server, (not rails), I
| needed max performance.
|
| Our front end flex guy didn't understand backing off on
| retries.
|
| Some system had an issue, that caused the flex app for 1000
| people to go insane. Each calling ruby app endlessly with no
| delay.
|
| Ruby app remained healthy, server crashed in under 5 minutes
| from running out of disk space.
|
| Our alarm interval on free space ran every 5 minutes. So no
| time for an alarm to sound.
|
| A mix, of I was very proud of my little ruby app for scaling.
| And a a wtf, I needed to be more involved with front end
| people.
| vaduz wrote:
| First distrbuted computing fallacy is "Network is reliable".
|
| Detecting if you can deliver a packet in advance is not exactly
| a solved problem - network detection APIs at best inform you if
| the PHY/MAC layer for a given connection is up, not if any
| given packet you generate can be routed out - and fail to
| detect a number of rather common edge cases (all traffic being
| routed over VPN on WiFi, without mobile fallback, for instance,
| or mobile connection _appearing_ to be up, but not transferring
| data) - therefore ultimately the solution is to try, and try
| again, with proper backoff to prevent overwhelming your own
| infrastructure.
|
| In this case the latter appears to have been missing.
| maxpert wrote:
| I've seen "Network is unreliable" being used as an argument
| to hide laziness. I do agree you can have flaky network, but
| the biggest give of the laziness behind the argument is not
| inspecting error codes at all. In this particular case 508
| for example is clear indicator of some error status. Why
| would you even keep blind retries in place? I hate the
| argument of devs arguing for "dumb clients" and server
| "taking care of everything". Even for network failures IDK
| how having a retry within millisecond will actually help. It
| will be more detrimental to your infra, bandwidth, battery
| than a once or twice snappy experience argument.
| aftbit wrote:
| Smart server dumb client makes a lot of sense in the web-
| dev world, where the core client code is not under your
| control. As soon as you need to write your own app
| (Android/iOS/desktop), you should revisit this decision.
| The server should be able to serve a status code & header
| that requests the client to back off for a certain time
| period or with certain exponential parameters.
|
| This is a very hard thing to deploy after the fact and can
| really save your bacon, especially if you have a number of
| very old clients still out in the field refusing auto-
| updates. Sure, you can refuse to serve them, but if they
| get stuck in a tight loop retrying until you do, you're
| pretty hosed.
| theossuary wrote:
| Sure but that just falls under proper backoff. If they
| decide to use different backoff strategies for different
| classes if errors, that'd be great. But it's an easy
| mistake to make, and a very hard one to catch in testing.
| Sure if they wrote the code better (or been less "lazy")
| this wouldn't have happened, but that type of feedback
| isn't constructive imo. This wasn't a conscious decision
| that was made, it was written this way by default because
| of some combination of their team/framework/api
| client/testing/documentation/etc. Talking about why those
| led to this is much more interesting I think.
| zrm wrote:
| I've seen it regularly happen where a user's company account is
| used for email that whenever the user changes their password,
| the email app on their phone causes their account to get locked
| out by doing repeated retries with the old password.
| maxpert wrote:
| Mind sharing the app name?
| AareyBaba wrote:
| Microsoft Outlook does this.
| vbezhenar wrote:
| That kind of design is wrong because anyone can lock out any
| account (or all of them).
| abhi_kr wrote:
| Another scenario to keep in mind is the Thundering Herd
| Problem[0]. Exponential backoff without added jitter could
| still DDoS the servers.
|
| [0]:
| https://en.wikipedia.org/wiki/Thundering_herd_problem#Mitiga...
| orojackson wrote:
| Which form of jitter is better: adding a random wait time to
| a predetermined wait time that grows exponentially with each
| retry attempt, or following something like [0] where every
| retry attempt increases the possible wait time choices and
| the "jitter" is to randomly pick one of them?
|
| To illustrate the latter option, suppose the smallest retry
| time unit is 1 second. The first attempt gives you a random
| choice in {0, 1}. The second attempt gives you a random
| choice in {0, 1, 2}. The third attempt gives you a random
| choice in {0, 1, 2, 4}. The fourth attempt gives you a random
| choice in {0, 1, 2, 4, 8}. This goes on until a ceiling in
| the number of attempts or a set wall clock time is reached.
|
| [0]: https://en.wikipedia.org/wiki/Exponential_backoff#Exampl
| e_ex...
| toast0 wrote:
| It depends on what you want. The first option will tend
| towards longer waits, and the second towards shorter waits.
|
| I would tend towards increasing the minimum wait at each
| iteration (until you get to some maximim wait), because it
| it failed 10 times in the last minute, it's likely going to
| fail many times in the next minute, so we don't need to try
| more than once or twice.
|
| Also: in case the client random is broken, you don't want
| to accidentaly end up with everyone retrying after zero
| seconds forever.
| yyhhsj0521 wrote:
| Ethernet uses the second one to avoid collision.
| VWWHFSfQ wrote:
| exponential backoff will still kill the servers because the
| first thing people do is kill/restart the app or reload the
| webpage and it will just restart the backoff again
| anigbrowl wrote:
| So just set 'backoff until [timestamp]' rather than
| 'backoff for [time interval]'. Generally users restart
| because they don't know what's going an assume the client
| is stuck in loop or something. Think of a client that can
| say 'internet is up but my.server is having [local,
| regional, global] problems. I will try again at 11:37am
| EST.'
|
| Downforeveryoneorjustme is exploring an API for service
| monitoring and it seems to me like every cloud client
| should have a standardized approach of trying to reach its
| own server, then checking internet access, then checking a
| service monitor, then checking social media status updates.
|
| https://downforeveryoneorjustme.com/services/api
|
| Also think apps and devices should have limited peer-to-
| peer information sharing instead of only talking to the
| operating system. In many ways our devices are like a
| roomful of people whispering status updates to the
| operating system and/or user but never talking to each
| other because 'security'.
| edoceo wrote:
| I make a mobile app and we do this. When the app wakes up
| it tries to GET a file from /.well-known and if that
| works, proceed. If it fails we have a notice with "will
| retry at T". Timing is random between 10 and 50 seconds
| anonymousab wrote:
| Yes, but even they much back pressure provides a lot of
| reprieve.
|
| Restarting an app or even manually refreshing a web page
| takes a lot longer than an in-memory automatic retry()
| function call.
| rglullis wrote:
| Take the flip side of it. If you are working on the backend,
| you should always assume/provision for the case where there
| will be hostile clients. If your backend can only work if the
| client plays nice, I'd be weary of pointing fingers too
| quickly.
| als0 wrote:
| Trouble is...you can't tell who is malicious in a DDoS
| scenario.
| rglullis wrote:
| Yeah, that's my point actually. If your service is failing
| because of a DDOS, don't blame those trying to use your
| infra. Before pointing to issues of bad clients, Signal
| should be ready for this influx of users - _especially_ if
| your leader believes that centralized services are the only
| ones able to create a viable alternative for the masses.
| wadkar wrote:
| it's also quite possible to design and implement this one the
| server side. I mean one can implement Circuit breaking,
| exponential backoff etc. at network level (e.g. istio) based on
| the "internal service" (could be VMs/nginx rever proxies)
| outage error.
| baby wrote:
| Or back pressure.
|
| How does bulkhead helps here?
| maxpert wrote:
| 10 parallel calls already in progress that have not received
| a response from server. Why start 11th request?
| husainalshehhi wrote:
| Also token bucket: https://en.wikipedia.org/wiki/Token_bucket
| tpirx wrote:
| Is it common for linear growth to cause these sorts of issues?
| Any postmortems to share from similar app failures?
| hayst4ck wrote:
| What are you assuming is linear growth?
___________________________________________________________________
(page generated 2021-01-16 23:01 UTC)