[HN Gopher] Deno's July 13th incident update
___________________________________________________________________
Deno's July 13th incident update
Author : mostafah
Score : 104 points
Date : 2022-07-16 08:07 UTC (14 hours ago)
(HTM) web link (deno.com)
(TXT) w3m dump (deno.com)
| tetha wrote:
| Ew, we've had similar issues in the past. These are really messy
| and confusing to recognize.
|
| In our case, 1 out of 5 LB instances lost its connection to the
| service discovery and later on ended up not knowing about a
| failover of one of the 5 backends for a service. As a result,
| something like 1 in 20 to 1 in 25 requests got answered with a
| connection refused. That took a minute to find.
| nijave wrote:
| Had something similar when a k8s node broke but k8s thought the
| pods (envoy) on it were still running so it routed 1/nth of
| traffic into a black hole
| sentrms wrote:
| Is anyone running mission critical software on this rather new
| platform? In my experience every added cloud service becomes
| another potential weak link in your chain. A distributed DB for
| your data, a CDN for static assets, a couple of lambda functions
| for background processing. With every move away from the monolith
| your surface for potential downtime or "elevated error rates"
| increases.
| FBISurveillance wrote:
| Hugops to the team. A quick question: is it intentional that
| there's nothing on https://denostatus.com/?
| weird-eye-issue wrote:
| What would be the benefit for Deno to have an accurate status
| page? It would only stand to detract potential
| customers/investors
| quickthrower2 wrote:
| They are establishing their brand, transparency, ethics and
| trust is what should be part of that for what is the next
| Heroku/Digital Oceam.
| weird-eye-issue wrote:
| You guys took my comment way too literally. It was tongue in
| cheek :P
| lucacasonato wrote:
| No, it wasn't intentional. Because the incident was not
| triggered by automation, the incident on the status page was
| not automatically created. The team did not remember to update
| the status page while we were investigating the issue. Sorry
| for that!
|
| I have retroactively added the outage to the status page now:
| https://denostatus.com/cl5ob2i5s943266vk890ushwov.
| Shadonototra wrote:
| zx8080 wrote:
| Interesting. Any links to submissions that were flagged or
| removed (screenshots in that case)?
| Shadonototra wrote:
| croes wrote:
| >YCombinators silencing everything as usual
|
| That's not asking.
| jmarneweck wrote:
| They mentioned that their automated monitoring did not pick up
| the issue and that they are working on improving the tooling
| around this.
| ctvo wrote:
| > On July 13th, at around 18:45 UTC we started to receive reports
| of an outage from a small number of users. We investigated the
| status of our services, but were unable to confirm any of the
| reports. All of our status monitoring and tests reported that
| everything was operating normally.
|
| > Over the course of the outage, we continued to monitor our
| service status, and worked with some of the affected users to
| narrow down the source of the problem.
|
| > On July 14th, at 19:14 UTC we were able to identify that the
| problem was within our us-west3 region, which we then took
| offline, directing traffic to other nearby regions instead.
|
| The time difference between when the first reports came in and
| when it was confirmed is a little concerning.
|
| As an aside:
|
| > ... approximately 18:00 UTC ...
|
| > ... just over 24 hours ...
|
| > ... For a period of around 24 hours, some users in the us-west3
| region
|
| > ... less than 30 minutes ...
|
| > ... On July 13th, at around 18:45 UTC we started to receive
| reports of an outage from a small number of users. ...
|
| "Approximately", "just", "around", "some", "small number of". It
| goes on and on. I disagree with the stylistic approach of being
| _less_ specific in posts like these. A "small number of users"
| is relative. As readers, we have no idea what your typical load
| may be. Small may be a large number to us. "Just" over 24 hours
| is 26 hours? 24.5 hours? I implore you to be specific when you
| have the actual data.
|
| These terms read as weasel words, and impact your effort at being
| fully transparent.
| TechBro8615 wrote:
| This is a brand new platform and I assume this blog post is one
| of their first post-mortems. I'm inclined to give them a break,
| and respect them for establishing a process of post-mortems.
|
| That is all to say, regardless of whether the post itself is "a
| little concerning," it would be more concerning if the post
| didn't even exist. And if you weren't one of their affected
| customers, you likely wouldn't even know this happened. So they
| did the right thing by publishing it and opening themselves to
| your criticism, which is a positive sign for the future of the
| platform IMO.
| steve_adams_86 wrote:
| In this case it's disappointing especially because we know they
| have exact numbers.
| geysersam wrote:
| Arguably the exact numbers aren't interesting either. 456 982
| requests from 5674 projects by 3091 user accounts were
| dropped during the 23:53:42.104 hours the outage lasted. I
| think it makes sense to make an interpretation and present
| that instead of raw numbers.
| [deleted]
| turtlebits wrote:
| Seem like a huge gap in observability - Low/zero healthy targets
| for a load balancer should be a P0/critical alert, especially
| when traffic is getting black holed.
|
| LBs should also be alerting on health checks failures/no data for
| targets as well.
| Shadonototra wrote:
| > several services provided by the Deno company experienced a
| service disruptions in our us-west3 region for a period of just
| over 24 hours.
|
| '"JUST" over 24 hours', no big deal of course /s
|
| https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
|
| no mention of that issue
|
| either nobody uses Deno so 0 complains
|
| or people use Deno and for some reasons 24h+ downtime didn't
| impact anybody, wich is surprising, to say the least
| atwood22 wrote:
| "Just over" 24 hours. As in slightly more than 24 hours. Which
| is different than "just" 24 hours.
| capableweb wrote:
| > On July 13th, at around 18:45 UTC we started to receive
| reports of an outage
|
| > On July 14th, at 19:14 UTC we were able to identify that
| the problem was within our us-west3 region
|
| So yeah, 24 hours and 29 minutes.
| alluro2 wrote:
| I don't mean anything bad to Deno's team (I'm very partial to
| what they're building), but I'm rather surprised whenever a
| widely-publicized service has an outage that lasts hours or more
| than 24h. I'm genuinely curious to understand whether it's
| typically due to complexity of infrastructure and how hard it is
| to find route causes, how long it takes to redirect traffic /
| patch temporarily when the cause is found, or is it due to
| attitude where it's considered normal for these things to happen,
| and to take time to solve step by step.
|
| Our services are of what I consider medium complexity (~70
| services, ~10 different "layers" of logic, db, caching, load
| balancing etc, AWS, mostly self-managed centralized logging and
| monitoring) but still quite low-volume (< 100 requests / second),
| and any more serious issue (let alone outage) is spontaneously
| met by my team as absolute emergency and typically fixed in < 10
| minutes.
|
| We're very modestly funded compared to Deno (in this example) and
| the team is small...
|
| Not sure whether that changes with traffic volume, complexity,
| team size, or is more primarily attitude-based and should
| continue to be cultivated.
| lucacasonato wrote:
| Our issue here was very much in finding the root cause. Because
| the failed traffic was "black holed" (TCP connections were
| being dropped), we had very little information other than "it
| isn't working" from the users that reported the issue. This
| caused us significant headaches in trying to figure out what
| the commonality between the incident reports of our users was
| (the geo region). Up until the point this was clear, we were
| also checking database clusters, DNS configurations, TLS
| certificates etc to try to isolate the issue.
|
| After we managed to successfully isolate the issue we were able
| to disable the region within 30 minutes, because we had an
| established protocol for how to do that.
|
| Here is a more typical incident update for us:
| https://deno.com/blog/2022-05-30-outage-post-mortem
|
| Part of the issue was also that we did not realize the scope of
| the issue right at the start of the incident, because our
| automated monitoring did not catch the dropped traffic.
|
| All that is to say: the outage is obviously unacceptable, and
| sincerely apologize for it. We are working very hard to make
| sure nothing similar can occur again in the future.
| alluro2 wrote:
| Thanks for the insight - I definitely wasn't trying to dump
| on the team or handling of the issue - really just understand
| better so I have more awareness and can hopefully help my
| team (as a young CTO) be more prepared for different types of
| challenges.
|
| As mentioned, I'm looking forward to continuing to follow
| Deno's progress and all the best in hardening your devops!
| viraptor wrote:
| > is spontaneously met by my team as absolute emergency and
| typically fixed in < 10 minutes
|
| Unless they had the "route the whole region over another one"
| in their prepared and practiced DR procedure, it would take any
| team a significant time to get that planned, approved,
| implemented and tested.
|
| If you're running something at tens of services scale and
| recovered in 10min, you're extremely lucky. I'd suggest that if
| you don't have risks on your list that will take hours to
| resolve, your list is not complete.
| alluro2 wrote:
| That's a fair point and a good suggestion to consider.
|
| One alleviating circumstance is that, running on AWS, a big
| portion of such issues (ones that would take a lot of time to
| resolve) would come from wider AWS outages - when there's
| significant leeway - the old adage that customers / big part
| of web would have bigger issues than us being down if an
| entire AWS region (or multiple) is down.
|
| In Deno's case, most of "those" parts are self-managed and
| surely much more difficult to keep running reliably.
| jdlshore wrote:
| I'm always curious to learn about why people create complex
| architectures. It's off-topic, but why so much complexity for
| such a low volume?
| AtNightWeCode wrote:
| "... (a TCP load balancer). It does not record any diagnostics
| about dropped connections, nor does it have a return channel to
| return diagnostic information to the user (unlike HTTP
| loadbalancers, which can return a response header)."
|
| And there is no API monitoring apparently.
| lucacasonato wrote:
| A bit of a blunt statement on my part. There is monitoring on a
| multitude of other connection related issues (eg TLS handshake
| failures, missing SNI, etc). We should have had monitoring for
| this specific failure where the load balancer did not have any
| healthy backends, but as mentioned in the post, the load
| balancer was programmed in way that this should never have been
| able to happen in the first place (as the LB should have un-
| advertised itself if there are no unhealty backends).
|
| We are capable of learning from past mistakes though, and as
| such we'll make sure to add more monitoring for these kinds of
| scenarios so we can be alerted to a root cause earlier. We will
| do better.
| remram wrote:
| So three failures:
|
| - The load balancer lost its connection to etcd and did not
| reconnect
|
| - The load balancer had no healthy backend and did not un-
| advertise itself
|
| - The load balancer did not report either of those issues to
| monitoring
|
| Honestly this is a little concerning. Are they using their own
| load-balancing software? If yes, why?
| lucacasonato wrote:
| The system does have mitigation against the first two failures
| in isolation (as described in the post). The mitigations did
| not work correctly in this scenario with the combined failures
| unfortunately. This is obviously unexcusable, and we need to do
| better in the future.
|
| To your final question: yes, we using our own load balancing
| software. We are building a global hosting platform that needs
| to be able to run on bare metal servers, not an end user
| application where load balancing is an afterthought. As such we
| can not use much of the software that a "regular" SaaS
| application may be able to. Some constraints our system needs
| to be able to solve:
|
| - Our load balancers handle routing to 100s of thousands of
| unique deployments (services), all of which need to be
| accessible and routeable within milliseconds of a request
| coming in.
|
| - We need to terminate TLS connections for thousands of unique
| domains.
|
| - We need to be able to carefully control TLS handshakes, to be
| able to prewarm downstream services for an imminent request for
| a given deployment based on a TLS client hello SNI, before even
| having received an HTTP request yet.
|
| - The system needs to handle hundreds of millions of hourly
| requests.
|
| - The system needs to be able to run on bare metal.
|
| - We currently handle 34 regions globally (up from 28 at the
| start of the year), which means that all of the data needed to
| fulfill the above requirements needs to be accessible from all
| of our PoPs in a matter of milliseconds.
|
| For many companies global load balancing is something they can
| outsource to AWS, GCP, or Cloudflare. For us, this is core
| "business logic" that we need to have full control over. It's
| difficult for us to outsource, and it's questionable if it
| would be wise for us to do so. Building new systems is
| obviously always a complex undertaking, and there will be some
| stumbling stones in the way, but they can be overcome. We are
| still bullish that our path is the right one, even if we still
| have a lot of work ahead.
|
| (if this seems interesting, and you want to work with us on
| building load balancers, among other things:
| https://deno.com/jobs)
| lowwave wrote:
| instead of the AWS ones?
| remram wrote:
| Yes or even a more turn-key software package. It sounds like
| they had very custom software, I would expect that
| established load-balancing software doesn't fail to
| reconnect.
___________________________________________________________________
(page generated 2022-07-16 23:01 UTC)