[HN Gopher] Major Incident at Fastly
___________________________________________________________________
Major Incident at Fastly
Author : afrcnc
Score : 121 points
Date : 2021-01-12 15:50 UTC (7 hours ago)
(HTM) web link (status.fastly.com)
(TXT) w3m dump (status.fastly.com)
| move-on-by wrote:
| Ah, that explains the minor hiccup I just saw on my Github Pages.
| I thought it was odd that Github status was all green.
| BrentOzar wrote:
| Affected live streaming on both Mux.com and Vimeo.com (the two
| streaming companies I use for my training classes, hahaha - so
| much for redundancy with paid companies. Had to switch to
| YouTube.)
| wcfields wrote:
| Ugh, on pins and needles over here. We have a handful of Vimeo
| livestreams going on right now.
| andrewbarba wrote:
| With Mux you can enable "Redundant Streams" (multiple CDN's
| used in HLS manifest) which I think solves the issue here. We
| are also a big Mux customer but admittedly haven't made the
| full switch to redundant streams, although our testing was very
| promising. Since their ingest is all GCP I don't think that was
| affected by this, and it would have just been delivery.
|
| https://www.mux.com/blog/survive-cdn-failures-with-redundant...
| mmcclure wrote:
| Thanks for the kind words, Andrew!
|
| Redundant streams would definitely help mitigate the problem,
| but we're also days away from shipping a feature that would
| have allowed us to seamlessly switch over to another partner
| in a situation like this. C'est la vie.
| therealwardo wrote:
| there is a good chance that redundant streams would have
| helped here assuming that the player could load the manifest
| to fail over, but that manifest sadly hits Fastly. the
| problem with redundancy is that it really only works if you
| are redundant all the way up and down the serving stack. in
| our standard serving path (even without redundant streams) we
| automatically select the optimal CDN for chunk serving which
| does take into account availability, but during this incident
| we measured Fastly as about 5% less available than normal
| which wasn't enough to trigger a full automatic drain off of
| them. turns out, a 5% failure rate when you are serving 50
| chunks to enable a video playback means 1 of them will likely
| fail and you get a playback error or in the best case,
| rebuffering.
|
| sadly, Mux's manifest servers have not been redundant because
| the edge logic is a pile of Fastly's VCL. as mmcclure says,
| we're working on adding another CDN that can run our manifest
| serving logic so that we can be fully redundant.
|
| I am really looking forward to a standard and portable edge
| platform that makes redundancy at this application layer
| easier. wasm here we come?
| jgrahamc wrote:
| #hugops
| philip1209 wrote:
| And yet, their stock is up +5.32% today.
| 0xbadcafebee wrote:
| Fastly: if you could add context to the incident as you track it,
| it would help customers and users feel better.
|
| Also, linking a public postmortem to the incident would be great.
| It doesn't inspire confidence as a customer if we can't see that
| you are working on identifying the cause and are preventing it
| from recurring. (We don't need to see the full RCA or other
| process, just an overview of the problem, the cause, the fix, and
| the follow-up. This is a great way to iterate & improve on your
| RCA process)
| floatingatoll wrote:
| "A fix has been implemented": for those just now seeing this,
| you've probably read this post unnecessarily :)
| paulluuk wrote:
| I noticed that pythonhosted.org was shaky, which meant that `pip
| install` failed over and over again on different packages.
| CircleCI and Github were also affected.
| thunderbong wrote:
| Is it just me or is the text practically non readable with very
| light font on a white background?
| silicon2401 wrote:
| I just don't understand how UI/UX designers, managers, whoever
| it was, managed to convince the world that grey on white makes
| any sense. Pure trend-following
| foo_barrio wrote:
| If you have something else other than text that you want to
| be the focus of the page like a photo, reducing the contrast
| on the text allows the photos "pop" more if it using the
| range of light and dark.
|
| My personal color schemes are all dark background and light
| grey text. So even in my world of IDEs, consoles, gvim,
| sublime and emacs where text is king, I prefer having less
| than 100% contrast between background/text. I am not using
| any kind of custom color scheme so at least 1 other person
| prefers this as well!
| dylan604 wrote:
| I come the world of video, and old enough to have spent
| time with graphics for SD. Specifically, NTSC compliant
| video. Since it was analog, there were severe consequences
| of color choices. Red was just a solid NO. On an 8-bit
| 0-255 scale, white was limited to 235, and black was
| limited to 16. The reasons for this are beyond the scope of
| this chat, but that limited range has still stuck with me.
| I never use #000 or #fff and opt for #111 to #eee for my
| limits. Seems like a decent enough compromise in web
| graphics.
| Macha wrote:
| There's an argument that high contrast ratios are too high on
| the eyes, which is why "not quite black" is used on "not
| quite white" backgrounds.
|
| How people ran away on this to 50% brightness text on 70%
| brightness backgrounds is another question.
| AndrewUnmuted wrote:
| Isn't the idea here that they don't want people snooping
| around and hanging out on the part of their website that
| describes all the problems going wrong with their service?
| Shivetya wrote:
| How does any site in the US not run afoul of Americans with
| Disabilities Act (ADA)? Surely color choices like this are an
| issue that someone can raise; see WCAG 2.0 AA. which talks
| about how there must be a color contrast ratio of at least
| 4.5:1 between all text and background.
|
| Even W3C reminds people that colors are important.
| [deleted]
| subsubzero wrote:
| When asked why going with a light on light colored web page
| rendering readability to almost nil, the Designer spoke at
| length eloquently stating obscure mission statements, having
| the site born at the intersection of arts, activism and the
| global community at large. Finally they lastly mentioned
| "because its the right thing to do". /s
| vbezhenar wrote:
| Designers just want their work to be unique and eye-
| catching. They don't care about readability. They care
| about user remembering that page. If most pages have good
| contrast, designers will want to use light contrast to
| stand different. If light contrast would become the norm,
| they will reinvent hard contrast. It all goes in cycle.
| dylan604 wrote:
| I remember that page as something I can't read and will
| tell everyone I know about how horrible it is. Is that
| really what "designers" want?
| codeduck wrote:
| Is this a cry for help? :)
| gabrielsroka wrote:
| I thought maybe that was the major incident. All their design
| people have left and messed up their CSS on the way out.
| hanoz wrote:
| I haven't seen so much barely visible grey text since the
| last Hacker News thread discussing current affairs.
| uncledave wrote:
| _"It's the wild colour scheme that freaks me," said Zaphod
| whose love affair with this ship had lasted almost three
| minutes into the flight, "Every time you try to operate on of
| these weird black controls that are labelled in black on a
| black background, a little black light lights up black to let
| you know you've done it. What is this? Some kind of galactic
| hyperhearse?"_
| shrimp_emoji wrote:
| https://www.contrastrebellion.com/
| josefresco wrote:
| It's not just you, there are 73 contrast errors (WCAG):
| https://wave.webaim.org/report#/https://status.fastly.com/in...
| jshprentz wrote:
| As compared with 139 contrast errors on this HN page as of
| 2021-01-12 19:04 GMT: https://wave.webaim.org/report#/https:/
| /news.ycombinator.com...
| user5994461 wrote:
| The 139 contrast errors are all the headers "jshprentz" "12
| minutes ago" not having enough contrast.
| rectang wrote:
| That "12 minutes ago" has insufficient contrast is still
| a valid critique from an accessibility standpoint, even
| if the design choice seems reasonable at first. There
| will be sight-impaired people who struggle to consume
| those "age" snippets, as opposed to perceiving them as
| legible but de-emphasized.
| zdragnar wrote:
| I don't think the point was that they didn't matter, but
| structurally, it is 2 errors repeated 70 times as opposed
| to 140 distinct errors- might even just be one if they
| are lumped together as "post label" rather than "name"
| and "time".
| thelean12 wrote:
| HN has always had horrible UX and no desire to fix it. This
| tracks and does not excuse the original site at all.
| dxdm wrote:
| > HN has always had horrible UX and no desire to fix it.
|
| I thought the other day that's their way of discouraging
| new users, thereby slowing down the local Eternal
| September.
|
| https://en.m.wikipedia.org/wiki/Eternal_September
| pdevr wrote:
| Very true - I did Inspect Element and changed #808080 to
| #202020 without even thinking. I guess I am by now used to
| these low contrast designs :-)
| smt88 wrote:
| You should use Firefox and toggle Reader Mode. No CSS changes
| necessary.
| StavrosK wrote:
| For some reason, Reader Mode isn't available on all sites.
| It wasn't available on this one, either.
| ffpip wrote:
| Reader mode requires a certain layout of text and
| paragraphs.
|
| You can add `about:reader?url=`
|
| before the URL to force enable it.
| StavrosK wrote:
| Oh nice, thanks for that!
| Reason077 wrote:
| Safari's reader mode works on this site.
| [deleted]
| nvarsj wrote:
| Dark mode browser extension is a wonderful thing.
| 411111111111111 wrote:
| I was wondering what everyone was taking about. I totally
| forgot I installed the dark reader extension on my phone...
| Thanks for reminding me!
| m3kw9 wrote:
| The font colour they choose was named "really don't want people
| to know but still have to"
| HomeDeLaPot wrote:
| Yep, it's next to unreadable. Who thought this was a good idea?
| josephorjoe wrote:
| It gives it that really cool look of weatherworn text chiseled
| into stone.
|
| What you should do is spread some parchment over it and get
| some charcoal and rub the charcoal on the parchment over the
| chiseled letters... oh. yeah. sorry...
| karmakaze wrote:
| If you turn up the brightness to max when your eyes hurt the
| text is legible.
| thatguy0900 wrote:
| I'm 25 and had to use reader mode for this site, never seen a
| site this bad before
| TheAnswerMan wrote:
| Unreadable gray text on white background is the latest in Web
| Designer fashion.
| ec109685 wrote:
| Luckily I think it's on the way out, thank goodness. I agree
| that was an awful trend.
| mobilio wrote:
| They definitely need to work on accessibility.
| kps wrote:
| HTML is accessible by default, so nobody really has to work
| on accessiblity; they just have to refrain from working on
| breaking it. Unfortunately that seems too difficult for many
| web designers.
| kordlessagain wrote:
| It's a status page, so do they need to fix the font color or
| the actual problem first?
| exikyut wrote:
| I guess the litmus test is that it doesn't become
| inaccessible when infrastructure goes down.
| Sodman wrote:
| They obviously need to fix the actual problem first, but
| they only just found out about the new "actual problem".
| Surely somebody somewhere has actually looked at this
| status page before now, so this should have been a known
| issue already.
| [deleted]
| cdubzzz wrote:
| Is this major (per title)? Anyone experiencing it? I've got two
| sites in different affected regions that do not seem to be
| affected at all.
| arecurrence wrote:
| It's intermittent but I'm seeing it on sites that are using
| fastly (EG: terraform runs).
| bussierem wrote:
| PyPi is hosted through Fastly, and pip installs were all
| failing in our automated builds this morning, so I would call
| that pretty major.
| ctrlrsf wrote:
| Noticed a few 503 and 502s around 10:40AM EST from different
| POPs.
___________________________________________________________________
(page generated 2021-01-12 23:02 UTC)