[HN Gopher] Azure Active Directory down
___________________________________________________________________
Azure Active Directory down
Author : Decabytes
Score : 77 points
Date : 2021-03-15 21:11 UTC (1 hours ago)
(HTM) web link (twitter.com)
(TXT) w3m dump (twitter.com)
| deathanatos wrote:
| It is a huge pet peeve of mine that,
|
| a. the status page update took as long as it did.
|
| b. the status page still only claims that AAD is down. You're not
| "up" if a required dependency of yours is down, IMO, and _most_
| of the Azure services right now are severely degraded to the
| point of not being useful due to the AAD outage. Now, that would
| make your status page look bad, yes. (And I would clearly message
| on it that the other outages are fallout of AAD on that.)
|
| c. I don't have any real expectation that we'll get a public PM
| about this. (And a PM _must_ include sufficient information for
| me to understand what went wrong, and what 's been done to
| prevent it going forward.)
|
| I had been starting to wonder what sort of incident it would take
| for them to actually update the status page. I guess this is it.
| We've witnessed a _number_ of outages in various services over
| the past few months, none of which made it to the status page. I
| pushed a support rep on the issue, and was told that they don 't
| want to cause a panic.
| breatheoften wrote:
| Bad status pages are a very bad sign in my opinion.
|
| I had to use the IBM cloud for a project once upon a time and
| the status monitoring was almost always wrong ...
|
| I really hope that as customers we can succeed in making
| "excellent status reporting" an absolute minimum requirement
| for cloud services. Observing the quality of the status page
| reporting history is one of the first things I do now when
| evaluating a new service -- and no I'm not just looking for a
| page that shows all greens -- I'm looking for a page that
| connects to some conceivably meaningful metric as well as
| detailed reports about major and minor outages ... something to
| prove that they actually even know if their service is working
| at a given point in time ...
|
| The IBM cloud page was a page that tended to update in response
| to my own issue reports (usually if I also pointed out that
| nothing about the outage appeared on the status page ...). I
| don't think that's at all an acceptable standard ...
| jrockway wrote:
| I'm also here because my work is blocked by the inability to do
| anything on Azure right now.
|
| I looked at their past status histories, and they have provided
| pretty detailed postmortems for much more minor incidents. I
| don't have any reason to believe that they won't do that this
| time. I'm new to Azure, so this kind of outage really looks bad
| in my mind, but it's par for the course with Cloud stuff.
| They'll fix it and there will be one less thing to break in the
| future.
| spondyl wrote:
| As someone who has been using Azure for 6 months, it's fairly
| bad but most of the Azure pain is more death by 1000 cuts
| than large explosions.
|
| Documentation that contains triple negatives, UI elements
| that have no consistency (clicking off a modal on one page
| does nothing while doing so on another closes it and clears
| your data), widgets and modals forgetting state when you
| navigate to different submenus.
|
| The sort of unexciting stuff that by itself is ok but it
| slowly makes you descend into madness.
| NDizzle wrote:
| Just wait until you see their made up rules for using their
| Redis implementation. You have to roll your own Redis to do
| anything interesting.
| zanecodes wrote:
| Their managed PostgreSQL offering also has fun quirks,
| such as that you have to connect using
| `username@db_name`, but your actual username in the
| database is still just `username`, so any third party
| software needs to support using a different username to
| connect than it uses to perform user-related queries
| (they have some sort of application-level load balancer
| in front of it that uses `db_name` to route the
| connection).
| bombcar wrote:
| This is ... something - wow - they couldn't just make the
| actual username "username@db_name"
| zanecodes wrote:
| Nope. Their load balancer seems to strip the `@db_name`
| from connections, and anyway some software doesn't like
| having an @ in Postgres usernames (which is probably also
| a sign it's vulnerable to SQL injections)
| asabla wrote:
| Been an active user of Azure for almost five or six years
| now. Things were a lot more rough in then (especially if
| you'd seen the old portal), unlike what it is now.
|
| People always talk about cloud as an entity which
| can/shouldn't have problems at all. When it's more about
| convenience then anything else. So I wouldn't worry too much
| about this if I were you.
| deathanatos wrote:
| I don't particularly expect the cloud to be problem free. I
| do expect it to be possible to, when an issue is caused by
| an outage or a service issue, get that escalated to actual
| engineers. That has not been easy, IMO. Even obvious stuff,
| like "we sent X request around Y time and got Z 500
| Internal Server Error from service S" have required
| multiple round-trips.1
|
| Messaging around transient issues helps. E.g., AWS would
| often email us to tell us "hey, sorry, such & such VM is
| being hosed by the underlying hardware, it is migrating but
| you'll see a forced reboot". That's all I need to know: I
| expect some interruption to that portion of the system, and
| I expect to see it heal within a certain time frame, and if
| it does, great, no support ticket required.
|
| Azure sort of has that with "Resource health"; but, e.g.,
| one of our recent support tickets is that we had a VM
| reboot unexpectedly due to a resource health event with its
| disks. And then, a few hours later, another VM, same
| resource health event, same issue, same reboot. And then
| again, a few hours later. And that pattern, with no further
| communication, requires me to write the (obvious) ticket
| of, "So... what's happening?"
|
| (That ticket ended with such and such service was
| experiencing an incident. Never made the status page. It
| did make the internal "Service alerts" page.)
|
| Part of issuing a PM is also to help show & convince me,
| the customer, that I won't be filing support issues for the
| rest of my life. But this last 6 months has just felt like
| support ticket after support ticket, and it's sort of
| depressing, since I'd rather be coding. I do think, last
| spring or so, it was no nearly this bad.
|
| 1Every place I've worked at... we page on 5xx codes sent to
| a client. If Azure is internally doing that, it sure is
| hard to tell from where I sit.
| NDizzle wrote:
| Affecting Teams, Azdo, and likely more services.
|
| It turned the whole afternoon into learning time at our company.
| Thankfully our Okta integration goes through our on prem AD
| servers and not purely AAD. Otherwise I wouldn't be able to get
| to learning resources which authenticate through AD!
| spondyl wrote:
| As someone working at a company that pays for Azure support, it's
| pretty sad that Microsoft 365's Status Twitter account has been
| providing more insight than Azure's own status page
|
| https://twitter.com/MSFT365Status/status/1371554704518352896
|
| They actually acknowledged it was due to a "recent change to an
| authentication system" where Azure Support just vaguely mentions
| an outage
| heyflyguy wrote:
| I am naive, but it is crazy to me that we experience outages of
| any magnitude like this from any major supplier (GOOG/MS/AMZN). I
| get alot of crap for being crotchety about making my backups in
| and out of the cloud, and it's stuff like this that proves my
| point!
| ocdtrekkie wrote:
| "The more they overthink the plumbing, the easier it is to stop
| up the drain." - Scotty
|
| Trying to provide globally available, replicated service that
| meets every and all needs 24/7/365 is basically making Active
| Directory about 600x more complicated than it would be
| otherwise. It's basically impossible for a service that
| complicated to meet the uptime of... installing Windows Server
| 2019 on a VM and patching it monthly. (Bear in mind, if I know
| when my business is not affected by an outage, I can do it
| without being disruptive. Microsoft, by definition, cannot.
| Every moment is critical since their customers are everywhere.)
|
| There are types of businesses to which the former might be a
| necessary solution, but most would be better off with the
| latter. It'll be really interesting when people start realizing
| the cloud is mostly just a scam to get people on subscription
| revenue streams, and not actually providing any greater
| reliability or less management overhead than what they had
| before.
| heyflyguy wrote:
| I agree!
| Spooky23 wrote:
| Plus, with a massive customer base and eating their own dog
| food makes their schedule be a priority over other customers.
| Your priority is lower than theirs.
|
| Microsoft is always a shitshow this time of year.
| Infrastructure changes usually land ahead of spring releases.
| ocdtrekkie wrote:
| Indeed, no matter how much we pay Microsoft, they will not
| care about our business as we care about our business.
| _Understated_ wrote:
| You know, in some ways I'm actually glad that Azure AD is down: I
| ran a SQL update and right after it I started seeing errors and
| thought I'd done something wrong... I've been shitting myself for
| an hour :D
| Severian wrote:
| Teams completely borked because of this. My employer hugely
| depends on this as we all WFH, and basically has stopped all
| communication outside of email (thank god we are on-prem).
| bombcar wrote:
| I was seeing:
|
| DX10501:+Signature+validation+failed.+Unable+to+match+keys:+
| kid:+'[PII+is+hidden]',+ token:+'[PII+is+hidden]'.
|
| and sure enough, it seems it was some authentication thing going
| sideways
| dnndev wrote:
| I run a saas company and we use azure ad. B2B sales. Good thing
| is we are down but so are they. This is strangely not that bad.
| Left the office early today
| king_magic wrote:
| This should be a huge shitshow for Microsoft. This never should
| have happened.
|
| Azure AD taking down pretty much the entirety of Azure / O365 /
| Teams is frankly inexcusable. Astounding incompetence, and an
| astonishing single point of failure that needs to be re-
| architected.
| macintux wrote:
| In fairness to Microsoft, there's no way to make authentication
| _not_ a single point of failure.
|
| You have to make the service as resilient as possible, which
| clearly they failed at, but you can't very well fail over from
| your AD service to something else.
| king_magic wrote:
| Microsoft is rolling back some update they pushed that borked
| Azure AD.
|
| Early signs are not pointing to this as a service resilience
| issue - this appears to be sheer incompetence - pushing an
| update that was likely not tested well enough that broke
| pretty much everything. More than that, why aren't updates to
| Azure AD being rolled out regionally? Why is Azure AD
| architected in such a way that the _entire thing_ going down
| can do so much damage?
|
| It's pretty hard to be understanding with a preventable
| screwup with this kind of global scale.
| adflux wrote:
| You are making a whole bunch of assumptions here without
| ANY info about how Microsoft rolls out updates to its Azure
| services.
| extrapickles wrote:
| For something as critical as auth, I would expect most
| changes to be rolled out over a few days just to make
| sure something like this cannot happen to everyone.
| Ideally a change would be shadowed for a few % of traffic
| before it was allowed to start rolling out.
| regecks wrote:
| To add another datapoint, we recently faced an issue
| where Microsoft rolled out a change to their OIDC
| endpoints, which broke our client's tenant, but not other
| tenants. (It was crashing with an HTTP 500 if the Accept
| request header was a certain value: the default value in
| OpenJDK 8).
|
| Some days later and after our complaints, Microsoft
| rolled back the change.
|
| I would call this evidence that they do practice gradual
| deployments of changes.
|
| I would guess that this outage was caused by something
| more complex than MS not slow-rolling their deployment.
| king_magic wrote:
| Sure, but those assumptions are largely backed up by _a
| single update to Azure AD_ knocking it out worldwide. End
| of the day, whatever they pushed was not nearly well
| tested enough, and they paid for it with a massive global
| outage that borked pretty much everything they sell.
| hsbauauvhabzb wrote:
| They could quite easily have progressive migrations,
| blue/green deployments, or a failover environment..
| ocdtrekkie wrote:
| Every time someone tells me we should move to the cloud, it seems
| like there's an outage the same day. Microsoft isn't actually any
| good at managing your IT infrastructure for you.
|
| Between Exchange 365 outages and Azure AD outages, I'm not sure
| how anyone thinks moving Windows infrastructure to the cloud is
| not an existential risk to business operations.
| EMM_386 wrote:
| It seems to be coming back online now.
|
| Strange that HN was down at exactly the same time, although this
| is reported to be unrelated.
|
| > CURRENT STATUS: Engineering teams are currently rolling out
| mitigation worldwide. Customers should begin seeing recovery at
| this time, with full mitigation expected within 60 minutes. This
| message was last updated at 21:12 UTC on 15 March 2021
| hateful wrote:
| cloud=a single point of failure for the whole world!
___________________________________________________________________
(page generated 2021-03-15 23:02 UTC)