https://www.theregister.com/2023/01/30/wan_router_ip_address_change/ [user] [user] Sign in / up The Register(r) -- Biting the hand that feeds IT [magn] [burg] [burg] Topics Security Security All SecurityCyber-crimePatchesResearchCSO (X) Off-Prem Off-Prem All Off-PremEdge + IoTChannelPaaS + IaaSSaaS (X) On-Prem On-Prem All On-PremSystemsStorageNetworksHPCPersonal Tech (X) Software Software All SoftwareAI + MLApplicationsDatabasesDevOpsOSesVirtualization (X) Offbeat Offbeat All OffbeatDebatesColumnistsScienceGeek's GuideBOFHLegalBootnotesSite NewsAbout Us (X) Special Features Special Features Government Tech Week Vendor Voice Vendor Voice Vendor Voice All Vendor VoiceAmazon Web Services (AWS) Business Transformation Google Cloud for Startups (X) Resources Resources Whitepapers Webinars Newsletters [networks] Networks 35 comment bubble on white WAN router IP address change blamed for global Microsoft 365 outage 35 comment bubble on white Command line not vetted using full qualification process, says Redmond. We think it involved chewing gum somewhere icon Paul Kunert Mon 30 Jan 2023 // 13:35 UTC # The global outage of Microsoft 365 services that last week prevented some users from accessing resources for more than half a working day was down to a packet bottleneck caused by a router IP address change. Microsoft's wide area network toppled a bunch of services from 07:05 UTC on January 25 and although some regions and services had come back online by 09:00, intermittent packet loss woes weren't fully mitigated until 12:42. The wobble also affected Azure Government cloud services. In a postmortem, Microsoft said that changes made to its WAN had hit connectivity between clients and Azure, across regions and cross-premises via ExpressRoute. [networks] "As part of a planned change to update the IP address on a WAN router, a command given to the router caused it to send messages to all other routers in the WAN, which resulted in all of them recomputing their adjacency and forwarding tables. During this re-computation process, the routers were unable to correctly forward packets traversing them. [networks] [networks] "The command that caused the issue has different behaviors on different network devices, and the command had not been vetted using our full qualification process on the router on which it was executed." This meant users were unable to access resources hosted in Azure or other Microsoft 365 and Power Platform services. [networks] Microsoft said monitoring systems detected DNS and WAN-related troubles at 07:12, some seven minutes after they began. By 08:20, resident techies at Microsoft had spotted the "problematic command that triggered the issues" and some 40 minutes later networking telemetry indicated many of the services were running again. * UK government in talks with datacenter operators over blackouts * Datacenter outages are costing more, $1m+ failures now common * Exchange Online and Microsoft Teams went down in APAC because Microsoft broke itself * Microsoft Teams outage widens to take out M365 services, admin center However, Microsoft said the initial problem with the WAN meant automated systems for maintaining its health were paused. This included systems for identifying and expelling unhealthy devices, as well as the traffic engineering system for optimizing the flow of data across the network. "Due to the pause in these systems, some paths in the network experienced increased packet loss from 09:35 UTC until those systems were manually restarted, restoring the WAN to optimal operating conditions. This recovery was completed at 12:43 UTC," the postmortem added. Efforts Microsoft is taking to make similar incidents less likely or severe include blocking "highly impactful command from getting executed on the devices" and requiring all command execution on devices to follow safe guidelines. [networks] The final post-incident report is scheduled to be published a fortnight after the outage. (r) Get our Tech Resources # Share Similar topics * Azure * Microsoft * Microsoft 365 More like these x Similar topics * Azure * Microsoft * Microsoft 365 * Outage * Saas Narrower topics * Active Directory * Bing * BSoD * Excel * Exchange Server * HoloLens * Internet Explorer * LinkedIn * Microsoft Build * Microsoft Edge * Microsoft Ignite * Microsoft Office * Microsoft Surface * Microsoft Teams * .NET * OS/2 * Outlook * Patch Tuesday * Pluton * SharePoint * Skype * SQL Server * Visual Studio * Visual Studio Code * Windows * Windows 10 * Windows 11 * Windows 7 * Windows 8 * Windows Server * Windows Server 2003 * Windows Server 2008 * Windows Server 2012 * Windows Server 2013 * Windows Server 2016 * Windows Subsystem for Linux * Windows XP * Xbox * Xbox 360 Broader topics * Bill Gates * Cloud Computing * Office 365 * Software Similar topics # Share 35 comment bubble on white COMMENTS Similar topics * Azure * Microsoft * Microsoft 365 More like these x Similar topics * Azure * Microsoft * Microsoft 365 * Outage * Saas Narrower topics * Active Directory * Bing * BSoD * Excel * Exchange Server * HoloLens * Internet Explorer * LinkedIn * Microsoft Build * Microsoft Edge * Microsoft Ignite * Microsoft Office * Microsoft Surface * Microsoft Teams * .NET * OS/2 * Outlook * Patch Tuesday * Pluton * SharePoint * Skype * SQL Server * Visual Studio * Visual Studio Code * Windows * Windows 10 * Windows 11 * Windows 7 * Windows 8 * Windows Server * Windows Server 2003 * Windows Server 2008 * Windows Server 2012 * Windows Server 2013 * Windows Server 2016 * Windows Subsystem for Linux * Windows XP * Xbox * Xbox 360 Broader topics * Bill Gates * Cloud Computing * Office 365 * Software TIP US OFF Send us news --------------------------------------------------------------------- Other stories you might like Microsoft to offer ChatGPT-as-a-service from Azure real soon now Addition of the chatbot everyone's talking about, and to, comes as Azure OpenAI goes GA AI + ML17 Jan 2023 | 19 Microsoft: You're not out of love with cloud, you're just 'optimizing' it for a bit Q2 results include modest revenue increase, steep consumer and net income decrease Software25 Jan 2023 | 8 Azure Stack HCI gets extra protection with 'long-requested feature' Kerberos support in WAC and a host of other goodies also added Software19 Jan 2023 | 1 What does it really take to be a leader when it comes to cloud databases? Here's what the AWS customer obsession means to you Sponsored Feature [networks] Microsoft's mixed reality dream meets harsh reality of job cuts Updated Just so, so much enthusiasm for Second Life 2.0 right now Personal Tech24 Jan 2023 | 6 Microsoft applies coat of Rust to Azure Sphere IoT platform The hope? To grease the security skids for internet-connected devices - and they need all the help they can get Edge + IoT13 Jan 2023 | 9 If your Start menu or apps are freezing up on Windows, Microsoft has a suggestion No, it's not install Fedora Linux, sadly OSes26 Jan 2023 | 95 Microsoft can't stop itself blowing billions on OpenAI Embrace, extend ... extinguish, but whom or what this time? AI + ML23 Jan 2023 | 9 Microsoft boosts Azure networking, storage with composable infra grab Slurps 'Fungible', a manufacturer of DPUs and fabrics that will join Redmond's engineering gang Off-Prem10 Jan 2023 | 2 Microsoft shells out for 2.5GW of solar. Not that it'll make a big dent in its emissions Unless it's hooking up all those panels to your PCs and servers Systems26 Jan 2023 | 24 Microsoft squashes Windows 11, Server 2022 bugs with preview patches Remote Desktop, domain controllers giving you gyp? Gamble on these fixes OSes28 Jan 2023 | 4 Polish for Windows Spotlight and tabs for Notepad in latest Insiders build You will soon know more about what your PC lockscreen is showing you OSes23 Jan 2023 | 20 The Register icon Biting the hand that feeds IT About Us* * Contact us * Advertise with us * Who we are Our Websites* * The Next Platform * DevClass * Blocks and Files Your Privacy* * Cookies Policy * Privacy Policy * T's & C's * Do not sell my personal information Situation Publishing Copyright. All rights reserved (c) 1998-2023 no-js