https://www.fastmail.com/blog/partial-outage-on-30-june-2023/ fastmail mobile menu button * Why Fastmail * More Productive * How it Works * Pricing * Blog * Support * Log in * Try for free Home / Blog / Partial outage on 30 June 2023 Partial outage on 30 June 2023 06 Jul 2023 Bron Gondwana Advanced Blog Partial Outage On 30 June 2023 Hero Image Blog Partial Outage On 30 June 2023 Hero Image A partial network outage occurred at Fastmail on June 30th, 2023. Most did not notice, but it was severe for the directly affected 3-5% of our customers. Whether or not you noticed, we're truly sorry that this happened, and have a plan to avoid it happening again. This article will get very technical about how our network works! We hope you find it useful. A short summary For about 19 hours, Fastmail and a portion of the internet were completely unable to communicate with each other, including a fraction of our customers, and also some email servers. This led to delayed email, some mail fetches being disabled, and if you were using one of those networks, Fastmail being entirely offline for you! We had a single path for traffic out to the internet, and hence were unable to fix this ourselves. As a Reddit comment said, this wasn't within our control, but it is our responsibility. It is not acceptable to us that a subset of our customers were unable to access their email for so long, and also that we were unable to send email to a section of the internet. To prevent this from happening again, we plan to connect directly into an additional transit provider, so we can select between multiple pathways for outbound data. Having this control will allow us to route around individual provider network issues, and be back online for everybody faster. What, when, where, who * What: access to all Fastmail services, including mail delivery, was broken - in particular, the ability for network packets leaving our network to reach their destination. * When: starting approximately 03:00 UTC on June 30th and ending about 22:00 UTC on the same day, for a total outage time of 19 hours. We're quite certain that the root cause of this outage is now fixed. * Where: a fraction of the internet. * Who: We estimate 3-5% of our customers, based on comparing server metrics to similar days. We first became aware of the issue when our customers informed us via support tickets and social media. While we have monitors from multiple locations around the world, neither those monitors nor any of our staff were on networks that had issues. We could tell it was quite isolated, because some reported being able to access from phone but not home network, or vice versa. Also VPNs allowed some users to find a functioning pathway to our servers. Since we couldn't see the failing links ourselves, we relied on helpful customers on those networks who provided us with traceroutes -- which then gave us some addresses to trace back to from our end and see where the traffic was dropping. We very much appreciate everybody who assisted us with datapoints! Some history A few years ago, we moved from New York Internet's Manhattan datacenter to their Bridgewater, New Jersey location. A bit later, this datacenter was sold to 365 Datacenters. We were using the IP address range 66.111.4.0/24 -- a range which still belongs to the original NYI. Thankfully, NYI were willing to lease us those addresses, because IP range reputation is really important in the email world, and those things are hardcoded all over the place -- but it has caused us complications due to more complex routing. Over the past year, we've been migrating to a new IP range. We've been running the two networks concurrently as we build up the trust reputation for the new addresses. In past years, we have had DDoS attacks against our service, and our previous DDoS protection services couldn't keep us up through the worst of those attacks, so we're now bringing all our incoming traffic via a service called MagicTransit from Cloudflare. It routes everything at the IP layer, so all the traffic is encrypted just like it was going over any other network -- but Cloudflare has the capacity to soak up a lot of attack, and they're really quick and proactive. As part of the MagicTransit setup, we also added a second way into our network. Cloudflare can either route to us over our "Private Public" network IP on the 365 managed network, or via Megaport, a service which provides private virtual cables. We have a fiber link to Megaport, so we can route traffic from Cloudflare into our network without using the uplink to 365's network. Traffic normally comes in that way, it's a little faster, so we give it priority. active cloudflare configuration Megaport was up during the entire incident, so packets were getting in to our network reliably. It was outbound packets that weren't all getting through. Our current network structure diagram of our network layout The Future Work is now underway to select a provider for a second transit connection directly into our servers -- either via Megaport, or from a service with their own physical presence in 365's New Jersey datacenter. Once we have this, we will be able to directly control our outbound traffic flow and route around any network with issues. Thanks for reading, and as always thank you for using Fastmail (both those who were unaffected, and those of you who have kept using Fastmail after that very bad day!) twitter share link facebook share link linkedin share link [blazy] Bron Gondwana - CEO Read more by this author Upgrade your privacy and productivity and join the best in email. Sign up for your free 30-day trial > Want more information? Visit our side-by-side comparison chart to learn more about why Fastmail is a great alternative to Gmail. Older post Fastmail and Morgen: Empowering Productivity while Protecting Privacy Fastmail and Morgen: Empowering Productivity while Protecting Privacy 13 Jun 2023 Company Partnerships Productivity Tips Fastmail and Morgen: Empowering Productivity while Protecting Privacy We're thrilled to announce a new partnership between Fastmail and Morgen, creating a productivity dream team that prioritizes the privacy of customers' data and delivers exceptional user experience. This Self-Taught Systems Engineer Builds on His Knowledge Alongside a Values-Driven Team at Fastmail 25 May 2023 Meet Our Team This Self-Taught Systems Engineer Builds on His Knowledge Alongside a Values-Driven Team at Fastmail Macon Gambill has jumped into his new role at Fastmail. Pros and Cons of Hosting Your Own Email 24 May 2023 Advanced Privacy & Security Pros and Cons of Hosting Your Own Email Are you considering hosting your own email? Learn more about the pros and cons from Fastmail. letters background Learn more about our privacy features subscribe input label [ ] subscribe submit button You have successufully subscribed! Product * Why Fastmail * More Productive * How it Works * Compare to Gmail * Compare to HEY * Try it Resources * Help Center * Contact support * Blog * Podcast * Press Company * About Fastmail * Privacy-first * Developing Open Standards * Partnerships * News * Careers Get Started * Download the App * Setting Up Fastmail * Pricing * Log in google play app store facebook link twitter link linkedin link Mastodon link fastmail (c) 2023 Fastmail Pty. Ltd. | All rights reserved facebook link twitter link linkedin link Mastodon link Privacy Terms Fastmail Policies google play app store Website by