[HN Gopher] Linode Outage in Dallas, Fremont, Atlanta, Newark, a...
___________________________________________________________________
Linode Outage in Dallas, Fremont, Atlanta, Newark, and Toronto Data
Centers
Author : boplicity
Score : 144 points
Date : 2022-11-04 12:45 UTC (10 hours ago)
(HTM) web link (status.linode.com)
(TXT) w3m dump (status.linode.com)
| mbesto wrote:
| Seems like more and more outages are happening every day...tsk
| tsk
|
| Shameless plug - we're helping orgs fix this and get better
| clarity:
|
| https://www.awareops.com/
|
| Shoot me a message mb @ awareops dot com or go ahead and sign up.
| mistertrotsky wrote:
| My sites in Newark are up. The situation is not as alarming as
| the submission title would suggest. They've updated the status
| page to indicate that it is limited to an issue with DHCP.
| underdeserver wrote:
| 3 minutes of monitoring? That's enough to declare the issue over?
| rossdavidh wrote:
| In almost 20 years with Linode, I've had nearly zero problems.
| Akamai buys them, and within months we have an outage. Not
| encouraging...
|
| On the other hand, it's only hitting my personal website, and the
| customer I have on Linode seems unaffected, so that's nice at
| least.
| tiffanyh wrote:
| Others have had different experiences.
|
| https://news.ycombinator.com/item?id=14571234
| bananapub wrote:
| I've also been with them nearly twenty years (!?!), and they
| definitely have had problems throughout that entire time.
| perhaps the difference was that you'd get caker@ and co on IRC
| explaining what was up.
| tomxor wrote:
| I'm wondering how much of this is perception...
|
| The number of _reported_ issues coming through has exploded
| since Akamai. Over the last week alone, almost every day a new
| networking issue for London is in my inbox. However - none of
| the recent ones have actually affected any one of my Linodes,
| even though it 's possible since it falls under the service and
| datacenter - perhaps the change we are seeing is more
| transparency?
|
| I'm not saying no one is affected... as others in this thread
| clearly are, and I was affected by the datacenter wide outage
| in London earlier in the year - but it's possible that being
| more upfront in notifications of issues could change the
| perception of what before would have been considered a
| collection of isolated incidents.</speculation>
|
| [edit]
|
| I generally appreciate transparency, but when it comes to
| varying degrees of service disruption maybe there is a balance
| to be struck... right now it feels like the signal to noise
| ratio is low in terms of it's affect across all users even
| within an area.
|
| Right now I am pretty much ignoring the Linode issues filling
| my inbox, if a serious one came through I'd probably miss it.
| williamscales wrote:
| I moved off Linode 5 years ago due to outages and issues. I
| don't have any experience post-akamai to compare but it was
| definitely never a reliable service for me in the past.
| ilyt wrote:
| Which site ? I've been in Fremont, then London with basically
| zero problems (few scheduled migrations for when they were
| changing hardware)
| rapind wrote:
| I've had a non trivial number of issues with them over the
| years. Enough that I moved all of my infrastructure to other
| (more expensive) providers.
|
| That being said, it was probably value for money spent while it
| lasted.
| rossdavidh wrote:
| Well, it appears to be fixed now (for my websites at least), so
| I'm not as worried about Akamai as I was a couple hours ago. If
| it's another decade before I have an outage, I'm satisfied. On
| the other hand, if this happens again any time soon, it will be
| time to start evaluating options, I guess.
| bragr wrote:
| To be fair, I know from personal experience Fremont has gone
| down hard for hours before.
| lukeqsee wrote:
| I think "Outage" as stated is a bit over-the-top. We run
| significant operations in most of these DCs, and we have seen
| zero impact. YMMV of course, but I think it's a big disingenuous
| to say this is a full outage.
| boplicity wrote:
| It's a full outage for _me_ -- wasting quite a lot of time. It
| 's been 7 hours so far. Hopefully it won't drag on longer.
|
| Who knows how many people it's affecting?
| squidbeak wrote:
| For us, too.
| CoffeeOnWrite wrote:
| Mine in Dallas became inaccessible around 7:20am PT and came
| back around 7:50am.
| wolrah wrote:
| For what it's worth I have around 25 Linodes active, mostly in
| Newark but a few in Atlanta and Dallas as well.
|
| Of those I had two in Newark which actually had their leases
| expire before we were aware of what was going on and then found
| three more (two Atlanta, one Newark) once we knew what to look
| for which were failing to renew. All five of those machines
| already had network helper enabled as Linode suggests but were
| unable to get a lease even with manual operation of the DHCP
| client until they were rebooted.
| eatonphil wrote:
| They were just bought by Akamai so unfortunately things like this
| are bound to happen if they are trying to integrate with Akamai
| the way they'd need to successfully keep growing. I'm not talking
| just software integration but process and people integration as
| well. We'll see I guess!
| ubermonkey wrote:
| I'm absolutely certain Akamai will fuck Linode up, same as
| Rackspace ruined Slicehost -- which is what sent me to Linode
| in the first place.
|
| I'm waiting for a new Linode to pop up.
| fuckstick wrote:
| > I'm waiting for a new Linode to pop up.
|
| Eh, what? Vultr, DigitalOcean, Hetzner, OVHcloud already
| exist. The first 2 at least have feature parity I thought.
| mdasen wrote:
| I don't think that will happen with Linode (but I certainly
| could be wrong). When Rackspace bought Slicehost, people
| didn't really know what the future was going to look like. At
| this point, we've seen AWS/GCP/Azure and Akamai wants to have
| their own "run your apps with us" system.
|
| Unlike when Rackspace bought Slicehost, Linode probably isn't
| even the leader in its niche. That's probably DigitalOcean.
| Akamai probably figured out which of the two it could buy and
| DigitalOcean probably would have cost $4+ billion while
| Linode only cost $900 million. When Rackspace bought
| Slicehost, there were really two alternatives: Linode and
| AWS. AWS came with a lot of complexity and Linode was small.
|
| If you're looking for a new Linode to pop up, you don't have
| to wait. Vultr, DigitalOcean, and tons more already exist and
| have for years. It's not 2010 where you don't have loads of
| options in this space. Go with Hetzner or OVH Cloud and get
| really cheap prices (I've found Hetzner Cloud to be really
| nice). There are so many places that will give you a VPS
| today. Heck, AWS will give you a Linode-like experience with
| Lightsail, right?
|
| That's why I think Akamai wants to use Linode as a way to
| build something that will let them start getting into the
| AWS/GCP/Azure space. It's somewhat where Linode was already
| heading, albeit slowly. Linode started offering hosted
| Kubernetes, object storage, block storage, hosted databases,
| and load balancers. How long before they might have added a
| queuing system, an app-runner so you don't even need to think
| about boxes/load balancers, etc.
|
| For the low price of $900M, they could jump-start that
| process at Akamai. Rackspace thought that the future was
| their high-touch, high-cost, managed services. Akamai
| probably wants to use Linode to get into the AWS space and
| with a separate brand, they can do it without as much risk to
| their mainline business. There are too many alternatives for
| Akamai to be deluded enough to think they can just ruin
| Linode and not like that $900M on fire. That wasn't the case
| when Rackspace bought Slicehost.
| cheriot wrote:
| Akamai looks at Cloudflare's valuation with envy (and some
| contempt because Akamai has the larger edge network).
| Definitely agree Linode was purchased as a growth
| opportunity.
| jacooper wrote:
| Cloudflare's valuation is for sure because its free
| therefore everyone uses it(~20% of the internet).
|
| If Akamai was also free and as accessible as Cloudflare,
| they would start to grow for sure.
| esskay wrote:
| Out of curiosity what is a new competitor going to offer that
| the existing well established ones like DigitaoOcean, Vultr,
| etc arent?
| ubermonkey wrote:
| I would much rather deal with a human-scale firm than a
| giant one. That alone would get me to switch, even at
| greater cost.
| mekster wrote:
| DigitalOcean doesn't have much Asian presence, Linode's
| backup doesn't support non default file system like zfs, DO
| / Linode don't have HDD block storage like Vultr does
| (which I think is a hidden gem you can't realize until you
| go add one as it's not mentioned in the pricing page) that
| can be a very nice option to throw backup data in at
| 40GB/$.
|
| Vultr's CPU isn't as good as the other 2 by the look of
| sysbench, even for high frequency types.
|
| Those 3 vendors only have 2 or 3 backup slots which can be
| kind of unreliable if you didn't notice data loss over a
| weekend and weekly backup can be too far apart.
|
| Hetzner is almost perfect if they have a bit more location.
| Very cheap, very high performing, daily backup with plenty
| of slots but no HDD block storage.
|
| Sometimes Amazon Lightsail can be a decent option as they
| have 7 daily backup slots, free bandwidth is comparable to
| the others and prices aren't bad except their CPU suck.
|
| Small things but none are perfect.
| webmobdev wrote:
| Switch to Pair - https://www.pair.com/ - they are older than
| Linode and have a great reputation. (PS: However, I am not
| sure if they have been acquired by another company now. The
| "old" Pair had a sterling reputation, and I can vouch for
| that as a former customer.)
| duskwuff wrote:
| Pair is absolutely not price-competitive. Pair's "Starter
| I" plan is $30/mo, for 1 vCPU, 2 GB RAM, 20 GB storage, and
| 500 GB transfer. Linode's most similar plan ("Linode 2 GB")
| is $10/mo, and includes 2 TB transfer.
| webmobdev wrote:
| Does look like they have hiked their prices. But it also
| depends on how you are making the comparison. For
| example, just a cursory look at both pricing shows me
| some major differences - Pair is offering "fully managed"
| VPS hosting which means the hosting providers employees
| will help you do most of the common hosting tasks (and
| provide better customer support) unlike "unmanaged"
| hosting where you have to do everything yourself with the
| tools provided. Is the Linode plan you cited also similar
| or for "unmanaged" hosting (which is what Linode was
| famous for)? Note that Linode (Akamia) also makes a
| distinction between "shared" vCPU vs "dedicated" vCPU and
| prices it differently. Don't know how Pair looks at that.
| The point is that the pricing variation can be justified
| by the services being offered. I admire and respect
| Akamai and also like Linode a lot. But I also recognize
| that Akamai is big B2B company offering enterprise
| services and small or independent developers will always
| be an after thought for them.
| ubermonkey wrote:
| I'm paying for a few small Linux VMs. Pair has nothing all
| price competitive.
|
| They're also now part of a giant media company (!), so
| yeah, pass.
| webmobdev wrote:
| Look up "managed" hosting vs "unmanaged" hosting - that's
| one explanation why their services are costlier.
| teknopaul wrote:
| I don't think many users will be affected by dhcp outage. Most
| linodes have fixed IPs.
|
| Mine don't seem to be affected
| simmons wrote:
| For Linode Ubuntu 22.04 installations, at least, the default
| seems to be DHCP, even if the IP address is fixed. And I
| certainly wouldn't have thought anything about this, since
| even when using fixed IP addresses, I'm in the habit of using
| DHCP to provide a central source of IP address truth for my
| networks, to ease the pain of the inevitable renumbering.
| jpmattia wrote:
| > _Most linodes have fixed IPs._
|
| We run some Windows instances, which were DHCP out of
| laziness. <facepalm> [edit: It's a PITA to fire up a Windows
| instance on Linode. Once working, switching out of DHCP is
| near the bottom of the todo list.]
| bombcar wrote:
| Most have DHCP-fixed IPs.
|
| The IP lease never changes but it's assigned by DHCP - check
| your settings (mine was in _/ etc/netplan/01-netcfg.yaml_
|
| dhcp4: yes
| mritzmann wrote:
| > Most linodes have fixed IPs
|
| I would expect the opposite. Doesn't DHCP make more sense,
| for example, if the user creates a snapshot and restores it
| with a new IP? I don't use Lenode, so I can't judge, but I
| would expect DHCP to be default.
| kyrra wrote:
| Default created Debian based linodes use a static IP. How
| they assign you the IP is they have an initial network boot
| script that injects that static IP into your config.
|
| I learned this because I wanted to test image recovery of
| deleted linode. When I restored it, I got a different IP
| address from linode (the static IP assigned to me), and I
| couldn't connect to it. Using the web console I found the
| IP address was hard coded to the old one still.
| bogantech wrote:
| Maybe this changed recently but I created a few Debian
| instances this year, both were configured with DHCP by
| default
| ryandrake wrote:
| Just checked my tiny Nanode 1 GB image (Fremont) running
| Debian and it's indeed running dhclient. Still have
| access, but I need the instance this weekend for a
| virtual event. Kind of sucky timing for a potential
| outage, Linode.
| exikyut wrote:
| That being said, maybe check if you have `dhclient`,
| `dhcpcd`, `systemd-resolved`, or (less likely) `dnsmasq` or
| `bind` running (htop / ps aux).
|
| If you do, your instance may fall over after its DHCP lease
| expires and it then presumably fails to acquire a new one and
| goes all 169.254.x.x on you.
|
| The possible counterpoint / information-vacuum bit here is
| that I don't know what the DHCP lease time is, so the expiry
| may already have happened.
|
| IOW, apparently this started 4h ago, and so the last leases
| were presumably issued 4h ago, and maybe the lease length is
| 1h, so everything's already as dead as it's going to get, so
| _maybe_ if you 're not already dead, you're fine.
| bongobingo1 wrote:
| The Akamai acquisition has had me kind of stressed to be
| honest. Here in Oceania there aren't a whole lot of established
| "cloud" hosts to pick from that host locally.
|
| Linode is fantastic and I have loved them for 10+ years.
| Digital Ocean doesn't host here, nearest is Singapore. Vultr
| does but I have had poor experiences with them. The other
| options are AWS and Google which are potentially unbounded
| expenses which is hard to sell to clients that already see
| $12/m (aud) as a cost sink, no matter what the server+service
| will save them in seat time.
|
| There are of course local hosts but they're either oversold
| cpanel hosts or comparatively expensive (3-10x the price for
| the same hardware).
|
| I guess it just feeds from my existential dread of being a
| small fish where if any one of these upstream providers decide
| "oh, no we flagged that account because of [black box reason],
| sorry!", you can be out a lively hood with no real recourse.
|
| Guess it just kinda sucks to be at what feels like the end of
| the long tail of infrastructure.
| prng2021 wrote:
| Just curious, what caused the poor experience with Vultr?
| texasbigdata wrote:
| Not OP but just a guess, vultr was founded 2014 and linode
| is 2003. Obviously not a proxy for quality, but it's one
| data point as to potentially why.
| doctor_eval wrote:
| Was going to ask the same. I've used vultr for years and
| found them rock solid, and have had good support when
| needed.
|
| They also just opened a DC in Melbourne.
| ht85 wrote:
| Unrelated to the Linode thing, but in 15 years building and
| selling SaaS products, I've encountered a number of customers
| who nitpick prices or try to haggle. I'm not talking about
| non-profits with micro budgets, rather very profitable
| businesses.
|
| Nowadays, whenever someone goes there, I just change my usual
| lighthearted tone to a serious one and say "Yeah, our prices
| are firm, if that's outside of your budget it's better we
| just move on". They 180 real fast. Every. Single. Time. Then
| they never mention prices again.
| dsr_ wrote:
| In the procedure list, someone included the line "Try to
| get a discount".
|
| When it works, it works.
|
| When it fails, it didn't work, but the next step is not
| "Quit" but "Continue".
| flakeoil wrote:
| These guys are in Australia amongst other locations:
| https://www.ovhcloud.com/
| sofixa wrote:
| A caveat with them is that they're really trying to be
| price competitive, and can sometimes go overboard (like
| design their own custom datacenters and servers with
| watercooling, and skipping some things like fire
| suppression systems, and using cheap material such as
| plywood). Having a backup with a different provider,
| somewhere else, is always a good idea.
| dsr_ wrote:
| If you're very clever, watercooling is also the fire
| suppression system!
|
| Possibly too clever.
| ErneX wrote:
| Downtime for me lasted almost 4 hours.
| linodecustomerm wrote:
| Linode London has been experiencing "greater than normal"
| intermittent packet loss to Akamai network which they use to peer
| with other player such as Cloudflare in LDN for nearly two days -
| before they peered with CF via Linx if I remember correctly. It
| isn't just isolated to UDP customers. Currently it's fairly
| stable so for most it's hard to notice but certainly
| uncomfortable how far they want to drag this problem on - see
| status.linode.com for how grim the LDN situation is - their
| support said we'll tell you in a few days what it was once we
| fixed it. i.e. it's not the rock solid of the last 7 years and
| their support are largely deaf to how unacceptable this is based
| on previous standards.
| j45 wrote:
| That's too bad. Which other vps hosts are folks using these days
| besides aws/azure/gc?
|
| If anyones thinking of switching to digitalocean, I'm currently
| switching linode from digitalocean who had an unforgivable,
| catastrophic loss of vm.
|
| Digitalocean also prevebts customers from downloading disk image
| backups offsite for any kind of adult DR. Linode lets you
| download backup images right from the admin console.
| mxuribe wrote:
| I was thinking on moving to Hetzner (from Digital Ocean) - now
| that they have/will have a data center in Virginia....but i
| have heard iffy things about hetzner (some people are happy,
| others not so much). I should clarify, i'm only thinking on
| moving due to costs (not because i have encountered any ops or
| support issues). The stuff that i host is not as essential, and
| could do with slightly lower costs. ;-)
| pbowyer wrote:
| I'm using UpCloud who are good, but a bit more expensive than
| Linode or DigitalOcean.
| jaachan wrote:
| Normally I'd agree, but... Have you seen today's status ?
| https://status.upcloud.com/ 6 hour downtime and no updates
| jacooper wrote:
| Hetzner looks good.
| j45 wrote:
| Thanks I'll revisit them. Had a lot of dedicated servers in
| my past and turned the self hosting part of my brain off
|
| I have also heard iWeb is or was a good choice for value.
| Montreal data centres can be a well kept secret.
| jacooper wrote:
| Hetzner's dedicated servers are unbeatable in pricing.
| Especially the ryzen 5 3600x version.
| 0xbadcafebee wrote:
| > an unforgivable, catastrophic loss of vm
|
| deets?
| samtho wrote:
| For what it's worth, I've been a Linode customer for about 12
| years and have never had a problem. This current issue does not
| even affect me as my boxes are on static IPs. My setup consists
| of a number of Linode VPSes, a single Hetzner Box in Germany,
| and an S3 glacier bucket (the only Amazon service I use) for
| backups. It's probably not optimal, especially for the business
| case, but it works for all my hobby and side projects.
| j45 wrote:
| For sure.
|
| I've had linode for a very long time.
|
| Not leaving them because of how good their tech support is or
| was to help me early on in my career, but they can't be a
| point of failure either.
|
| I'll just get to see how easily a hybrid cloud might be
| orchestrated like yours. Things are drastically easier to
| learn to setup and maintain like you have said compared to 5
| or 10 years ago.
|
| Docker is docker, and traefik is traefik.
| windhamdavid wrote:
| Mine in Atlanta are up. Already had the 'Network Helper' enabled.
| nimbius wrote:
| to whomever is at the top of the automation repo commit log, i
| wish you a safe and speedy recovery. Enjoy this friday, that it
| may succor a distant memory of fridays past when the latest build
| didnt break production.
| simmons wrote:
| Arrgh! :( Just this week I needed to do some emergency
| rearranging of personal infrastructure, and moved one nameserver
| to Linode Fremont, while the other nameserver is in Linode
| Dallas. I was fully aware that I needed a more diverse long-term
| solution, but I figured "What are the chances that both Fremont
| and Dallas will go down in the next week or two?"
|
| Sorry, folks. I hate that my dangerous flirting with Murphy's Law
| is hurting other people.
| simmons wrote:
| For whatever it's worth, since it was a DHCP problem, I was
| able to manually configure the static IPs and get the hosts
| back up. The last incoming connections logged before the outage
| were around ~ 6:24:00 am MDT (about one hour ago). I guess
| that's when my server needed to renew the DHCP lease and
| couldn't.
| markstos wrote:
| I found my Linode-hosted site was down this morning. I logged
| into the Linode console, rebooted and it came back up.
|
| Generally I've been very happy with Linode over several years.
| chasd00 wrote:
| I can still login to the backend and to my vm through Lish but my
| websites are down. (I'm in Dallas)
|
| edit: followed the instructions in this link mentioned upthread
| and rebooted, that solved it for me
| https://www.linode.com/docs/guides/network-helper/#individua...
| [deleted]
| patrickdavey wrote:
| We ended up rebooting our server, that brought everything back
| up again.
| bombcar wrote:
| If it's DHCP and you know your IP (probably because it's your A
| record) try manually forcing the IP.
| mcculley wrote:
| Amusingly, I am trying to log in to the dashboard to reboot my
| nodes, but it requires a verification code which was sent to my
| email, which is hosted on Linode.
| velcrovan wrote:
| Consider signing up to a service like MXGuardian that your MX
| records can point to, and that then forward email to your
| server. This will give you an "emergency inbox" that you can
| use in cases like this (might also help with spam depending on
| how you are currently dealing with that).
| arjvik wrote:
| What happens when MXGuardian goes down?
| boplicity wrote:
| FYI: Linode says that most people can fix this by enabling
| "Network Helper." You'll then likely need to reboot your server.
| It worked for me!
|
| https://www.linode.com/docs/guides/network-helper/#individua...
| zzzeek wrote:
| this.... replaces the use of DHCP with a static file rewriter
| on boot, OK, why is this recommended in general, should we
| expect DHCP to have ongoing issues that their "network helper"
| would not ?
| bombcar wrote:
| For the vast majority of people, a Linode is just a virtual
| server that sits on a static IP like a fat toad.
|
| This network helper just assigns that static IP instead of
| assuming DHCP will find it.
|
| You could turn it off later I guess.
| TheNewsIsHere wrote:
| Yeah, we've only reallocated an IP address once (in the
| past week, as it happens), but that was an exception. Our
| servers are redeployable and backed up, but they're not
| really ephemeral. So they just have their IP "forever". I
| think the oldest virtual server we've had was deleted when
| it was almost 3 years old.
| apocalyptic0n3 wrote:
| Most of my Linodes are still accessible, but this has worked
| for the two that went down so far.
| Ologn wrote:
| My Newark node was down for over six hours, this is how I fixed
| it. Uptime on the VPS had been over a year before this (don't
| remember why it was rebooted a year ago). Like others here
| said, I did not greet the news of Akamai buying them with joy,
| and things like this starting seem to bare my initial
| impressions out.
| chasd00 wrote:
| Thank you! this worked for me too. I was totally down, went in
| and enabled Auto Configure Networking per your link, and then
| rebooted.
| cmg wrote:
| I had 6 servers out of about 50 go down, all in Dallas/Atlanta
| even though most servers are in Newark. IPv6 connectivity seemed
| to remain up for all of them, though.
| tttd wrote:
| tiffanyh wrote:
| Off topic: any updates on Linode's Bare Metal service?
|
| It's been 2-years since it's been announced as "Coming Soon" and
| I haven't seen any updates.
|
| https://web.archive.org/web/20201116142627/https://www.linod...
| joecot wrote:
| I gave up. I submitted an inquiry on their sales form for
| dedicated and got no response, I contacted their support and
| got "we'll let you know." They've at least added nvme storage
| for instances, but in at least their Dallas data center the
| iops suck. In Newark it's a bit better.
|
| If you've got a bunch of servers at a linode data center and
| want bare metal ones, I ended up just looking at what other
| hosts are peered with their nearest exchange. Our servers are
| in Dallas and Vultr, Equinix via IBM Cloud, and Servermania all
| had peering to them through a local exchange at infomart or
| nearby. We ended up moving database servers to those bare metal
| services and connecting over wan to our linode ones. Latency
| less than 5ms.
|
| Linode also contacted us to ask why our monthly usage halved.
| We told them, they offered no alternatives.
| tiffanyh wrote:
| Same experience, no response. I tried contacting them both a
| year ago and about 6 months ago and never heard back.
|
| I went with Vultr (as you mentioned as well) and couldn't be
| happier.
| dboreham wrote:
| A bare metal data point, since I just went through the process
| of finding an alternative to Packet.net since they were
| acquired by Equinix : try Hivelocity. My experience with them
| has been so far so good.
| Gelob wrote:
| Limestone networks (http://lstn.net) offers great bare metal
| services. I used to work there 7-8 years ago and they are a no
| frills, no bs company. They will give you a good deal also :)
| ehayes wrote:
| https://www.linode.com/products/bare-metal/ --still prefaced
| with "coming soon" but it looks like you can "create your
| build".
___________________________________________________________________
(page generated 2022-11-04 23:02 UTC)