[HN Gopher] Sourcehut network outage post-mortem
___________________________________________________________________
Sourcehut network outage post-mortem
Author : ggpsv
Score : 195 points
Date : 2024-01-19 15:55 UTC (7 hours ago)
(HTM) web link (sourcehut.org)
(TXT) w3m dump (sourcehut.org)
| kyrra wrote:
| I've been using Sourcehut for a couple years now. One thing this
| outage taught me about the service that I didn't know is that
| Mercurial (hg) is community maintained:
|
| > We also did our best with hg.sr.ht, but it is community
| maintained
|
| It looks like git.sr.ht is hosted on OVH in France, while
| hg.sr.ht is hosted on High5! in the Netherlands.
|
| It's not entirely clear to me how this affects their product
| roadmap or support, but definitely good to know.
| skywal_l wrote:
| > _It looks like git.sr.ht is hosted on OVH in France_
|
| They explain it here:
|
| > _However, we found that OVH's anti-DDoS protections were
| likely suitable: they are effective, and their cost is
| amortized across all OVH users, and therefore of marginal cost
| to us. To this end the network solution we deployed involved
| setting up an OVH box to NAT traffic through OVH's DDoS-
| resistant network and direct it to our (secret) production
| subnet in AMS_
| pelagicAustral wrote:
| That's such an odd choice for this type of infra. I've had
| horrendous experiences with OVH in the past and what even
| worse, terrible customer service. Yes, this was about 8 years
| ago, and not with France based metal, but still...
|
| Being that this is Drew, I wouldn't be shocked to know that
| this provider choice has more to do with an anti-
| establishment manifesto than any practicality. Then again, I
| might be wrong.
| DistractionRect wrote:
| Well, it's certainly better than their last provider who
| they couldn't reach during a critical time, and still
| cannot reasonably communicate with.
|
| They can at least reach and reason with OVH, as mentioned
| when they got flagged as an out bound DDoS.
|
| > Being that this is Drew, I wouldn't be shocked to know
| that this provider choice has more to do with a anti-
| establishment manifesto than any practicality
|
| I feel this is a pretty unfair barb considering one of
| their first moves was reaching out to Cloudflare.
| Unfortunately, non-http traffic + the need for tls
| termination on their own servers (pretty sure cloudflare
| calls this Keyless SSL) squarely lands them as an
| enterprise customer w/ enterprise pricing.
|
| Drew probably had already entered into agreements with OVH
| when cloudflare came back around, and we don't have insight
| on the terms or period for which Cloudflare's second offer
| was good for.
| twic wrote:
| This also came as a surprise to me! Not only that but:
|
| > restoring service was delayed until we could get the
| community maintainer, Ludovic Chabant, online to help
|
| Maintainer, singular!
|
| The only reason i use Sourcehut, and the main reason i pay for
| it, is because i stubbornly still use Mercurial, and want
| first-class support for it. With the utmost of respect to M.
| Chabant, that is not exactly first-class.
| nequo wrote:
| > With the utmost of respect to M. Chabant, that is not
| exactly first-class.
|
| It would appear that Ludovic Chabant is working full-time at
| Epic Games. He is unlikely to have the capacity to be on call
| for Sourcehut.
| vanderZwan wrote:
| I think the complaint was aimed at Sourcehut leaning on a
| sole volunteer for this service, not at Ludovic Chabant
| drewdevault wrote:
| hg.sr.ht is operated by SourceHut, but the software is
| maintained by the community. Ludovic is the primary maintainer
| and various other Mercurial users participate in its
| development.
| moberley wrote:
| For me there a bit of a language barrier with the terminology.
| After reading the sentences about hg.sr.ht and community
| maintenance it seems that some notable meaning is being
| conveyed about what that means for the operation of the service
| but its one I'm not smart enough to understand.
|
| I appreciate the service though so I hope the differences
| between maintained and operated doesn't mean anything in the
| long term.
| hypeatei wrote:
| > As unfortunate as these events were, we welcome opportunities
| to stress-test our emergency procedures;
|
| This right here is invaluable and something you only get from
| experience. Planning and theory only get you so far.
|
| I extend this thinking to deploying large infrastructure changes
| you've never done before - you can only plan so much before
| pulling the trigger and just doing it and seeing what happens.
| shrubble wrote:
| Would have liked to know what the difference was in response
| between Cogent and Level3. Did only Cogent respond at all, or was
| Cogent the one handling all their IPv4 space?
| zeroclicks wrote:
| Seems only Cogent was advertising their routes. Once Cogent
| blackholed their prefixes, there'd be no way to reach their
| services via the internet.
| scandox wrote:
| I'm still left not quite certain what would happen if they were
| hit with another L3 DDOS tomorrow.
|
| That said I'm very happy to use Sourcehut and I think they'll
| overcome these challenges over time. They seem to have the
| staying power.
| frakkingcylons wrote:
| They're on OVH now and should have protection from it by virtue
| of being on their network now.
|
| > However, we found that OVH's anti-DDoS protections were
| likely suitable: they are effective, and their cost is
| amortized across all OVH users, and therefore of marginal cost
| to us. To this end the network solution we deployed involved
| setting up an OVH box to NAT traffic through OVH's DDoS-
| resistant network and direct it to our (secret) production
| subnet in AMS; this met our needs for end-to-end encryption as
| well as service over arbitrary TCP protocols.
| treesknees wrote:
| I'd consider it mostly protected, because no their servers
| are not on OVH, just a single box performing front-facing
| NAT/proxy essentially. The attacker now just needs to find
| the "secret" production subnet and attack it directly instead
| of through the front-facing NAT addresses.
| makeworld wrote:
| My reading is that OVH would handle it.
| mrled wrote:
| I am really curious if the DDOS tried to follow them to the new
| infra and failed to cause an outage or not. Apparently the
| perpetrator noticed when they got Cogent to narrow the null
| route, but the blog post notes they still can't access the
| original subnet in that datacenter. Are they still trying to
| knock Sourcehut offline? Is the DDOS still pointing at now
| deprecated infra for some reason?
| caboteria wrote:
| > At about 06:30 UTC the following morning, the DDoS
| escalated and broadened its targets to include other parts of
| our PHL subnet. In response, our colocation provider null
| routed our subnet once again. This subnet has been
| unreachable ever since.
| mrled wrote:
| Right, that's expanding to the rest of the subnet in their
| old DC. They've since migrated to the new DC with new
| countermeasures. Did the DDOS follow and the
| countermeasures are working? Or if it didn't follow, why
| not?
|
| There's also the question of whether the DDOS is still even
| trying the old infrastructure. The post says it's
| unreachable, but that would be true if the null route
| hadn't been removed yet.
| drewdevault wrote:
| Yes, the DDoS followed us to networks with
| countermeasures, and yes, the countermeasures worked. We
| don't want to disclose too much about that, though.
| wpm wrote:
| When they switched DNS over to point to the AMS datacenter,
| the DDOS attack followed it until it got smacked down by the
| OVH NAT.
| OsrsNeedsf2P wrote:
| Unrelated, but TIL Drew DeVault is one of the SourceHut
| maintainers. His blog[0] is strongly opinionated and always an
| informative read.
|
| [0] https://drewdevault.com/
| mortallywounded wrote:
| Maintainer? More like creator.
| otachack wrote:
| ?Por que no los dos?
| trevyn wrote:
| Opinionated? More like firebrand.
| j4yav wrote:
| Arent open source project maintainers typically the creators?
| matthews2 wrote:
| You don't need to agree with all of his opinions to use
| SourceHut :)
|
| I'm not a big fan of some of his hot takes, but I still respect
| him and trust him with my data.
| sneak wrote:
| The reason I don't use srht is because of his opinions about
| product development (of srht itself), not his personal
| opinions.
|
| Social/collaboration features are explicitly deprioritized by
| design; I think this is a natural consequence of srht being
| built by and for lone wolf developers. GitHub and Gitea
| (which is basically a github clone) seem much more geared
| toward collaboration by groups, something most small-time
| f/oss developers don't need.
|
| Also, the emphasis on email and irc is bad, imo. The web won
| because it is better. A lot of the anti-web stuff is just
| tradition.
| tslocum wrote:
| As someone who was there in the early days, who joined the
| chorus of people warning Drew about the effects of such a
| policy, I just want to say that Forgejo is a treat to self-
| host and use. Gitea is now open-core, and its future is
| unclear.
|
| https://forgejo.org
| mroche wrote:
| This really comes down to the intended workflow. By
| design, SourceHut aims to provide the Linux kernel
| development model to a wider audience (with extra
| features beyond mail and Git). It is a very different
| collaboration model than the likes of GitHub and its
| peers. I summarize the comparison of the two as "to each
| their own"; I'm okay with both models and see the merits
| of both, but my preferences and willingness or ability to
| work with a given model won't always line up with
| contributors.
|
| I also self-host Forgejo in my homelab and really enjoy
| it.
| zufallsheld wrote:
| The only mention I can find that gitea is open core comes
| from forgejo. Do you have some kind of proof that there
| are parts of gitea that are not MIT licensed?
| johnmaguire wrote:
| Gitea Ltd's stance seems to be that it does "custom
| development" support contracts.[0] It may be a matter of
| perspective whether you consider this "open-core" or
| "contract work."
|
| See also their clarifications on Gitea the company[1]:
|
| > Gitea Ltd. will be open to building special versions
| for special clients and will contribute any features back
| to the main repository when possible
|
| This was in a followup to the original announcement.[2]
|
| Forgejo (i.e. Codeberg, a FOSS non-profit) maintains that
| the project should be led by the community, not a
| company[3]:
|
| > Sadly, Gitea Ltd broke that trust by a lack of
| transparency: its existence was kept a secret during
| months. After the initial announcement, Gitea Ltd
| published another blog post but it was still vague and
| there has been no other communication since. Who are the
| Gitea Ltd shareholders? Who, among the Gitea maintainers,
| are employees of Gitea Ltd?
|
| [0] https://about.gitea.com/pricing/
|
| [1] https://blog.gitea.com/a-message-from-lunny-on-gitea-
| ltd.-an...
|
| [2] https://blog.gitea.com/open-source-sustainment/
|
| [3] https://blog.codeberg.org/codeberg-launches-
| forgejo.html
| tarxvf wrote:
| That the social and communication tools they prefer are not
| the tools you prefer does not mean they are asocial.
| kstrauser wrote:
| That's so true, but I agree with sneak here (did I just
| write that?). If my code is on GitHub or GitLab or Gitea
| or whatever, and I want to work on it with a friend, I
| can invite them to join me on a website using a workflow
| similar to 1,000 other not-source-code-related
| collaboration tools. It's damn near impossible to talk
| someone into joining an email-based process unless that's
| something they've already been doing elsewhere. Look at
| the git-send-email docs[0] which talk about configuring
| SMTP auth. Followup question from the new person I'd be
| trying to rope in: "I dunno, my work uses Outlook. What's
| SMTP?"
|
| If someone contended that SourceHut optimizes for devs
| who've been writing Linux kernel code for 25 years, so
| you weed out all the newbs and can get the hardened
| veterans involved in your project, I could buy that. I'd
| disagree that it's what _I 'd_ want for my project, but
| to each their own. I couldn't recommend it as an
| alternative to other services that require participants
| to know how to use a web browser.
|
| [0]https://git-scm.com/docs/git-send-email
| myaccountonhn wrote:
| Once you learn the git-send-email flow, it is a lot
| better, especially for distributed development.
|
| With the PR flow, people need to sign up to the website,
| create a fork, clone the repo, make their changes, go
| into a slow web ui etc. It mostly works because everyone
| is on Github. However, even that solution sucks if you
| are having a polyrepo setup and need to make changes in
| many places.
|
| For bazaar style development where you accept
| contributions from anyone and don't use Github, the email
| flow is so much faster and simpler. Yes, you need to set
| it up once. But the other day I contributed to a open
| source project that was self-hosted, and it's amazing
| that I just can clone the repo, make my changes, commit
| and then git-send-email, bam done. Had I needed to sign
| up and create an account, set up a fork, I probably
| wouldn't have bothered because it was a small
| contribution. However no need to register to a website,
| no need to click through a slow ui, no need to create a
| fork, it reduces the ritual to make contributions by
| quite a lot, given that you've set it up.
|
| There is also https://git-send-email.io/ which provides a
| nice tutorial for people.
|
| I am glad that there is a good alternative that supports
| this flow, because I think it is superior. There are a
| ton of alternatives if you want the PR flow (Gitlab,
| Gitea, Github, Codeberg).
| avgcorrection wrote:
| I've done the email workflow for a bit. I'll say this
| much: it might be comparable to configuring a power
| editor vs. using some powerful and ready-to-go IDE. You
| can set up things how you like and the preferences of
| everyone else doesn't really matter. You can also just
| edit anything because it's fast and there is probably a
| good enough configuration for all kinds of languages and
| modes.
|
| But in some ways it isn't. Like any fool (like me) can
| just get some Emacs configuration for free from others.
| There doesn't seem to be that kind of sharing for all the
| fiddly little things you need to do with git-send-email
| and the rest. All I've heard so far is that, oh yeah I
| usually deal with this specific issue by running some
| Perl scripts that I wrote eight years ago and that I've
| been nurturing ever since. But it wouldn't be very useful
| for you because it's very, very idiosyncratic. Might not
| even work outside Debian and my Apt state...
| kstrauser wrote:
| I adore Gitea. 99% of the stuff I keep there is private
| code, where Gitea is basically an SSH-able Git remote.
| However, I occasionally want to share a project with a
| friend, and then it's trivially easy to invite them to
| collaborate with me using the same infrastructure I was
| already using.
|
| Minus that last part, I'd just stick with plain Git. It's
| everything I need for my own personal, only-for-me
| projects.
| xigoi wrote:
| Everyone has an e-mail account. That means if you want to
| contribute to a project on SourceHut, you don't need to
| create an account there.
|
| Also, I hate when I'm looking for useful forks of something
| on GitHub and have to sift through tens of useless forks
| that were created just to be able to submit a pull request.
| avgcorrection wrote:
| Are they deprioritized (spelling dunno)? Or are they just
| different in a way which you judge as being not-conducive
| to collaboration? (I mean you mention mailing lists.)
|
| There's not really much need for a "forge" without
| collaboration. I wouldn't pay the price of SourceHut just
| so that I can fetch and whatever between my machines.
| That's like a pricey sneaker net.
| gray_-_wolf wrote:
| I stopped paying for sourcehut because his opinions are
| relevant here since he bans types of projects based on them.
| You never know when another restriction will be added.
| cornstalks wrote:
| If you're talking about banning cryptocurrency and
| blockchain projects, personally that earned some favor in
| my eyes. I'm happy to use and pay for a service that
| doesn't contribute to that blight.
|
| For the curious, the terms are here:
| https://man.sr.ht/terms.md#permissible-use
| gray_-_wolf wrote:
| I also do not like "crypto", but I do not think this type
| of restriction is great on a _paid_ service. Maybe, maybe
| it could be argued for public repositories. Or if it was
| free. But like, why does Drew DeVault care that I would
| have a private repository with "explicit sexual
| content"? On an account I _pay_ for?
|
| And even if you agree with the current set of
| restrictions, are you sure it will not be further
| expanded? I am not.
| eesmith wrote:
| > why does Drew DeVault care that I would have a private
| repository with "explicit sexual content"?
|
| For the same reason GitHub does? GitHub's AUP at
| https://docs.github.com/en/site-policy/acceptable-use-
| polici... says:
|
| "We do not allow content or activity on GitHub that: ...
| is sexually obscene or relates to sexual exploitation or
| abuse, including of minors".
|
| Atlassian's AUP at
| https://www.atlassian.com/legal/acceptable-use-policy
| says "Inappropriate content" includes "Posting,
| uploading, sharing, submitting, or otherwise providing
| content that ... Is deceptive, fraudulent, illegal,
| obscene, defamatory, libelous, threatening, harmful to
| minors, pornographic (including child pornography, which
| we will remove and report to law enforcement, including
| the National Center for Missing and Exploited Children),
| indecent, harassing, hateful"?
|
| GitLab's AUP at
| https://handbook.gitlab.com/handbook/legal/acceptable-
| use-po... says "unacceptable use of our services [which]
| applies to all users of all GitLab services including
| those on the Free, Premium, and Ultimate GitLab tiers"
| mean "you must not: Create, upload, submit, execute,
| transmit, or host anything that ... is vulgar, obscene,
| or pornographic, or gratuitously depicts or glorifies
| violence."
|
| Now, there are differences between "explicit sexual
| content", "sexually obscene" and "pornographic", but if
| you are worried about possible further expansion, you
| shouldn't use any of these code hosting services.
| gray_-_wolf wrote:
| The reason does not seem to be stated at the provided
| link. If you know the reason (which your message seems to
| imply), could you please share it?
| cornstalks wrote:
| It's hard to find a payment processor for pornographic
| providers. Existing payment processors are likely to stop
| supporting you if you become a porn provider.
| Additionally, there are branding risks in being
| associated with adult content. There's also more legal
| scrutiny involved, and it's outright illegal in some
| jurisdictions.
|
| A simple Google search on the topic should be
| educational.
| eesmith wrote:
| I was conjecturing it was the same reason as the other
| hosting providers, not saying that was the same or that I
| had special insight.
|
| Instead, I was pointing out that since all the providers
| I looked at have essentially the same restriction, you
| likely shouldn't use any of them. Certainly there are a
| lot of people who use GitHub despite having no guarantee
| the ToS won't be more restrictive in the future.
|
| Sourcehut's ToS is certainly not exceptional in that
| regard, so really you are objecting to essentially every
| 3rd party code hosting provider, yes?
|
| Or is there one you had in mind where you aren't
| concerned about further expansion?
| farhaven wrote:
| > On an account I pay for?
|
| On an account that you pay _Drew_ for. Do you also
| complain because someone renting you a garage doesn't
| want you running a strip club out of there?
| mrmanner wrote:
| I like when people bring their values when they do
| business. Especially when those values are more than
| "make money", and expressed in more ways than product
| design.
| cinntaile wrote:
| This also keeps me away from sourcehut. I like everything
| else about it but this is a deal breaker.
| beanjuiceII wrote:
| same for me
| dijit wrote:
| You know, it's fair not to support the service on that
| principle,
|
| However, Sourcehut _is actually_ FOSS software.
|
| IE: if _you_ wanted to run one of their banned things, you
| could, just on your own hardware.
|
| It's fine, in my opinion, to moderate your services if
| people have an escape hatch to get out of your service if
| you require them to move along.
|
| This is a far cry from services such as GitHub, or even
| Gitlab (with their open core) as transferring to your own
| system is actually possible, though not without some
| relative pain.
|
| I don't like crypto projects, so of course I am biased
| here. But if you like free speech then there's not many
| options and I think sr.ht is the best one (especially if
| you plan to self-host).
|
| GitHub is _well_ known to be controlling of speech and even
| championed some measures that affected the entire industry,
| and as others have mentioned they have restricted projects
| on a relatively arbitrary basis. Sometimes even due to
| geographic region.
| jraph wrote:
| I find this refreshing that someone does business according
| to their values, not allowing money to buy everything.
|
| I believe generally letting things happen as long as money
| comes without any regards to values behind the things might
| have been detrimental.
| rezmason wrote:
| Wild speculation: maybe the attacker's motive was to usher
| specific Sourcehut hosted repositories to the jurisdiction of the
| EU.
| ploum wrote:
| On a more serious note, I'm really wondering about the
| motivations. I see the following hypothesis:
|
| 1) Test/demonstration of a DDOS against a random target.
|
| 2) Attack against a project hosted on sourcehut to make it
| unavailable (there was even the speculation of disabling a
| master repository so an end-user could not check that his own
| local version was the correct one, thus using it with a
| security hole or a trojan)
|
| 3) Attack against a page hosted on sourcehut (I joke that
| someone wrote "Putin = Fag" on his sourcehut hosted blog).
|
| 4) What else ?
| svieira wrote:
| Looks like Cloudflare did change their minds later and offered to
| mitigate the attack _pro bono_ :
|
| > Following our initial quote from CloudFlare, we understand that
| some CloudFlare employees undertook a grassroots effort
| internally to convince the leadership to sponsor our needs, and
| eventually CloudFlare came back to us with an offer to sponsor
| our services for us free of charge. This was a very generous
| offer for which we are very appreciative; in the end we did not
| take them up on it as we had made substantial inroads towards an
| alternative solution by that time. I have had my reservations
| about CloudFlare in the past, but they were there for us in a
| time of need and I am grateful for that.
| zeroclicks wrote:
| Typical "corporate pricing"--they offer a really high price
| they'll expect you'll negotiate downwards to something
| reasonable. The Sourcehut negotiators probably never dealt with
| this kind of "sales model" before.
|
| That said, what will happen when more companies publish their
| experiences with "enterprise sales"? There's an article from
| HEY[1] about how broken the sales process is. To get a quote,
| you normally have to endure 2 or 3 zoom calls before the price
| is unveiled.
|
| There's probably room for an innovator to fix all of this.
|
| 1: https://world.hey.com/dhh/the-only-thing-worse-than-cloud-
| pr...
| drewdevault wrote:
| We did negotiate them down a bit but we didn't feel that we
| could come to an agreement within our budget and decided to
| move on. Apparently this was an excellent negotiation tactic
| because they came back with an offer of $0!
| tetha wrote:
| I find it somewhat chilling how their original colo left them to
| hang and dry.
|
| Maybe I'm weird, but I'd consider colo to be a closer cooperation
| than just renting some virtual servers from wherever. And just
| getting told "Yupp, your null-routed. No, we can't give you
| access for specific sources over a different path. Get fucked" -
| or, in fact, not getting told that - is ... one of our ex-hosters
| was like that.
|
| And as a service provider, I have strong feelings about the
| customer service there.
|
| Maybe I don't know big infrastructures, but this just leaves me
| with a weird feeling in my guts.
|
| But hell. Make sure to give your engineers - and their family -
| something. After some hell-weeks, we've given people some budget
| to do something fun with their family, because the company had to
| take so much private time during those weeks.
| downrightmike wrote:
| being null routed is really the only thing they can do. Then
| then undid it and they had to do it again. This wasn't a
| standard DDOS attack, which they normally handle just fine.
|
| Good coverage of the event: Security Now! Podcast
| https://www.youtube.com/watch?v=ehfV7cRLkFE
| aidenn0 wrote:
| About 29minutes in is where it picks up after reading
| verbatim the status report from Drew.
| jabart wrote:
| Depends on the contract and the attack size. Sometimes the DC
| has to pick all it's other customers over trying to handle a
| DDoS for one. Our DC had an issue where packets over 1492bytes
| were being dropped in Chicago by one transit provider and that
| took 3 hours to make the call to drop them.
| vander_elst wrote:
| Mostly curious about the k8s plans. From some past posts it seems
| that the team was strongly against employing containerization
| [0]. However, it seems something changed. If anyone has more info
| about this if love to hear more.
|
| [0] https://news.ycombinator.com/item?id=23030489
| doublerabbit wrote:
| > This outcome was unacceptable
|
| No it wasn't. The outcome is due to major networks being shite.
| Not accommodating newer technologies and gate keeping services to
| resolve DDoS attacks.
|
| All major network upstreams could do so much more to make the net
| more reliable and resilient to small ISP. Myself included.
|
| peer neutral networking, not having tons upon tons of e-waste
| prone to botnet behaviour, it wouldn't be like this.
___________________________________________________________________
(page generated 2024-01-19 23:01 UTC)