[HN Gopher] GitHub Incident
___________________________________________________________________
GitHub Incident
Author : wgjordan
Score : 208 points
Date : 2022-03-22 15:56 UTC (7 hours ago)
(HTM) web link (www.githubstatus.com)
(TXT) w3m dump (www.githubstatus.com)
| KronisLV wrote:
| Someone who decided to self-host their own source control system
| here, figured i'd comment about some of my experience due to some
| people mentioning it as an alternative. I've personally run
| GitLab in my own homelab for a few years but after some updates
| going wrong, i eventually moved over to Gitea, Nexus and Drone,
| also self-hosted: https://blog.kronis.dev/articles/goodbye-
| gitlab-hello-gitea-...
|
| GitLab is a large and complicated piece of software, which is a
| double edged sword: you do get an integrated issue tracking
| solution, Docker Registry with integrated access controls, as
| well as a pretty decent CI solution, which is great. At the same
| time, however, there's a chance for one of those subsystems to
| start misbehaving and bring down the whole thing (e.g. Omnibus
| install with default configuration for where to store container
| images --> running out of space; especially relevant if you don't
| find a good way to clean up old images), which may or may not be
| harder to solve than having 3 separate integrated systems.
|
| Also, GitLab is pretty resource hungry and migrating to another
| solution actually improved the performance of certain
| systems/parts of functionality bunches: Gitea is really fast,
| Drone CI is about on par with GitLab CI and Nexus is a bit faster
| than GitLab Registry, however also eats too much RAM (e.g. the
| same problem within both Ruby and Java, Nexus seems to fail to
| start up below having approx. 1 GB of RAM, GitLab needed at least
| 2 - 3 GB of RAM).
|
| However, running GitLab at work is still a pretty good idea, as
| long as keeping the instance up and running, as well as updated
| is someone else's responsibility and you can focus on just being
| the end user. At the same time, however, the update paths can be
| problematic sometimes, as was the case with my container based
| Omnibus installation that was multiple major versions out of
| date.
|
| But for personal use? The Gitea, Nexus and Drone setup is
| perfectly sufficient, alongside maybe some issue tracking
| software (be it Kanboard, OpenProject or something else). In
| short, to the folks who are considering self-hosting in the
| comments: you can definitely try it out and have it be a learning
| experience, maybe on an affordable VPS or a device you have
| laying around!
|
| Furthermore, there's nothing really preventing you from mirroring
| your repos over and keeping them in multiple systems, alongside
| any backups that you might be making!
| EMIRELADERO wrote:
| It would be nice if GitHub started operating the way GitLab does.
| Free community edition (open-source) and paid proprietary
| Enterprise Edition (source-available). Developement being
| completely public (employees push their commits, do code review
| and work on issue boards on the official GitLab repo[1] which is
| publicly available)
|
| [1] https://gitlab.com/gitlab-org/gitlab
| mbesto wrote:
| Can we please stop posting major system incidents and status
| pages for major cloud/SaaS providers? What's the point?
|
| Post mortems - TOTALLY fine. But just linking to a specific
| incident is IMHO not valuable to HN and just takes unwarranted
| frontpage space.
| bobthechef wrote:
| disgruntledphd2 wrote:
| I think it's so people realise it's not just them.
| mbesto wrote:
| Makes no sense. If you're having issues wouldn't you just go
| directly to <insert service here>'s status page first? Why HN
| first?
| 0x0000000 wrote:
| Because status pages lie.
| astrosi wrote:
| In many cases the status page of a service is an unreliable
| indicator (e.g. AWS's) so I find HN / Twitter to be a much
| better signal that an incident that affects many people is
| happening.
| crent wrote:
| Status pages are just not reliable unfortunately.
| aerovistae wrote:
| I find this valuable.
|
| -Sometimes for conversations about alternatives
|
| -Sometimes for conversations about the product's stability
| history
|
| -Sometimes just for solidarity in frustration
| mbesto wrote:
| > -Sometimes for conversations about alternatives
|
| So then submit a post mortem and drive the discussion there.
| Or alternatively post a "Ask HN" if you're legitimately
| considering alternatives.
|
| > -Sometimes for conversations about the product's stability
| history
|
| Which are readily available on those provider's status pages.
|
| > -Sometimes just for solidarity in frustration
|
| Isn't that what Twitter is for? Why should the frontpage be
| littered with <insert provider here>
|
| My point is this - let's say GH goes down 8x over 2 weeks for
| 10 minutes each. Should there be 8 posts on the frontpage
| about GH going down each time? No, there should be ONE,
| otherwise the valuable discussions you're referring to get
| lost.
| thieving_magpie wrote:
| I can understand your viewpoint but just to offer a counter I
| _always_ check HN for major outage news when it comes to github
| /AWS/GCP/Azure. There have often been comments with bits of
| information as to the cause and remediation efforts.
| mbesto wrote:
| > bits of information as to the cause and remediation
| efforts.
|
| That's what post mortems are for. Happy for people to submit
| those.
|
| Also, why are cause and remediation efforts important to you?
| If you rely on the service and it goes out, what more are you
| going to learn here that would help you? Genuine question.
|
| > I always check HN for major outage news when it comes to
| github/AWS/GCP/Azure
|
| Why not just check those status pages? Or check Twitter? HN
| seems like a weird place to check for it...
| chrisfosterelli wrote:
| HN is frequently discussing outages before the status page
| updates...
|
| (I recognize not in this case, but I certainly noticed this
| outage and checked HN before the status page was updated)
| thieving_magpie wrote:
| > Also, why are cause and remediation efforts important to
| you?
|
| Because I have to provide information to others in my
| company about why X is not doing Y. These bits of
| information help.
|
| > If you rely on the service and it goes out, what more are
| you going to learn here that would help you?
|
| Roughly the same answer - though I'm also just actively
| curious what's going on as well in the moment.
|
| >Why not just check those status pages?
|
| I do. HN often has more information.
|
| >Or check Twitter? HN seems like a weird place to check for
| it...
|
| I don't want to.
| dtech wrote:
| The point is notifying the active community here and discussing
| the reason(s) and impact of the outage, or in GH's case their
| regularity I guess
| mbesto wrote:
| So then write a post that says "Ask HN: What is going on GH's
| being out X many of days". Otherwise you get 6 posts across
| weeks with the same discussion.
| [deleted]
| bob1029 wrote:
| If there was no virtual public space in which to tar and
| feather shameful technical practices (or at least review them
| honestly), they would probably be even more prevalent.
|
| I also see these as an excellent learning opportunity.
| Typically, it is a lesson in managing complexity.
| intunderflow wrote:
| With all the recent incidents have they broken their SLAs yet?
| ProAm wrote:
| A SAAS company never breaks an SLA. SLA's are written by the
| legal team. Uptime is maintained by the engineering team. They
| aren't even in the same building.
| AlexandrB wrote:
| Yup. After all this isn't downtime, its "degraded
| performance":
|
| > Git Operations is experiencing degraded performance. We are
| continuing to investigate.
| abluecloud wrote:
| Github broken. CircleCI broken. I'm going for a coffee.
| trutannus wrote:
| Its the snow-day of the tech world.
| FpUser wrote:
| I just got bunch of errors - error: failed to push some refs to
| 'github.com'. Luckily after a while the error disappeared.
|
| I was going nuts. For my own products I use gitea and host repos
| myself. Some clients of mine however use github so in this case I
| have no choice
| relieferator wrote:
| went to downdetector first but this is betta
| kiloreux wrote:
| This is getting too much.I wonder what effects this will have on
| the reputation of Github. If I am blocked by Github every few
| days and can't properly operate. I will start looking for a more
| reliable alternative. And I am just a solo dev that pays $4, what
| happens to companies for whom time is worth a lot of money?
| Pretty disappointing.
| chockchocschoir wrote:
| How do you as a solo developer get blocked on anything related
| to GitHub? Are you doing all project planning on GitHub without
| any other copies elsewhere? Most if not everything you do on
| GitHub could be replicated locally, one way or another.
|
| I can understand the problem for larger development teams, as a
| lot of communication and workflows can happen via GitHub Pull
| Requests or similar.
| kayodelycaon wrote:
| Personally, I can't deploy if GitHub is down. CI pulls from
| GitHub. One of my projects includes libraries as submodules,
| with GitHub as the origin. The deployment configuration has
| everything cloning from GitHub.
| coldpie wrote:
| Try building any Rust project while GitHub is down.
| chockchocschoir wrote:
| That's only applicable if you use crates directly from
| GitHub repositories right? I've been sitting and
| developing+building a Rust project for the last +10 hours
| (except the minutes when I was here arguing on HN) and
| never hit any issues, but I'm using crates either directly
| from disk (cloned from GitHub initially) or from crates.io.
| coldpie wrote:
| I could be wrong, but try a fresh build with a clean
| CARGO_HOME (or maybe just a cargo update?). It seems to
| query a lot if you don't have a full cache.
| steveklabnik wrote:
| While it's true that Cargo will attempt to check on some
| things, you can pass flags to tell it to not bother.
| Specifically, https://doc.rust-
| lang.org/stable/cargo/commands/cargo.html#o...
| tyingq wrote:
| Maybe using their CI/CD?
| chockchocschoir wrote:
| CI/CD should you perform tasks that you could execute on
| any machine, besides the deployment, which should work
| differently locally VS via CI service. But it should still
| be possible to deploy without CI, otherwise you set
| yourself up for being unable to deploy when you really need
| it.
| mhitza wrote:
| > But it should still be possible to deploy without CI,
|
| Solo? Yes. In a team? Less often just because you might
| have credentials stored in secrets that are not shared
| with the entire team for security reasons.
|
| In larger corps it's easier to wait it out that go
| through the trouble of escalating permission requests.
|
| But I do get your point. Personally for deployments I
| have Ansible playbooks that are invoked via
| straighforward calls which can be done locally.
| DogLover_ wrote:
| Github-actions is not created to encourage that though.
| It uses syntax that only works on Github. Look at their
| workflow-examples and you will see.
| chockchocschoir wrote:
| Don't you define things in your repository first (usually
| via a `Makefile`) and then call those things in the CI
| environment? Or are you building things differently in
| the CI environment compared to the local environment?
|
| I mean, if you're working on a project that has tests,
| code coverage and binary builds setup, you usually have a
| `Makefile` or `package.json` or whatever to run your
| scripts, and your CI setup just calls those very same
| scripts but in their environment (sometimes with
| different arguments/environment variables).
|
| Not sure why it would be different for GitHub Actions.
| It's certainly how I use it day-to-day.
| singingboyo wrote:
| In my case, at least, GH actions is the only place with
| all the secrets necessary to deploy my (small) webapp.
| Sure, I can generate alternative tokens and pull some
| things out of 1password, but it'd be time consuming.
| (Also, changing things like JWT secrets is less than
| ideal.)
|
| There's also just the number of things it checks. jest
| runs, lint/build, e2e and acceptance tests, 2 docker
| builds pushed into ghcr, and then ansible to deploy. It's
| mildly error-prone to do myself, especially the docker
| and ansible steps because that's where the secrets come
| in.
|
| So sure, it CAN be done manually, but the entire point of
| CI/CD is to do everything consistently, repeatedly, and
| without the risk of manual error. It took me hours to
| figure things out the first time. Why would I want to
| risk doing things manually now?
| deckard1 wrote:
| no that's not how it works at all. The "actions" are
| proprietary to GitHub and hosted _on_ GitHub. People
| create custom actions and allow others to reuse them.
| Everything is hooked in to GitHub via their proprietary
| yaml config.
|
| > Not sure why it would be different for GitHub Actions.
|
| because vendor lock-in. GitHub doesn't want to make it
| easy for you to switch.
| greggman3 wrote:
| You can run github actions offline
|
| https://github.com/nektos/act
|
| I don't think github is trying to create lock-in, I think
| rather they were trying to make a way to easily share
| actions (not sure what other CIs systems are designed to
| have an ecosystem of publicly shared actions). The
| actions are public and therefore easy to make something
| that interprets them.
|
| I can only guess at some point there will be a push for
| CIs to converge on some "actions" standard, maybe?
| tyingq wrote:
| That's neat, though it is a 3rd party trying to replicate
| it.
|
| Gitlab's open source runner supports parsing the ci yml
| and running a job locally, presumably using the same code
| as the platform, like: gitlab-runner exec
| docker some-job-name
|
| Though they would both suffer on any dependencies on
| platform hosted environment vars, secrets managers, etc.
| Arnavion wrote:
| If you have vendor lockin with GitHub Actions, it's
| because you chose to do it to yourself. Nothing prevents
| you from using only the `run` action to run a shell
| script so that everything that CI does can also be done
| on your dev machine.
|
| Both my personal projects and my $dayjob repositories
| have every test, etc triggered via `make test` or
| `test.sh`, then the GitHub Actions workflow YAML just
| `run`s it. Secrets also work fine - the makefile / shell
| script expects them to be defined as env vars, so the
| developer running them locally just needs to define those
| env vars regardless of how they obtained the secrets.
|
| https://news.ycombinator.com/item?id=25104253
| DogLover_ wrote:
| Sure, but then I guess you always install the
| dependencies from scratch, can't utilize cache, and won't
| have a dependency graph as well.
|
| Parallelising your build/deployment will likely also be
| harder to do.
| Arnavion wrote:
| >Sure, but then I guess you always install the
| dependencies from scratch
|
| Sure? On your dev machines the dependencies are already
| installed. On GHA VMs the network is fast enough that
| installing deps is not slow. What's the problem?
|
| And if you really have a problem, presumably you have
| some master tarball / container image that your devs use
| to set up their dev machines because installing your deps
| is so complicated, so scp / pull that in your script?
|
| >can't utilize cache
|
| A cache is necessary for CI VMs that are cleaned for
| every run. When building locally your dev machine already
| has everything the cache would have.
|
| >and won't have a dependency graph as well. >
| >Parallelising your build/deployment will likely also be
| harder to do.
|
| You realize the two things shell scripts are good at is
| running commands either series or parallel, exactly how
| you want them? Instead of learning a brand new DSL to be
| able to do `if` and `&`, you just write `if` and `&`.
| This is already covered in the comment I linked.
| misnome wrote:
| Are there any CI/CD metasystems that aren't vendor
| locked-in? GitHub/even GitLab, Jenkins (freestyle and
| scripted)/Azure/Travis.... all vendor-specific, as far as
| I know.
|
| Sure, have all the heavyweight stuff in separate scripts
| that are just called, but platform specification/multiple
| platform builds/specifics of caching/secret
| handling/deployment handling are always different. Some
| tools (e.g. codecov) do abstract over some platforms, but
| not all, and the GitHub Actions model of "here is a
| literally pre-prepared step in your pipeline" can be
| pretty appealing.
|
| It's literally, pick your poison, and resign yourself to
| reimplementation if you ever need to switch platforms.
| kiloreux wrote:
| Yeah, but theory and practice are very different things.
| If CI always worked for me and my experience was great
| with it. I will not invest time to do independent CI for
| everything. Hence depending on Github Actions. Clearly
| not the best approach, but it's about being pragmatic
| also.
| chockchocschoir wrote:
| What do you mean "independent CI"? You start with a
| `Makefile` with the commands for testing, building and so
| on, and CI is just calling those commands but on another
| instances, that likely gets triggered for each
| commit/tag/pull request/whatever.
|
| When you first setup CI, you don't actually just develop
| for the CI environment first, that's the second step
| after you know how to perform the same things locally. So
| not sure why there would be "independent CI for
| everything".
| tyingq wrote:
| It's easy enough to unintentionally box yourself into a
| hard 3rd party dependency. Like the github secrets store,
| conditional logic or triggers in the ci.yml files,
| pipeline driven semantic versioning, etc.
| kiloreux wrote:
| I am a solo dev, but I work in a company and now have been
| working on pushing something to test it on Github actions. If
| it was something different, maybe. But I understand how I
| might have miscommunicated that.
| misnome wrote:
| I'm also in a company, but today am building up something
| on Github Actions.
|
| Every single time I start doing development on Actions,
| GitHub goes down or actions start failing. I'm not sure if
| this is saying something about me, or GitHub.
| bilekas wrote:
| > Most if not everything you do on GitHub could be replicated
| locally, one way or another
|
| That's not really the point though.. If you have services in
| github etc, why would you replicate all of them locally too
| for unplanned downtime ?
| chockchocschoir wrote:
| Well, usually you figure out how to run your tests, check
| code coverage and build your binaries locally first, then
| figuring out how to get it to work in the CI environment,
| not the other way around. So the replication has already
| been done, but in the direction of local environment >
| remote environment.
| bilekas wrote:
| I don't disagree, really local environments should be
| able to manage dev and most daily tasks, but large
| integration pipelines, test automations etc.
|
| I just don't feel the need to defend GitHub here for the
| reason of : "Well you should be able to manage locally
| anyway"
| hunter2_ wrote:
| This is quite similar to the somewhat philosophical
| question of having backups: if your job is to make sure
| your company has backups of their data, and you entrust a
| single vendor with that data because they take care of
| backups so you don't have to, are you really doing your
| job? Said another way: if you (or your immediate team) are
| not the party that turns one thing into two things, then
| there is still a SPOF in your wheelhouse, and the whole
| point is for that to not be the case.
| citizenpaul wrote:
| Since most executives dont actualky care if backups work
| for real just that they are not liable if the backups
| fail. Yes.
| kayodelycaon wrote:
| Maintaining an entire secondary deployment and CI system
| just seems unrealistic to me. I'd get so much flack about
| the extra spend and when the systems get out of sync and
| causes downtime, that investment ends up with net
| negative for all involved.
| misnome wrote:
| SPOF is something that you have to make trade-offs for
| and decide what you can cope with.. How much work to
| maintain a parallel infrastructure _just in case_ GitHub
| goes down for more than an hour once or twice a month?
| How many people actually are in the position to pivot
| their entire company processes around this failure case?
|
| It can't be "remove all SPOF or you aren't doing your
| job". How many offices are resilient to continue working
| if the building power goes out?
| hirako2000 wrote:
| 'Be what's next', or how to ruin the brand of a 7.5B acquisition.
| colinmhayes wrote:
| Microsoft has big plans for github. Maybe the brand is a bit
| tarnished by the time they realize those plans, but if they
| succeed in fully integrating the developer productivity stack
| into both the github app and site they'll see a very large
| amount of revenue.
| JonChesterfield wrote:
| What GitHub app? edit: turns out to be a flavour of vendor
| lock in, https://docs.github.com/en/developers/apps/getting-
| started-w...
| colinmhayes wrote:
| Visual studio will be rebranded as the github app.
| grumple wrote:
| Was wondering about multi-remote git workflow and found this:
| https://jigarius.com/blog/multiple-git-remote-repositories
|
| This gets tougher with CI/CD though. How urgent is it to run your
| tests or deploy right now as opposed to when the outage is
| resolved later, or tomorrow? In emergencies I think you need ways
| to deploy without the CD pipeline (for those that can't
| currently).
|
| In my experience there's always more work to do and downtime is a
| temporary inconvenience. Of course... my customers would be
| blowing up our phones and emails if we were down, threatening to
| quit, etc. I assume Github doesn't have quite the same level of
| pain when they have downtime.
| zthrowaway wrote:
| Anyone think the warnings issued of Russia's cyber attacks has
| any correlation here?
| dgellow wrote:
| No
| fishnchips wrote:
| I'd personally correlate it with their Azure migration. Either
| that, or recent launches. Stability aside, they've been
| shipping a lot of changes recently. And more often than not,
| changes lead to outages.
| JonChesterfield wrote:
| Seems likely that migrating to Azure is related to being
| acquired by Microsoft. I blame the new ownership for the
| decreased stability.
| pm90 wrote:
| A migration of that scale is difficult to execute on. I
| wouldn't be so quick to blame the whole Azure platform,
| unless it really is something in Azure that's responsible
| for these outages.
| zthrowaway wrote:
| Yeah I could see that... with just them going to Azure alone
| lol. Azure was very unstable at a previous company I was at.
| time_to_smile wrote:
| I would expect any serious Russian cyber attack would more
| closely resemble "polonium tea" in the sense that it would be:
| incredibly destructive, quite obviously from a state actor, and
| yet officially not connectable to Russia.
|
| Strategically cyber attacks are the most effective when they
| damage your adversary heavily, and yet at the same time make it
| so that politically retaliation looks like aggression since
| there is no unquestionable link between the actor and the
| action.
|
| Consider Stuxnet as an example. Widely understood to be a joint
| US/Israeli cyber attack on Iran, and at the same time difficult
| to retaliate against.
|
| While annoying, these outages we've been seeing are hardly
| doing any serious harm. Very little money is ultimately lost
| because of them, and it's not remotely clear that this is a
| cyber attack. So if they were cyber attacks it wouldn't make
| much sense. Not saying that it's impossible they are, but it
| seems there are many other more plausible explainations.
| bob1029 wrote:
| Ah the semi-daily github fuckup.
|
| For this one, I found myself rage-screaming at my monitor as
| their issue markdown preview API was timing out ~80% of the time.
|
| Github enterprise is probably happening for us this year. I don't
| give a shit about the new features in the public build. I just
| want it stable and working at this point.
| eutropia wrote:
| As someone who is responsible for CI/CD...
|
| welp.
|
| I'm thankful that at least I'm not on security investigating the
| severity of the okta hack.
| 5e92cb50239222b wrote:
| This has been happening pretty much every day for a long time.
| Maybe this will push more people towards some decentralization. A
| self-hosted Gitea instance my organization has been using had
| exactly zero downtime in the past year. None. Same with our CI
| system. Yes, I know, GitHub is 10999 times larger than our puny
| Gitea instance, and that's why they're having issues, but why
| should I care when one is working, and the other one isn't?
| skeeter2020 wrote:
| >> A self-hosted Gitea instance my organization has been using
| had exactly zero downtime in the past year. None. Same with our
| CI system.
|
| And when they do go down, how long will they be unavailable?
| When this happens (and it will happen) please post here so we
| can say "why are you running infrastructure yourself?"
| chousuke wrote:
| Depends on your setup. For a simple git server on AWS I might
| have it on a single-node ASG so that when something
| terminates the instance, the ASG notices and launches a new
| one. The initialization scripts would then attach and mount a
| pre-existing EBS volume and start services.
|
| Very simple to set up, and as long as your data volume is not
| lost (which can happen, but is very rare), such a system
| would recover from a host failure by itself faster than most
| people could react to it going down. In the unlikely case
| that you lose the EBS volume, you'd just recover from
| snapshots manually.
| thrashh wrote:
| I once worked at a place that self hosted a lot and they had
| fail-over, backups and restores dialed in
|
| If something like Git went down -- which it never -- it was a
| few clicks away from coming back
|
| Working with competent people is a blast
| x86_64Ubuntu wrote:
| You also need an organization that will spend the money to
| make that happen. Many of us would like to have such
| systems where we work, and are capable enough to implement
| them, but the Finance people say 'NO'.
| johnisgood wrote:
| Infra I have been running myself only goes down when I reboot
| INTENTIONALLY.
| dijit wrote:
| I'm honestly sick of this conversation.
|
| Have we as an industry forgot how to make systems? that's
| pretty telling if so.
|
| Why are people so utterly afraid of running their own
| systems.
|
| If a SaaS platform goes down it's followed by "#hugops;
| running large systems is hard!", when someone says "Well, if
| it's hard, why not break it into smaller pieces and run it
| myself" it's met directly with "Do you think you can do
| better!?!".
|
| Honestly, probably, maybe not. Why does it matter?
|
| Those two things cannot be true simultaneously. You cannot
| say "running big systems is hard" _and_ "they can run a big
| system better than you can run a small system".
|
| I'm not going to fall into the trap of arguing points like
| the fact that if you own the data you can have your own D/R &
| backup strategies and the fact you can run maintenance's on
| your own time (hell: you _can_ run maintenance 's to migrate
| things around, which is more than github can do, which is not
| an indictment of them, just the nature of being a hosted
| service).
|
| Honestly, I miss sysadmins. People who were not afraid of
| servers, even if they said "no" sometimes. Because, you know,
| being afraid of servers and hosting things is just pathetic.
| sblom wrote:
| > Those two things cannot be true simultaneously. You
| cannot say "running big systems is hard" and "they can run
| a big system better than you can run a small system".
|
| They can absolutely be true simultaneously. In fact, I
| can't name a logic system where those two statements
| comprise a contradiction. They are logically completely
| unrelated to each other.
| KolenCh wrote:
| I think you misunderstand the intention. The parent comment
| is comparing uptime to their own system and declare
| superiority, and the one you reply to is asking the the
| opposite -- the duration of downtime.
|
| Sure you can build your own infrastructure but don't
| pretend that's superior. Each has its pros and cons.
|
| The last line probably is making fun of their reaction --
| any person as dogmatic as the parent commenter would
| comment someone with their own infrastructure going down
| with that comment.
| PradeetPatel wrote:
| From an organisational perspective, outsourcing
| infrastructure is the safest and cost effective way to meet
| Regulatory & Compliance requirements.
|
| When you run your own servers, not only do you need to
| maintain them, you are also responsible for their
| certification and getting them audited.
|
| As a manager in a medium sized company, I think it is
| essential to identify what "needs building" and what could
| be bought to ensure the needs of stakeholders are met in a
| timely manner.
| dijit wrote:
| Heh, time was "compliance" was the reason to host your
| own services. It is a _lot_ easier to be compliant with
| many things if you 're hosting your own services.
|
| Back when I was working in eCommerce (PCI-DSS Tier 1,
| Cardholder data on premises) it was basically impossible
| to use hosted services.
|
| As then, it is now: "Compliance" is just enough nebulous
| of a word to prevent critical thinking.
| PradeetPatel wrote:
| I wouldn't call compliance a deterrence to critical
| thinking. The end goal for any business is to make a
| profit, if an option exists that complies with industry
| standards, which also happens to be the most inexpensive
| one in terms of resourcing, it would be foolish not to
| take it.
| manicdee wrote:
| It only takes one month long outage of an international
| undersea cable to negate all the benefit of SaaS when that
| SaaS is hosted in or heavily depends on other services hosted
| in the USA or Europe.
|
| DNS goes down for a particular dot com? BGP hijinks? Backhoe
| through the major fibre serving your workplace?
|
| All of these are going to happen and will lead to downtime
| out of your control or the control of the SaaS provider.
|
| Thankfully git is designed to be decentralised and developers
| can continue to work even if they have to use paper books as
| reference material and not Stack Overflow.
| Nextgrid wrote:
| Very likely to be lower than GitHub/etc, because it's a much
| simpler system that can be reinstalled from scratch in 30
| minutes by an experienced admin on brand new hardware if
| needed.
| 5e92cb50239222b wrote:
| Less than you've already wasted waiting for GitHub to come up
| from the dead in the last year alone. If you're insinuating
| that we'll have to restore everything from scratch, don't
| worry, the machine is being continuously replicated and
| sysadmins are trained to restore everything chop-chop.
| jchw wrote:
| I ran a GitLab instance for a company pretty much solo and it
| never wound up going down outside of when I was upgrading it,
| that I can recall. The thing about that is, it's a small
| scale instance. If an upgrade or change failed, I could
| literally delete everything and restore from backup in under
| an hour, and never really needed to do that (although to be
| clear, I did actually test that it was possible.)
|
| Now I'll tell you the real problem with it: it worked great
| for people who were located in the U.S., but it was terrible
| for certain remote employees due to increased latency, and
| the fix for that would be to set up a complicated replication
| setup that puts data closer to people not near the main
| instance.
|
| There's obviously other issues, but that is one that is
| probably commonly overlooked that most SaaS solutions can do
| better on.
| Jarwain wrote:
| What gitlab features do you utilize where latency actually
| is a problem?
|
| I currently manage an internal gitlab instance with some
| remote coworkers, but I haven't heard any complaints about
| latency. Although tbf, I feel like we've barely scratched
| the surface of what gitlab supports
| jchw wrote:
| The most impacted user had ~500ms latency to the server,
| which is enough that even pageloads are bound to feel
| slow (especially because Gitlab itself is not necessarily
| the fastest thing in the world sometimes.) And if I am
| remembering correctly, I think that was basically it: the
| UI pageloads just had enough chained round trips that
| they were way too slow. In retrospect, there may have
| been some ways to work around this that we didn't end up
| exploring.
|
| When I moved to a new company, the GitLab instance was
| still up, and last I was aware, many years down the road,
| it is still in use. Feature-wise, we were very much
| satisfied, and the GitLab CI model with Docker executors
| worked very well for us.
| sytse wrote:
| If you have high latency to a GitLab server consider
| setting up a GitLab Geo server locally
| https://docs.gitlab.com/ee/administration/geo/ to improve
| things (this is a paid feature).
| jchw wrote:
| Looks like you got downvoted for some reason, but yes,
| this is exactly what I am referring to. (We _did_ pay for
| GitLab EE, so I think we had access to Geo.)
| scandox wrote:
| 5 years of a Gogs instance. Zero downtime. I'll come back to
| you when it goes down.
| ramesh31 wrote:
| does gogs plan to support any kind of CI integration like
| gh actions?
| scandox wrote:
| No it doesn't. I fully confess the feature set is
| extremely minimal. But the point here is that running
| your own infrastructure can make a lot of sense if you
| know what your precise requirements are (or are not).
| sreevisakh wrote:
| Gogs and gitea are designed to be a single-binary minimal
| source code hosting service. Additional services like CI
| within that binary is likely to bloat it up and make it
| less reliable. However, it's easy to integrate a CI
| service to it - like Woodpecker CI [1]. Woodpecker also
| has a minimalist approach like gogs. If you instead
| prefer a full package, then gitlab is the best solution.
|
| [1] https://woodpecker-
| ci.org/docs/administration/vcs/gogs
| aprdm wrote:
| I've been running gitlab for ~5y in companies with ~200 devs
| and haven't had a single downtime, it also takes very little
| work/effort.
| ghusto wrote:
| With something as simple as serving git for your own
| organisation, when you do get downtime it's going to be a
| case of restoring from backup and changing DNS. No need for
| complicated infrastructure, so bringing it back really is
| that simple.
|
| Worst case is losing commits between going down and backups,
| but even then it'll be minimal if you set your backup to
| simply push to a another origin.
|
| I pay for someone else to worry about most services, but git
| doesn't have to be one of them.
| munificent wrote:
| The worst case is discovering the hard way that your backup
| system isn't working and you haven't been diligent about
| spending the time to periodically test restores.
| dspillett wrote:
| That's why testing of backups should be at least
| partially automated. Though you will always find a point
| where the automation, and automated tests to make sure
| that is working, and so on, get to the point where making
| a human responsible for sanity checking is most resource
| effective.
| ghusto wrote:
| Pushing to multiple origins is not a hard thing.
| thefreeman wrote:
| If you think "serving git" is the only role github is
| serving for organizations you are probably not the target
| audience.
| ghusto wrote:
| I don't. Which is why I was explicit in the problem I'm
| targeting.
| candiddevmike wrote:
| Gitea is nice, what CI/CD system do you use with it? I used it
| in the past with Drone, and Drone left a lot to be desired
| (coming from/going back to GitHub actions, Drone's job syntax
| is a bit more complicated and focused on containers).
| ncrmro wrote:
| I've used Drone CI in the past with Gitea. I still prefer
| gitlabs CI tho
| 5e92cb50239222b wrote:
| Drone CI has been working fine for our purposes. I agree that
| it may not be the most feature-packed thing in the world (for
| example, caching support is missing completely), but there
| are lots of plugins and it's easy to write your own.
|
| We've been using exec and ssh runners when containers are not
| enough (for example, when you need to juggle a few VMs or
| build some complicated Docker image, and using docker-in-
| docker is not fun).
|
| https://docs.drone.io/pipeline/exec/overview/
|
| https://docs.drone.io/pipeline/ssh/overview/
|
| GitHub Actions are nice... when they're actually working.
| contravariant wrote:
| 10^999, really?
|
| Rule of thumb is that 10^100 of anything doesn't fit in the
| observable universe.
| rvz wrote:
| > A self-hosted Gitea instance my organization has been using
| had exactly zero downtime in the past year.
|
| > Yes, I know, GitHub is 10999 times larger than our puny Gitea
| instance, and that's why they're having issues
|
| Exactly. That is also my overall point. [0] It makes no sense
| to go _' all in'_ on GitHub and something goes down and
| everyone is stuck once again.
|
| [0] https://news.ycombinator.com/item?id=30711442
| freedomben wrote:
| Mind sharing some details on your install? Which database are
| you using (especially wondering about sqlite with gitea)? Are
| you using the docker install, or from package, or in k8s?
|
| No pressure, but I'm seriously considering standing one up but
| it would need to be production ready.
| qbasic_forever wrote:
| IMHO storage is the big issue to figure out with a production
| ready gitea installation. Gitea only supports the local
| filesystem for storing bare repo data and not anything like
| S3 or other distributed or highly available storage systems
| (it supports S3 only for git LFS data right now, it is a long
| awaited open issue to get the bare repos there too).
|
| So the big issue for any non-trivial or highly available
| setup is going to be how you get high availability for the
| local storage volume. There are tons of options with various
| tradeoffs and levels of complexity here--simple local disk
| RAID, distributed filesystems like gluster or ceph, etc. I
| think this is the real crux of getting a good gitea instance
| going.
|
| No matter what figure out and test a good backup and recovery
| solution!
| freedomben wrote:
| Ah interesting! I read somewhere that it uses the database
| for that, but I think they were mistaken.
|
| Thanks that's quite helpful. If we do it in prod we'll have
| to accept single-instance.
| qbasic_forever wrote:
| Yeah I forgot to mention too a lot of complexity goes
| away if you host it on a good IaaS platform. On AWS for
| example you could just mount an EBS volume as the bare
| repo root and use RDS for a database. Then you just have
| to deploy and manage an EC2 instance to host gitea's main
| process. It's more expensive to use those managed
| platforms but might be well worth the cost vs. doing it
| all yourself with a bare metal deployment of everything.
| 5e92cb50239222b wrote:
| It's a really boring setup, I'm unlikely to share anything of
| much value here.
|
| No k8s or anything like that. An Ubuntu LTS virtual machine
| on top of I don't know what hypervisor (probably Hyper-V).
| Gitea requires very few resources.
|
| Data is stored in PostgreSQL 11. Probably should be upgraded
| at some point, but it's working fine for now.
|
| A Drone CI instance runs on that same machine. It controls CI
| workers on a dozen other VMs.
|
| Everything is in Docker containers under docker-compose for
| two reasons: easy upgrades, and the ability to shove all data
| into a single directory.
|
| caddy for HTTPS.
|
| Sonatype Nexus for package repositories (caching upstream
| npm, nuget, maven, composer, and our own internal repos).
| This one is pretty heavy and will probably be moved to a
| separate machine at some point.
|
| Never had any issues with any of these in about three years
| we've been running this setup.
|
| I was a bit facetious with the "zero downtime" statement. It
| has about 30 seconds of downtime per month, but it is always
| planned and outside of working hours deep into the night.
|
| There's been zero seconds of _unplanned_ downtime.
|
| Basically you do: $ docker-compose pull
| $ docker-compose up -d
|
| and half a minute later it is running new versions. If
| anything is broken (it hasn't been yet), call the sysadmin
| guy, and a few minutes later he restores a VM snapshot.
|
| Backups are being done on the hypervisor's level. I don't
| know about that much.
| freedomben wrote:
| cool, thank you!
| raxxorrax wrote:
| Gitea is absolutely production ready, we use an older version
| by now and don't even think about it much because it just
| works so well. Exactly what you want in production
| (production = software should work more often).
|
| Actually not "vendoring" dependencies from github is very
| much not production ready in many cases.
| corobo wrote:
| Opposite anecdata my GitLab install absolutely self destructed
| itself at one point and I had to rebuild it myself, waste of a
| weekend
| ashildr wrote:
| > my GitLab install absolutely self destructed itself
|
| That's frightening, we've been using it for multiple years
| now. It's running fine and the only short downtime is every
| few weeks for the update.
|
| What happened to your instance?
| corobo wrote:
| More fairly to GitLab it was the underlying server that
| broke initially. The issue from there was that I couldn't
| restore from the backup as the newly installed server was
| also a newer version of GitLab
|
| Tried restoring it to the older version to run through the
| upgrade path and the process didn't want to work. Wish I
| could be more specific here but I gave up on it and decided
| it'd just be quicker to whip up a script to recreate all
| the groups/projects/CICD in a new install and push repos
| from backups
|
| Don't let your GitLab server get outdated and don't
| consider it a valid backup unless it was taken with the
| latest version to avoid the same
|
| Also it wasn't relevant here but make sure you're backing
| up the etc files too, while that wasn't an issue for me it
| could easily trip you over in a worst case scenario (
| https://docs.gitlab.com/charts/backup-
| restore/backup.html#ba... )
|
| Not a complaint, it's just one of the hats that needs
| wearing sometimes, would have been easier on me if someone
| else was handling it though :P
| chockchocschoir wrote:
| It's not about being larger, it's about doing changes. GitHub
| used to be pretty stable before, when they didn't really do any
| (user facing) changes. It stood still in terms of features, but
| was also pretty stable overall.
|
| Compared to GitHub, how often does your Gitea instance go
| through code changes? How many major features have been
| switched on in that instance during the past year without any
| hiccups?
| newswasboring wrote:
| An infrastructure company focusing on feature churn just
| sounds wrong.
| woah wrote:
| What? Github is a social networking company.
| newswasboring wrote:
| I'm a bit dumb about these things on text, but did you
| drop an "/s" somewhere? I understand it has sort of a
| social networking layer on top, but that is not its main
| function right? This would be like calling a calculator
| app a social networking app because it allows me to share
| my results :D.
| chockchocschoir wrote:
| GitHub has always branded itself as a "social
| collaboration platform" rather than anything else (well,
| first "git hosting" but secondly the social stuff).
|
| Here is a early version of their landing page from 2008
| (the year they launched): https://web.archive.org/web/200
| 81111061111/http://github.com...
|
| Notice the logo says "Social code hosting" and the
| messaging of the page is mostly around popular
| repositories, collaborative features and other social
| elements.
| dogleash wrote:
| GitHub's feature churn is irrelevant to the comparison unless
| those features are used in parent poster's workflow.
| chockchocschoir wrote:
| Really? No, comparing the uptime of two services (actually,
| one service and one open source self-hosted service) it's
| relevant to consider _why_ one would have more downtime
| than another, even if they don't use those features. Even
| if the parent posters workflow doesn't use the
| new/deprecated features, they will still be affected by the
| downtime GitHub have monthly.
| Nextgrid wrote:
| I don't think anyone is overlooking the reasons why one
| might have more downtime than the other.
|
| The problem is that ultimately the downtime itself
| matters and not the reason, and if you don't need any of
| the features that GitHub offers, then the self-hosted
| route is a better option.
| falcolas wrote:
| > if you don't need any of the features that GitHub
| offers
|
| In particular, all of the new features whose addition
| causes instability.
| Nextgrid wrote:
| I think it's the development process (as in moving things
| around and potentially introducing bugs) than the new
| features per-se.
| falcolas wrote:
| With all due respect to the developers, it would not
| speak well for GitHub if they can't maintain stability
| while refactoring. There's a lot of testing processes and
| CI processes explicitly around making those actions safe.
|
| New features breaking is a lot more understandable - even
| expected - than regressions and refactoring failures.
| Deukhoofd wrote:
| I run Gitea myself as well. I generally throw in a new
| release once a month or so, never had any real hiccups.
| rolph wrote:
| it sounds like you dont make code changes unless they are
| required for function.
| Deukhoofd wrote:
| Also for security reasons, and because some features are
| nice to have. But you're right in that I'm definitely not
| randomly updating sometimes. When I do update I basically
| just back up the database, throw in a new Gitea binary,
| and restart the service. As the releases have been fairly
| well tested in general, this hasn't given me issues so
| far.
| chockchocschoir wrote:
| Again, how much it actually change across those releases?
| GitHub most likely are going through more changes on a
| weekly basis (if not daily) than what Gitea does on a
| monthly basis.
| Deukhoofd wrote:
| The major releases generally change quite a lot in one
| go, for example:
|
| https://blog.gitea.io/2022/02/gitea-1.16.0-and-1.16.1-rel
| eas...
|
| https://blog.gitea.io/2021/08/gitea-1.15.0-is-released/
|
| But I'm sure Github changes a lot more. However, the
| question is if those changes are worth the instability.
| For my workflow I just want my repos to be accessible,
| have my source, issues, and wiki accessible, and of
| course my basic git operations working. If that core
| functionality is actually not working at times, I'd
| definitely consider changing the git provider.
| freedomben wrote:
| I tried to look at the Gitea repo to see how fast they get
| commits, but github is 500ing :-(
|
| https://github.com/go-gitea/gitea
| chockchocschoir wrote:
| Rest assure, a open source project with ~100 contributors
| (whereas maybe 10% regularly contributes) has less activity
| and changes than a multi-international company focusing a
| SaaS with "1000 to 5000" employees worldwide.
| judge2020 wrote:
| For perspective, GitHub has 4,000 employees who list them
| as their employer on LinkedIn and GitLab has 1,700.
| qbasic_forever wrote:
| Git is distributed, you should be able to completely detach
| from your remote and still work effectively. I think a lot of
| big projects get too lazy about good, reproducible build
| environments and just rely on centralized CI to do everything.
| gmueckl wrote:
| If you haven't noticed, the whole point of GitHub is to
| infest git-using environments with so many centralized
| features that user's get trapped using that service. Seems
| like issues and actions are doing that job nicely!
| MattGaiser wrote:
| GitHub isn't just a repo in many cases. It is project
| management software.
| uwuemu wrote:
| lolspace wrote:
| Thanks! Just had problem
| ksnll wrote:
| Cannot really rely on Github lately. Wondering if they have some
| sort of post mortem for the incidents of last month. I would like
| to see why we should trust it to not go down again.
| chockchocschoir wrote:
| > I would like to see why we should trust it to not go down
| again.
|
| You shouldn't ever trust anything to never go down again. It's
| just inevitable that services go down from time to time, you
| can't change that unless we have a major breakthrough in
| software engineering where somehow people stop making mistakes.
|
| So instead of figuring out if you should "trust it to go down
| again or not", assume it will at one point in the future and
| plan accordingly. This is how you build resilient systems.
| arnonejoe wrote:
| can't open a PR right now.
| yuuta wrote:
| Microsoft SRE always sucks. GitHub is just another victim.
| ahmadrosid wrote:
| Today, I saw a beta timeline feature. Could it be this incident
| related to this release?
| thadjo wrote:
| pretty wild how these outages are basically just tolerated now.
| folks just get some coffee refresh HN a dozen times and then
| resume work when it comes back online. no huge uproar, no exodus
| from GH
| dogleash wrote:
| It was a foreseeable inevitability that once SaaS took over
| from on-prem that the quality would degrade to what we were
| getting from on-prem.
|
| ...but if you said anything you were just interrupting people
| playing with their new toys.
|
| The shrug whatever coffee break while another department fixes
| things is the same now as it was back then.
| corobo wrote:
| Everyone with projects small enough to kneejerk move around
| migrated to GitLab when Microsoft bought GitHub
| rvz wrote:
| From [0]
|
| > Until the next time GitHub goes down again (hopefully that
| won't be in another month's time).
|
| Oh dear. [0] Never expected it to be this soon or that bad.
| Expected it to be down in a few weeks at least. Last time this
| happened was _just_ 5 days ago. [1]
|
| Yet I see complaints here of GitHub being unreliable as predicted
| and even some considering to eventually self-host despite paying
| $4 a month even.
|
| I don't think it make sense to go _' all in'_ on GitHub, does it?
|
| [0] https://news.ycombinator.com/item?id=30714123
|
| [1] https://news.ycombinator.com/item?id=30711442
| AtNightWeCode wrote:
| What surprises me with many of the outages from big companies is
| that it seems like they have a bad setup when it comes to zone
| redundancy. Fact is many of the smaller companies I worked with
| are much better at this. Facebook, Fastly, Github. How can
| companies like that fail at such basic concepts? Really mind
| boggling. There must be something fundamentally flawed in their
| development process.
| benmanns wrote:
| Perhaps they've all gotten big enough to run their own DNS?
| duckmasterflex wrote:
| higeorge13 wrote:
| Instead of releasing unnecessary social-like stuff they should
| have focused on their core git service. Seriously, what's their
| uptime? Do they even have a number somewhere? Actions must be
| down every second day or so.
| teaearlgraycold wrote:
| Has anyone made a good SaaS product for self-hosted Github
| Actions runners? I'd gladly pay for beefier VMs from a 3rd party
| provider. Actually self-hosting is a pain in the ass because
| Github gives you only the runner software but none of the
| containerization/stateless wrapping I'd expect. Without some
| containers and scripting you'll get permanent changes to your
| VM's filesytem as you run actions.
|
| And the VMs GitHub provides are far too low spec to run Cypress
| in. There's no option to pay for more resources.
| tokumei wrote:
| It seems incidents like this with the big providers are becoming
| a lot more common. Wonder why that is
| nrvn wrote:
| Github is a platform that comes with all platform-related risks,
| i.e. operational risk of having the whole development and SDLC
| lifecycle blocked, creating a vendor lock due to using some
| unique github features, inability to rely on third-party
| dependencies hosted on github.
|
| There are a couple of strategies one could rely on to mitigate
| the risk of github downtime:
|
| - create a map of features between github and competitors, find
| the common denominator and risk accept anything that is unique to
| github but viral for your business/product;
|
| - maintain a primary self hosted instance of
| gitea/gitlab/sourcehut and use github as a mirror; - use github
| as a primary platform, maintain mirror with public gitlab and
| switch to gitlab in case of outage;
|
| - if github is used beyond its primary purpose(git hosting) put
| some efforts to maintain the same features in
| gitlab/gitea/sourcehut, i.e. use CI features of both platoforms
| and push releases and artifacts to both of them so that end users
| have the choice.
|
| - separate concerns and not go all in with github as a platform
| for the whole SDLC and project management and instead use
| different tools/platforms for different purposes.
|
| Yes, it takes more time for the initial setup but it quickly pays
| back.
|
| The above applies to those cases where your products need to be
| hosted publicly.
| judge2020 wrote:
| It seems the pre-MS GitHub's approach of "we have maybe 1 new
| feature a year in the pursuit of keeping our duct-taped service
| afloat" might have been the right call. Nowadays tons of new
| features are pushed out every week[0].
|
| 0: https://github.blog/changelog
| darknavi wrote:
| I get what you're saying but I cracked up at this "GitHub is
| releasing too many features and breaking things!" because when
| I clicked your link the latest entry is:
|
| > Diffing Adobe Photoshop documents is no longer supported
| raxxorrax wrote:
| How can a system be stable if you don't at least scrap two
| features a year?
___________________________________________________________________
(page generated 2022-03-22 23:02 UTC)