[HN Gopher] Gitlab.com was experiencing elevated error rates for...
___________________________________________________________________
Gitlab.com was experiencing elevated error rates for Git, Web, and
API
Author : kaeruct
Score : 115 points
Date : 2021-03-15 11:38 UTC (11 hours ago)
(HTM) web link (status.gitlab.com)
(TXT) w3m dump (status.gitlab.com)
| damsta wrote:
| This status page is weird, it says that there is an active
| incident and that there is a databate latency issues, but
| everything is green: https://i.imgur.com/oXSiyTG.png
| dnsmichi wrote:
| Hi, GitLab team member here. Heads-up: GitLab.com appears to be
| stable again. Our engineers are monitoring and investigating the
| root cause of the incident.
|
| https://twitter.com/gitlabstatus/status/1371443832865222658
| perihelions wrote:
| Out of curiosity, why does your "about" page vouch for a
| suspended Twitter account? Is @GitLab -> @GitLabs a typo?
|
| https://about.gitlab.com/handbook/marketing/community-relati...
| <td><a href="https://twitter.com/GitLabs">@GitLab</a></td>
| sytse wrote:
| Thanks! It is a typo and I submitted a fix
| https://gitlab.com/gitlab-com/www-gitlab-
| com/-/merge_request...
|
| Please note that about.gitlab.com contains all our static
| content, including our 10,000 page handbook. This typo isn't
| on our main about page but deep in the marketing handbook.
| dnsmichi wrote:
| Thanks Sid, merged :) We've also fixed the formatting at
| https://about.gitlab.com/handbook/marketing/community-
| relati... in https://gitlab.com/gitlab-com/www-gitlab-
| com/-/merge_request...
| perihelions wrote:
| Cheers for that.
|
| I merely wanted to verify the @ was authentic, and that
| page was the top "site:gitlab.com" result.
| sytse wrote:
| Thanks for raising it.
| dnsmichi wrote:
| Thanks for flagging, that's a typo. Here's an MR to fix:
| https://gitlab.com/gitlab-com/www-gitlab-
| com/-/merge_request...
|
| The page has formatting problems, I'll rework this in a
| separate MR.
| ing33k wrote:
| /r took me a lot of effort to convince my CTO to move from
| Bitbucket to Gitlab. Since we have started using it ( 3 + months
| ), we have experienced more than 4 instances of downtime. If some
| one from Gitlab is reading this message, please consider a code
| freeze and spend some time on stabilizing the system.
|
| time to add a new backup remote to all our repos.
| ModernMech wrote:
| Agreed. I use gitlab for administering all of my courses. I
| started when they had more features compared to github (free
| private repos, built in test runners), but over time github has
| regained some but not all of that ground. I will still be using
| gitlab in the future, but am going to convince our systems
| admin to set up a local instance.
| ziml77 wrote:
| Run a private instance? We haven't had trouble with that
| approach.
| pbalau wrote:
| We are using bitbucket atm and I can't fathom a reason to move,
| care to elaborate?
| Xelbair wrote:
| Atlassian is Australian company.
|
| Australia has very weird and wide reaching laws regarding
| software backdoors.
| danaris wrote:
| Can you elaborate on a) what _kinds_ of laws on software
| backdoors? (Mandating them? Forbidding them? What sort?),
| and b) how that affects the GP 's request?
|
| Edit: Nvm on (b); somehow I got GitLab and BitBucket mixed
| up as to which one belonged to Atlassian.
|
| Still curious about what sort of backdoor laws they have.
| herodoturtle wrote:
| I think it's in reference to this from a few years ago:
| https://www.wired.com/story/australia-encryption-law-
| global-...
|
| EDIT: I've done some more reading up on this, and The
| Assistance and Access Act of 2018 explicitly states that
| government _cannot_ use backdoors:
| https://www.homeaffairs.gov.au/about-us/our-
| portfolios/natio...
| bshimmin wrote:
| I use GitHub and Bitbucket fairly extensively and I find
| Bitbucket painfully slow at times (eg. 7.38s to load a page
| just now with a whopping two open pull requests on it, whilst
| somehow transferring 8.8MB of crap; the equivalent page on
| GitHub is about 6 times better on both fronts) and the UI
| only ever seems to get more clunky. I fear the tight
| integration with JIRA is slowly infecting it with JIRA-esque
| problems.
| ing33k wrote:
| mainly for DevOps related features.
|
| https://about.gitlab.com/devops-tools/bitbucket-vs-gitlab/ (
| marketing page, but mostly valid points )
| notretarded wrote:
| Pweese fix my poor decision making OwO
| whalesalad wrote:
| This is what happens when you rename master to main.
| makeworld wrote:
| Oh come on
| rataata_jr wrote:
| He has a point, priorities under MS ownership.
| JosephRedfern wrote:
| Except he doesn't, because GitLab isn't owned by Microsoft.
| tester34 wrote:
| haha I wanted write comment like "since MS bought git the
| amount of crashes per week has increased"
|
| but I guess I'm too late to the party!
| detaro wrote:
| at least get right which company you are talking about ...
| whalesalad wrote:
| I'm not being facetious, or implying support for master/slave
| nomenclature. But if you ship a big change like this that
| impacts very core assumptions in 99% of the Git ecosystem,
| you're perhaps in for a treat.
| ahachete wrote:
| For those curious: the root cause for the incident is,
| ultimately, degraded statistics on a table.
|
| There are a set of high frequency (very frequently run queries)
| that are quite sensible to plan flipping (they normally run with
| a given execution plan; if the plan changes to a worse one,
| effects can be dramatic, given their frequency).
|
| That leads to a lot of query timeouts (GitLab's database limit
| query execution time to prevent further damage), which are
| visualized as database errors. This in turn leads to partial or
| complete downtime (effectively, if the database is running few
| effective --non timed out-- traffic).
|
| Some of these queries depend on a particular Postgres planner way
| of working that is less than ideal (in PG11, it is improved in
| PG12), and may lead to plan flipping. Which in turn, happens when
| table statistics degrade. In GitLab's case, they are normally
| close to the threshold, and slight statistics degradation caused
| a plan change, which in turn lead to many queries time out and
| excessive load on the fleet.
|
| At the end of the day, the fix is quite simple, however: update
| the table statistics (running ANALYZE). For more information:
| https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3...
| hobs wrote:
| s/sensible/sensitive/
|
| Would this be a place where a plan hint would help? It seems
| like having to re-analyze would mean that this is just a matter
| of time before it becomes problematic again.
| ahachete wrote:
| > s/sensible/sensitive/
|
| Thanks :)
|
| > Would this be a place where a plan hint would help?
|
| It is one option to consider (I wrote about it last week:
| https://gitlab.com/gitlab-com/gl-
| infra/production/-/issues/3...), but it is not an ideal
| solution, for several reasons.
|
| Re-analyze is not the solution either, but as a short-term
| measure, cron-ed ANALYZEs will be run. Longer term, apart
| from refactoring some queries which are quite prone to
| trigger this plan behavior, statistics gathering process is
| going to be fully reviewed.
| hobs wrote:
| Gotcha, that makes sense to me - refactoring queries which
| wanna flip would be my choice as well, just thinking that
| if the analyze fixes plans its (in my experience) "well we
| could have just used this plan anyway" - both of them are
| short term fixes for sure.
| mcintyre1994 wrote:
| I can't even get to the issue where they're tracking this any
| more. At least the status page is on different infra though,
| hopefully they'll keep it updated.
| boleary-gl wrote:
| GitLab employee here.
|
| That is correct - we also will be posting updates here:
| https://twitter.com/gitlabstatus/.
| [deleted]
| loloquwowndueo wrote:
| Oh gitLAB - for a moment there I misread "GitHub" and was
| worried. (J/k you rock guys!)
| INTPenis wrote:
| I love Gitlab because I love open source and competition but
| Gitlab either on-prem or gitlab.com has been having growing pains
| ever since I started using it around 2015.
|
| And it seems to be related to scaling because it's always iffy
| problems like higher error rates or jobs not executing.
|
| Either way, I love our on-prem Gitlab. It's free and it hosts
| over 200 projects, Gitops and the whole shebang. I'm just now
| beginning to use Kubernetes from it.
|
| They will sort this out eventually.
| hhh wrote:
| I have nothing but praise for our enterprise Gitlab
| implementation. It hosts 2000+ projects and for many teams is
| mission critical. Our team that maintains it is wonderful and
| I've only seen it down without planning once for about an hour.
| Using Gitlab at work actually swayed me to move to it for my
| personal projects as well.
| kevinob11 wrote:
| +1 for on-prem. I tried gitlab.com first but it was very slow
| (this was years ago, YMMV) but on-prem has been rock-solid and
| fast for us hosting ~100 projects. One thing I've been
| surprised by is how easy updates have been, I don't think we've
| ever had to even reconfigure anything, including runners. I
| know it seems like that is how it should work, but I've had a
| lot of opposite experiences with the open source tier of tools.
| irateswami wrote:
| My company does on-prem gitlab as well, and it's pretty
| fantastic. We're in the beginning stages of migrating off of
| Azure DevOps but so far we're wowed by the features.
| qeternity wrote:
| We switched from self-hosted to gitlab.com a couple months back
| and the migration has been extremely disappointing.
|
| Pushing commits often hangs completely, and we've had a number of
| smaller downtimes/degradations in recent weeks that have been
| super disruptive.
|
| Having been Gitlab users since our inception, we're now seriously
| considering Github (I know - grass is always greener).
| hughw wrote:
| We don't use Gitlab CI, but only their hosted git. Just thought
| I'd throw in our sterling experience over the past three years.
| We've really not noticed whatever downtime they've had.
| aaronharnly wrote:
| What prompted you to move away from self-hosted? What was the
| maintenance overhead like?
| robotmay wrote:
| Any chance you could drop me some details about pushes hanging?
| I'm an engineer in the source code team doing a lot of
| performance work and I might be able to correlate it with known
| issues or investigate it further. I know it's a bit awkward to
| get data after-the-fact but just some links to the problem
| repos would be useful. I might not be very useful if it's
| related to network/hardware issues but I do work in that aspect
| of the software at least.
|
| You can tag me on an issue on gitlab.com with @robotmay_gitlab
| if the details can be public, or email them to me at
| rmay@gitlab.com if not :)
| gilfoyle wrote:
| Gitlab is a great service and a great alternative to github and
| bitbucket. But it has been quite flaky over many years and it's
| quite surprising they haven't prioritised system stability over
| everything else. https://twitter.com/gitlabstatus
|
| Gitlab has been a fully remote proponent going back a few years
| and these persistent reliability issues add some doubt to that
| model - even if there might be altogether other issues in play.
| SkyPuncher wrote:
| Fully remotes has nothing to do with it.
|
| My company is full remote and we don't have these issue.
| hamburglar wrote:
| Agreed. Blaming a remote work culture is a complete non
| sequitur.
| sithlord wrote:
| My guess is it has to do with their prioritization of only
| hiring people with "significant ruby/rails" experience. Kills a
| lot of the talent pool, which doesn't make sense as easy is it
| is is to pick up a new high level language and framework.
| ezzato wrote:
| I was just pushing some new code to production when it stopped
| working. So frustrating.
| mondss wrote:
| I can't be the only one that feels that GitLab has extended
| downtime at least weekly, sometimes bi-weekly.
| bromonkey wrote:
| *any non self-hosted git provider
| stabbles wrote:
| Github seems to be doing worse
| Bombthecat wrote:
| Well, they are stretching and extending gitlab with more and
| more functions.. With a... Not so great code base..
| dnsmichi wrote:
| Hi, what could be improved in GitLab's code base in your
| opinion? :)
| YorickPeterse wrote:
| In the past people have suggested we rewrite GitLab in
| Scala, Haskell, Node.js, and IIRC also regular Java.
|
| I think it mostly comes down to "I didn't write it, so
| it's bad" :)
| Bombthecat wrote:
| Nah, i don't care which language. Like, at all. I just
| think that gitlab outgrow ruby.
| searchableguy wrote:
| Suprised no rust or elixir there.
| joana035 wrote:
| Is there any code base in this world that developers would
| not complain about? ;)
| brodock wrote:
| The only perfect code is the one you write, all others
| are crap
| rataata_jr wrote:
| Xmonad
| KptMarchewa wrote:
| The one that will never be deployed on production. Real
| world unfortunately corrupts beautiful code with it's
| pesky complications.
| wdb wrote:
| Gitlab is completely down until just now, similar happened
| not that long ago. Over the last three weeks continuously had
| issues with the Gitlab runners (even not shared ones) were it
| wouldn't jobs or only after a second or third retry attempt.
|
| I think the flakiness of jobs succeeding is also related
| whether you use their package registry. I can't say I had
| similar issues with Github.
| [deleted]
| rvz wrote:
| Yes. They (GitHub) have been doing a lot worse. Last time I
| checked, they where completely down 2 days ago. [0] Before
| that, GitHub Actions was down 14 days ago: [1].
|
| At least with GitLab, you can set up a self-hosted solution
| for free as a backup, unlike GitHub (Unless you want to pay a
| lot for GH Enterprise) where some have _' gone all in on
| GitHub Actions'_ and then some couldn't push that critical
| change [2] before the start of the weekend. Oh dear.
|
| Maybe its time to setup a self-hosted backup VCS and not
| depend _entirely_ on GitHub /GitLab web.
|
| [0] https://news.ycombinator.com/item?id=26439075
|
| [1] https://news.ycombinator.com/item?id=26301659
|
| [2] https://news.ycombinator.com/item?id=26439213
| tomxor wrote:
| > Maybe its time to setup a self-hosted backup VCS and not
| depend entirely on GitHub/GitLab web.
|
| I've been wanting to do this for some time purely for git
| hosting (don't care about CI at all) but wasn't sure about
| how to make it failover...
|
| Personally i'd be happy with a headless git server, but
| that's not fair on everyone else who wants the GUI to
| browse and organise stuff, so I want gitlab etc to deal
| with that. What would be nice is to have a headless backup
| server that allowed everyone to continue pulling/pushing
| from the CLI with their existing repos when gitlab is down.
| I can't see a smooth way of doing that since it would
| require messing with the git remotes, unless the solution
| is inverted and uses a single remote pointing at the
| "backup" server which then replicates to gitlab, but I
| don't think gitlab can be configured to change the default
| remote when people clone it from the UI.
|
| I suppose this is why people just end up self hosting
| gitlab instead.
| the_duke wrote:
| If you want transparent failover you need to use your own
| domain for the repo URL. In case of a failure you (or
| some automatic job) would have to change the DNS record.
|
| Gitlab has a repo mirroring feature [1]. But of course
| you'd also need to sync users public keys.
|
| Downtimes are so infrequent and relatively short that
| this isn't worth the effort for me.
|
| You can instead set up a read-only mirror so people can
| at least still pull and browse the code.. Gitea [2] might
| be a better choice than Gitlab, since it's much more
| lightweight and easier to host.
|
| [1] https://docs.gitlab.com/ce/user/project/repository/re
| positor...
|
| [2] https://gitea.io
| tomxor wrote:
| Thanks this is useful to know
|
| > Downtimes are so infrequent and relatively short that
| this isn't worth the effort for me.
|
| That's essentially the same conclusion I keep coming too,
| occasionally it has hit me when I go to push something
| but rarely has it blocked me or anyone else from
| continuing to work.
|
| > You can instead set up a read-only mirror so people can
| at least still pull and browse the code
|
| Yeah, this I need to do eventually just for peace of mind
| as a more automated backup solution. At least I don't
| have to care about failover.
| mroche wrote:
| Maybe some kind of pre/post-push hook to automatically
| push to a secondary repo? Not entirely sure if this is
| possible, I've never written a Git hook before.
|
| https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks
| Jamie9912 wrote:
| Our self-hosted GitLab instance is running fine!
| drcursor wrote:
| Pitty its licensing cost will go up ten fold next year.
| sofixa wrote:
| Only for the paid tiers, free tier stays free.
| drcursor wrote:
| 0 ten times fold = 0
| [deleted]
| loloquwowndueo wrote:
| How so? Pics or it didn't happen.
| smcleod wrote:
| https://about.gitlab.com/blog/2021/01/26/new-gitlab-
| product-...
| drcursor wrote:
| https://about.gitlab.com/blog/2021/01/26/new-gitlab-
| product-...
| mikepurvis wrote:
| Yup; I just updated ours to 13.9.4 and it was seamless as
| always (EE version).
| wongarsu wrote:
| Not downtime, but some kind of issue, yes [1]. However github
| is not much better [2] and gives much less detail and fewer
| updates. Bitbucket is more solid recently [3] (but wasn't
| really better than the others in 2020)
|
| 1:
| https://status.gitlab.com/pages/history/5b36dc6502d06804c083...
|
| 2: https://www.githubstatus.com/history (don't get fooled by
| them collapsing the incident list after three per month)
|
| 2: https://bitbucket.status.atlassian.com/history (also
| collapses after three per month)
| dt5702 wrote:
| Do you have info on how different providers stack up?
| RicoElectrico wrote:
| Obligatory "is bi-weekly 2 or 0.5 times a week" confusion
| comment ;)
|
| Though from the context it's 2.
| Jenk wrote:
| This is why we have "fortnightly" in the English language.
| ntechau wrote:
| ok
| kweinber wrote:
| Like it or not, fortnightly and "once every two weeks"
| are your only unambiguous choices.
|
| We actually had a programming interview question using
| biweekly at a previous job. It was about thorough
| requirements definition and understanding/agreeing to
| measurements.
| aufhebung wrote:
| The word fortnight is in common parlance. Four score is
| not. But for the record the term bi-weekly is still
| confusing.
| KptMarchewa wrote:
| Or just say "twice a week" and "once every two weeks" and
| not leave everyone in a meeting thinking you're talking
| about popular video game.
| herodoturtle wrote:
| TIL:
|
| "A fortnight is a unit of time equal to 14 days (2 weeks).
| The word derives from the Old English term feowertyne niht,
| meaning "fourteen nights".
|
| https://en.wikipedia.org/wiki/Fortnight
| 0x008 wrote:
| I use gitlab daily and have not experienced any outage in last
| 6 months probably.
| time0ut wrote:
| It is usually their shared runners. I experience outages with
| them a couple times a month. It isn't a big deal since we
| maintain our own private pool, so I just move jobs to them.
| (They make you pay for CI minutes so we use them for certain
| jobs.)
| wdb wrote:
| For me even our private runners stopped worked over the
| last few weeks
| time0ut wrote:
| Ouch. We haven't seen that, but we don't run them all the
| time either.
| skohan wrote:
| Same - I've had zero issues.
| danaris wrote:
| I use GitLab for my side project (source control only; don't
| have any kind of CI set up just at the moment) and it's been
| years since there was any really noticeable slowness for me.
| I can't say I use it daily, but fairly frequently.
| nops wrote:
| 6 months is a pretty low bar for a SCM tool.
| roblabla wrote:
| Github has had frequent downtime too in the last 6 months
| too. For a low bar, it seems to be fairly hard to clear.
___________________________________________________________________
(page generated 2021-03-15 23:01 UTC)