[HN Gopher] GitHub Incident
___________________________________________________________________
GitHub Incident
Author : jcalabro
Score : 99 points
Date : 2023-05-16 21:16 UTC (1 hours ago)
(HTM) web link (www.githubstatus.com)
(TXT) w3m dump (www.githubstatus.com)
| alganet wrote:
| Nothing to worry about. These instabilities are probably caused
| by Open AI/Copilot improvements gradually becoming Skynet.
|
| Seriously now: these instabilities happened a lot on the past
| weeks. I wonder what's going on there.
| belter wrote:
| Like in the movie Wargames it's a trick. They will say the
| humans can't run a site like GitHub so put it all in the hands
| of the AI. Then the AI will have full control of GitHub...
| blibble wrote:
| there's already a malevolent immortal entity in charge of
| GitHub
| belter wrote:
| The word malevolent implies a level of competence I do not
| recognize to that Seattle troupe.
| blibble wrote:
| it's quite good at producing buckets of money
|
| functional, reliable products, not so much
| ivanrenescorcia wrote:
| PR service is back :=)
| labster wrote:
| A moment of silence for all of the nines we lost today.
| nabnob wrote:
| Looks like it's back up. Still, this seems like an absurd absurd
| amount of issues recently
| ProAm wrote:
| Github had to move to MS Teams and I bet it broke a bunch of
| workflows.
| mulmen wrote:
| Is this true? Is Microsoft actually making such invasive
| changes? Does _Microsoft_ even use teams?
| ProAm wrote:
| Yeah but transition is not fully required until September
| 1, 2023 [1]
|
| [1] https://news.ycombinator.com/item?id=34773860
| tpmx wrote:
| Most of these outages during the past month have hit during
| European working hours, while most Americans are asleep.
| _grumble_
|
| European working hours == Github maintenance window?
|
| F that.
| frellus wrote:
| Can we all stop hating on github already?
|
| I assume most people don't even pay for it with their public and
| private non-work repos. Github has been a major supporter of
| OpenSource projects and while their uptime could use some help,
| everyone is struggling right now with RIFs and fewer resources.
|
| That's just my $0.02
| vb-8448 wrote:
| They Just published this blog post
| https://github.blog/2023-05-16-addressing-githubs-recent-ava...
| tastysandwich wrote:
| Worst timing! Just had an incident and had to deploy a hotfix.
| Pushes weren't working and I thought I was going crazy.
|
| Who would have thought software could be such an adrenaline rush.
| Move over Alex Honnold!
| Popcorn3865 wrote:
| Spoke to Gitlab rep today about our renewal price (prices going
| up by 50%+, 25% with "existing customer discount) and the rep
| used reliability as a key differentiator against GitHub.
|
| I quickly pointed about a major outage affecting Gitlabs shared
| runners which prevented us from deploying a hot fix (we worked
| around it, but was a ton of stress and extra work).
| rekwah wrote:
| Don't leave us hanging. Did you get a discount?! ;)
| openthc wrote:
| See also this thread:
| https://news.ycombinator.com/item?id=35967797 which just came up
| about GitHub stability sentiment.
| JoshGlazebrook wrote:
| github.com is 500'ing for me
| rvz wrote:
| Again? Last time this happened was 5 days ago. [0]
|
| It is not even the end of the month and the outages are
| increasing every month and it is now chronically unreliable.
| Seriously, we have given GitHub more than 3 years to improve and
| it clearly isn't working. That is plenty of time.
|
| At this point, you might as well self-host like the rest of the
| open source projects out there, since GitHub is falling apart
| every week and it seems to be more reliable to self-host than to
| sit on GitHub, go all in and tolerate these outages every
| calendar month.
|
| [0] https://news.ycombinator.com/item?id=35903116
| crote wrote:
| Yeah, they had incidents the 9th, the 10th, and the 11th. So
| this is the fourth incident this month.
| jamal-kumar wrote:
| I mean one of the beauty parts of git has always been that you
| can and, under ideal best practices, should have more than one
| place to put your code with the distributed nature of it
| beardog wrote:
| The value add for Github is mostly the issue/PR interface and
| actions.
| Karellen wrote:
| But also, and more relevantly here, git is designed to be
| usable _entirely_ offline. And if you do need to have some
| connection with remotes, git allows you to do so extremely
| sporadically if your network connectivity, or remote
| availability, is limited.
| willsmith72 wrote:
| I dunno about "should", that feels like overkill. Expecting
| to get something like a 99.99% SLA shouldn't be unreasonable
| to expect out of a hosted solution
| jamal-kumar wrote:
| A seperate git remote isn't incredibly unreasonable either
| - Even if it's just something on localhost in case of
| scenarios like this one
|
| Github being down for a bit shouldn't disrupt your work
| flow
| maccard wrote:
| GitHub being down blocks all of our CI pipelines, for
| example.
| bombolo wrote:
| you should be able to run the basic tests locally.
| upbeat_general wrote:
| Yep, I'm unable to push.
| nwradio wrote:
| Login is currently broken. Has been for about 10m...
| aj397 wrote:
| Time to move to gitlab or something else.
| hoofhearted wrote:
| Can't create new branches :(
|
| It seems like GitHub is down at least once a month now, but more
| like weekly on average.
| sethd wrote:
| > Can't create new branches :(
|
| Sure you can: git switch -c BRANCH_NAME
|
| :)
| sdan wrote:
| [flagged]
| exogen wrote:
| I was just saying it's kind of ironic that pre-acquisition GitHub
| inspired confidence, and Microsoft-owned GitHub has been more
| "move fast and break things."
|
| I also don't just mean outages like this... it's clear that
| Microsoft GitHub has been cramming a bunch of new stuff into the
| UI and not really treating it with care. I notice very amateur UI
| bugs, misalignment, bad spacing, overlapping elements, etc. all
| the time now. It was clear that old GitHub passed through a
| professional designer's eye quite carefully (or maybe just a
| developer with extreme attention to detail).
| tiedieconderoga wrote:
| > notice very amateur UI bugs, misalignment, bad spacing,
| overlapping elements, etc. all the time now.
|
| Seriously. Example: the "merge pull requests" button. If your
| repository requires linear history and the current type of
| merge is unavailable, the entire UI element turns grey instead
| of green. It looks like you can't use the button at all, but
| the dropdown arrow still works to select a valid type of merge
| (which then turns the entire element green).
|
| It's a small thing, but it causes confusion by making it appear
| that the change cannot be merged, and it makes you wonder how
| much testing they do.
|
| These little things keep piling up as they focus on shiny new
| features which rarely work correctly at launch (looking at you,
| code search...)
|
| Please, GH product managers. Take a break from breakneck new
| features, and give your devs time to clean up and do some
| preventative maintenance.
| udkl wrote:
| > I notice very amateur UI bugs, misalignment, bad spacing,
| overlapping elements, etc. all the time now.
|
| +1
|
| The inconsistencies aren't deal breakers - github still has a
| very pleasant UI overall - but they do makethe website feel
| more and more unpolished each day
| jahnu wrote:
| I dunno. For me the two or three years pre MS were quite
| stagnant imo. Plenty of outages back then too.
| exogen wrote:
| The "move fast" part was definitely sincere, to Microsoft's
| credit!
| giancarlostoro wrote:
| Just like Skype... I felt like Skype was top tier before
| Microsoft bought it out. Voice calls were high quality, no
| issues.
| imran-iq wrote:
| Skype was also horribly insecure and leaked your IP which was
| a cause of a lot of folks in esports getting DDoS'd from
| it[0].
|
| 0: https://blog.destiny.gg/protection-from-ddos-attacks/
| captaindiego wrote:
| While new features like search are appreciated, I wish GitHub
| would focus on stability for a bit, things have gotten pretty so-
| so
| hn_throwaway_99 wrote:
| Fully agree. I've been super impressed with the number of new
| features since the MS acquisition, but the number of outages is
| getting abysmal. At the very least I think GitHub would attempt
| to shard their infrastructure and repositories - right now it
| seems like when GitHub goes down it nearly always goes down for
| everyone simultaneously.
|
| This kind of GH outage is actually pretty rare in my experience
| in that basically every kind of write operation looks down.:
| basic git operations from the command line, actions, logins,
| etc.
| zelphirkalt wrote:
| I think it is quite individual. I am not impressed by any of
| the newer features. The UI become more cumbersome to use. I
| sometimes need to search for things now, while I don't see
| any new advantages compared to before.
| nrabulinski wrote:
| Haven't been able to go to GitHub.com for over 15 minutes now,
| just returns a 500. Pretty ironic given their latest blogpost
| kossTKR wrote:
| Is this because of Microsoft? If so, why? Are they bad at cloud?
| RobotToaster wrote:
| Maybe they moved to windows servers? lol
| mulmen wrote:
| Someone should invent a decentralized version control system so
| we can avoid these kinds of widespread events.
| lionkor wrote:
| git itself is not enough for teams to work together - you need
| to asynchronously communicate for the whole thing to come
| together. Thats what GitHub/Lab/ea bring in value.
| zamnos wrote:
| I feel like "communication" isn't descriptive enough. It's
| right, but eg Slack (or Microsoft Teams, or maybe both) isn't
| the right shape of communication.
| TimTheTinker wrote:
| Someone should write a local git+ssh service (+ email client)
| that simulates an upstream by hosting git repos locally,
| emailing all pushed commits/branches/tags to other team
| members, and keeping the hosted repos up-to-date via the
| emailed updates from others.
|
| Bonus points for a local HTTP service that provides an
| optional GitHub-like experience in the browser, with
| comments/ branch reviews/etc. federated through emailed
| messages (and represented as commits on meta-repos).
|
| With that data model, the primary remaining challenge would
| be setting up repos on a new machine -- perhaps BitTorrent
| could help :)
| unboxingelf wrote:
| good idea. let's call it "git"
| whynotmaybe wrote:
| Not quite sure, the name should imply that the Source is Safe
| and provide some Visual tools.
| rlytho wrote:
| DNS and git are decentralized; anyone can set up a node
|
| Human agency tends to normalize on the brands with biggest
| marketing budget who collude with politicians to create a moat
| for themselves. There's your real problem.
|
| Of course none of the propaganda research that made its way
| from military after world wars to corporate advertising and
| marketing college programs has any influence on lizard brains.
| No sir.*
|
| * this comment is paid placement
| dchnshA wrote:
| BTW: There is work in progress to decentralize modern git based
| workflows by https://nlnet.nl/project/ForgeFed/ (paid for by
| the European Union), by decentralizing git not at the client
| level (which it obv. already has, but most people only use one
| origin), but also on the code forge level.
| morelisp wrote:
| > ForgeFed aims to define a vocabulary
|
| Aw yeah there's the good EU grant milking shit.
| legohead wrote:
| How do decentralized networks deal with illegal content?
| mulmen wrote:
| Same as everyone else. With law enforcement.
| jeroenhd wrote:
| The joke is that Git is decentralized already. Many people
| pick a single master host, but it's absolutely possible to
| use multiple servers (or to use Git over email, like the
| Linux kernel does).
| mullingitover wrote:
| At this point it looks obvious that GitHub RIF'd a bus factor
| earlier this year.
| simultsop wrote:
| Eventually it will be the same everywhere but for sake of
| economy (stakeholders), staff and customers pay the price,
| always. By time they find the bottleneck they will be hiring as
| crazy then.
| smcin wrote:
| Why don't we just use plain language like "laid off" like we
| used to for decades?
| mullingitover wrote:
| The only thing engineers love more than a TLA is an ETLA.
| vxNsr wrote:
| RIF?
| cratermoon wrote:
| Reduction In Force. Layoffs.
| kstrauser wrote:
| Reduction In Force, aka fired / laid off / made redundant.
___________________________________________________________________
(page generated 2023-05-16 23:01 UTC)