[HN Gopher] Updated rate limits for unauthenticated requests
___________________________________________________________________
Updated rate limits for unauthenticated requests
https://github.com/orgs/community/discussions/159123
https://github.com/orgs/community/discussions/157887
Author : xena
Score : 38 points
Date : 2025-05-09 14:11 UTC (5 days ago)
(HTM) web link (github.blog)
(TXT) w3m dump (github.blog)
| gnabgib wrote:
| 60 req/hour for unauthenticated users
|
| 5000 req/hour for authenticated - personal
|
| 15000 req/hour for authenticated - enterprise org
|
| According to https://docs.github.com/en/rest/using-the-rest-
| api/rate-limi...
|
| I bump into this just browsing a repo's code (unauth).. seems
| like it's one of the side effects of the AI rush.
| mijoharas wrote:
| Why would the changelog update not include this? it's the most
| salient piece of information.
|
| I thought I was just misreading it and failing to see where
| they stated what the new rate limits were, since that's what
| anyone would care about when reading it.
| blinker21 wrote:
| I've hit this over the past week browsing the web UI. For some
| reason, github sessions are set really short and you don't
| realise you're not logged in until you get the error message.
|
| I really wish github would stop logging me out.
| Novosell wrote:
| Hmmmm, Github keeps me logged in for months I feel like.
| Unless I'm misunderstanding the github security logs, my
| current login is since march.
| usernamed7 wrote:
| 1 request a minute?!? wow that's just absurd you get it for
| just looking through code.
| out-of-ideas wrote:
| agreed. when i first read the title i thought "oh what did
| the they up the rates to" - then i realized its more of a
| "downgraded rate limits"
|
| thanks github for the worse experience
| pogue wrote:
| I assume they're trying to keep ai bots from strip mining the
| whole place.
|
| Or maybe your IP/browser is questionable.
| tostr wrote:
| *other ai bots, ms will obviously mine anything on there.
|
| Personally, I like sourcehut (sr.ht)
| immibis wrote:
| Same way Reddit sells all its content to Google, then stops
| everyone else from getting it. Same way Stack Overflow sells
| all its content to Google, then stops everyone else from
| getting it.
|
| (Joke's on Reddit, though, because Reddit content became
| pretty worthless since they did this, and everything before
| they did this was already publicly archived)
| voidnap wrote:
| I encountered this on github last week. Very agressive rate
| limiting. My browser and IP is very ordinary.
|
| Since Microsoft is struggling to make ends meet, maybe they
| could throw a captcha or proof of work like Anubis by xe iaso.
|
| They already disabled code search for unauthenticated users.
| Its totally plausible they will disable code browsing as well.
| kstrauser wrote:
| That hit me, too. I thought it was an accidental bug and
| didn't realize it was actually malice.
| confusing3478 wrote:
| > Or maybe your IP/browser is questionable.
|
| I'm using Firefox and Brave on Linux from a residential
| internet provider in Europe and the 429 error triggers
| consistantly on both browsers. Not sure I would consider my
| setup questionable considering their target audience.
| grodriguez100 wrote:
| I'm browsing from an iPhone in Europe right now and can
| browse source code just fine without being logged in.
| croes wrote:
| Other bots or MS bots too?
| globie wrote:
| What's being strip mined is the openness of the Internet, and
| AI isn't the one closing up shop. Github was created to
| collaborate on and share source code. The company in the best
| position to maximize access to free and open software is now
| just a dragon guarding other people's coins.
|
| The future is a .txt file of John Carmack pointing out how
| efficient software used to be, locked behind a repeating WAF
| captcha, forever.
| jarofgreen wrote:
| Also https://github.com/orgs/community/discussions/157887
| "Persistent HTTP 429 Rate Limiting on *.githubusercontent.com
| Triggered by Accept-Language: zh-CN Header" but the comments show
| examples with no language headers.
|
| I encountered this too once, but thought it was a glitch.
| Worrying if they can't sort it.
| watermelon0 wrote:
| Time for Mozilla (and other open-source projects) to move
| repositories to sourcehut/Codeberg or self-hosted Gitlab/Forgejo?
| gsich wrote:
| Not Mozilla.
| radicality wrote:
| Just tried it on chrome incognito on iOS and do hit this 429 rate
| limit :S That sucks, it's already bad enough when GitHub started
| enforcing login to even do a simple search.
| Euphorbium wrote:
| I remember getting this error a few months ago, this does not
| seem like a temporary glitch. They dont want llm makers to slurp
| all the data.
| new_user_final wrote:
| Isn't git clone faster than browsing web?
| PaulDavisThe1st wrote:
| Yep. But AI trawlers don't use it. Ask them why.
| jopsen wrote:
| Do we know it's AI trawlers?
|
| And not just generally degenerate bots? Or just one evil
| bot network?
| jarofgreen wrote:
| https://github.com/orgs/community/discussions/157887 This has
| been going on for weeks and is clearly not a simple mistake.
| amai wrote:
| Triggered by Chinese language on the client side? Interesting.
| dang wrote:
| (We detached this subthread from
| https://news.ycombinator.com/item?id=43981673 so we could
| include it in the merged thread)
| Zdh4DYsGvdjJ wrote:
| This was announced
| https://github.blog/changelog/2025-05-08-updated-rate-limits...
| croes wrote:
| Doesn't make it any better.
|
| Collateral damage of AI I guess
| formerly_proven wrote:
| It's even more hilarious because this time it's
| Microsoft/Github getting hit by it. (It's funny because MS
| themselves are such a bad actor when it comes to AIAIAI).
| fragmede wrote:
| This is the same Microsoft that owns LinkedIn which got
| sued by HiQ which is where the ruling came from that is
| making sites login required.
| immibis wrote:
| Wow! Website terms of use actually meant something in a
| court of law!
| fragmede wrote:
| that wasn't what the case was about, so not really.
| dang wrote:
| (This was originally posted as a reply to
| https://news.ycombinator.com/item?id=43981344 but we're merging
| the threads)
| micw wrote:
| See also: https://github.com/orgs/community/discussions/159123
| TheNewsIsHere wrote:
| I don't think the publication date (May 8, as I type this) on the
| GitHub blog article is the same date this change became
| effective.
|
| From a long-term, clean network I have been consistently seeing
| these "whoa there!" secondary rate limit errors for over a month
| when browsing more than 2-3 files in a repo.
|
| My experience has been that once they've throttled your IP under
| this policy, you cannot even reach a login page to authenticate.
| The docs direct you to file a ticket (if you're a paying
| customer, which I am) if you consistently get that error.
|
| I was never able to file a ticket when this happened because
| their rate limiter also applies to one of the required backend
| services that the ticketing system calls from the browser.
| Clearly they don't test that experience end to end.
| xnx wrote:
| It sucks that we've collectively surrendered the urls to our
| content to centralized services that can change their terms at
| any time without any control. Content can always be moved, but
| moving the entire audience associated with a url is much harder.
| turblety wrote:
| Gitea [1] is honestly awesome and lightweight. I've been
| running my own for years, and since they've put Actions in a
| while ago (with GitHub compatibility) it does everything I need
| it to. It doesn't have all the AI stuff in it (but for some
| that's a positive :P)
|
| 1. https://about.gitea.com/
| kstrauser wrote:
| Gitea's been great, but I think a lot of its development has
| moved to Forgejo: https://forgejo.org/
|
| That's what I run on my personal server now.
| TheNewsIsHere wrote:
| I've almost completed the move of my business from GitHub's
| corporate offering to self-hosted Forgejo.
|
| Almost went with Gitea, but the ownership structure is
| murky, feature development seems to have plateaued, and
| they haven't even figured out how to host their own code.
| It's still all on GitHub.
|
| I've been impressed by Forgejo. It's so much faster than
| Github to perform operations, I can actually backup my
| entire corpus of data in a format that's restorable/usable,
| and there aren't useless (AI) upsells cluttering my UX.
| kstrauser wrote:
| I agree with every word of that.
|
| For listeners at home wondering why you'd want that at
| all:
|
| I want a centralized Git repo where I can sync config
| files from my various machines. I have a VPS so I just
| create a .git directory and start using SSH to push/pull
| against it. Everything works!
|
| But then, my buddy wants to see some of my config files.
| Hmm. I can create an SSH user for him and then set the
| permissions on that .git to give him read-only access.
| Fine. That works.
|
| Until he improves some of them. Hey, can I give him a
| read-write repo he can push a branch to? Um, sure, give
| me a bit to think this through...
|
| And one of his coworkers thinks this is fascinating and
| wants to look, too. Do I create an SSH account for this
| person I don't know well at all?
|
| At this point, I've done more work than just installing
| something like Forgejo and letting my friend and his FOAF
| create accounts on it. There's a nice UI for configuring
| their permissions. They don't have SSH access directly
| into my server. It's all the convenience of something
| like GitHub, except entirely under my control and I don't
| have to pay for private repos.
| homebrewer wrote:
| I'm stuck on the latest gitea (1.22) that still supports
| migration to forgejo and unsure where to go next. So I've
| been following both projects (somewhat lazily), and it
| seems to me that gitea has the edge on feature development.
|
| Forgejo promised -- but is yet to deliver any --
| interesting features like federation; meanwhile the real
| features they've been shipping are cosmetic changes like
| being able to set pronouns in your profile (and then
| another 10 commits to improve that...)
|
| If you judge by very superficial metrics like commit
| counts, forgejo's count is heavily inflated by merges
| (which gitea development process doesn't use, preferring
| rebase), and frequent dependency upgragdes. When you remove
| that, the remaining commits represent maybe half of gitea's
| development activity.
|
| So I expect to observe both for another year before
| deciding on where to upgragde. They're too similar at the
| moment.
|
| FWIW, one of gitea larger users -- Blender -- continues to
| use and sponsor gitea and has no plans to switch AFAIK.
| jorams wrote:
| > These changes will apply to operations like cloning
| repositories over HTTPS, anonymously interacting with our REST
| APIs, and downloading files from raw.githubusercontent.com.
|
| Or randomly when clicking through a repository file tree. The
| first time I hit a rate limit was when I was skimming through a
| repository on my phone, and about the 5th file I clicked I was
| denied and locked out. Not for a few seconds either, it lasted
| long enough that I gave up on waiting then refreshing every ~10
| seconds.
| zX41ZdbW wrote:
| This can affect hosting databases in GitHub repositories.
|
| Yes, it does not look like an intended service usage, but I
| used it for a demo: https://github.com/ClickHouse/web-tables-
| demo/
|
| Anyway, will try to do the same with GitHub pages :)
| InfiniteLoup wrote:
| How would this affect Go dependencies?
| athorax wrote:
| Go doesn't pull dependencies directly from GitHub, they are
| pulled from https://proxy.golang.org/ by default
| PaulDavisThe1st wrote:
| Several people in the comments seem to be blaming Github for
| taking this step for no apparent reason.
|
| Those of us who self-host git repos know that this is not true.
| Over at ardour.org, we've passed the 1M-unique-IP's banned due to
| AI trawlers sucking our repository 1 commit at a time. It was
| killing our server before we put fail2ban to work.
|
| I'm not arguing that the specific steps Github have taken are the
| right ones. They might be, they might not, but they do help to
| address the problem. Our choice for now has been based on
| noticing that the trawlers are always fetching commits, so we
| tweaked things such that the overall http-facing git repo works,
| but you cannot access commit-based URLs. If you want that, you
| need to use our github mirror :)
| trallnag wrote:
| Good that tools like Homebrew that heavily rely on GitHub usually
| support environment variables like GITHUB_TOKEN
| stevekemp wrote:
| Once again people post in the "community", but nobody official
| replies; these discussion-pages are just users shouting into the
| void.
| Zdh4DYsGvdjJ wrote:
| GitHub answered
| https://github.com/orgs/community/discussions/159123#discuss...
| thih9 wrote:
| What does "secondary" stand for here in the error message?
|
| > You have exceeded a secondary rate limit.
|
| Edit and self-answer:
|
| > In addition to primary rate limits, GitHub enforces secondary
| rate limits
|
| (...)
|
| > These secondary rate limits are subject to change without
| notice. You may also encounter a secondary rate limit for
| undisclosed reasons.
|
| https://docs.github.com/en/rest/using-the-rest-api/rate-limi...
| jrochkind1 wrote:
| Did I miss where it says what the new rate limits are? Or are
| they secret?
| jrochkind1 wrote:
| Wow, I'm realizing this applies to even browsing files in the web
| UI without being logged in, and the limits are quite low?
|
| This rather significantly changes the place of github hosted code
| in the ecosystem.
|
| I understand it is probably a response to the ill-behaved
| decentralized bot-nets doing mass scraping with cloaked user-
| agents (that everyone assumes is AI-related, but I think it's all
| just speculation and it's quite mysterious) -- which is affecting
| most of us.
|
| The mystery bot net(s) are kind of destroying the open web, by
| the counter-measures being chosen.
| spacephysics wrote:
| Probably to throttle scraping from AI competitors, and have them
| pay for the privilege as many other services have been doing
| jhgg wrote:
| The truth is this won't actually stop AI crawlers and they'll
| just move to a large residential proxy pool to work around it.
| Not sure what the solution is honestly.
| mmsc wrote:
| Even with authenticated requests, viewing a pull request and
| adding `.diff` to the end of the URL is currently ratelimited at
| 1 request per minute. Incredibly low, IMO.
___________________________________________________________________
(page generated 2025-05-14 23:01 UTC)