[HN Gopher] Nitter is working again
___________________________________________________________________
Nitter is working again
Author : linusg789
Score : 367 points
Date : 2023-07-10 12:33 UTC (10 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| charcircuit wrote:
| This does not seem legal. Stealing an access token to bypass
| access controls is illegal and I suspect these people didn't get
| permission to just scrape anything they want.
| paxys wrote:
| Who are they "stealing" the token from?
| charcircuit wrote:
| Twitter as these are from the Twitter android app.
| beepbooptheory wrote:
| I am not a lawyer but take your word for it. Is it illegal
| specifically in US, or elsewhere too?
|
| Either way, you'd surely agree its a noble pursuit? Just
| considering the wider context here!
| charcircuit wrote:
| It in illegal in most jurisdictions to access data which you
| are unauthorized to.
|
| >Either way, you'd surely agree its a noble pursuit?
|
| Considering it hurts Twitter's profitability by not showing
| ads, hurts Twitter's metrics by not having people sign in or
| sign up, hurts users who were accidently signed out from
| signing in to twitter and having a better user experience,
| and hurts content creators because nitter doesn't allow you
| to like or retweet posts. I do not see it as a noble pursuit.
| gadders wrote:
| Can someone do Fritter next please so I have a decent android
| Twitter client?
| flotzam wrote:
| Right now the Squawker fork of Fritter is working fine. You can
| export from Fritter and import into Squawker (and presumably
| vice versa if Fritter catches up). It's in the IzzyOnDroid
| F-Droid repository:
| https://apt.izzysoft.de/fdroid/index/apk/org.ca.squawker
| jonjomckay wrote:
| The latest CI/GitHub Actions builds have a fix for this
| implemented already! I'll be publishing another beta once I
| have enough time to QA all the latest changes on the Fritter
| and Twitter side.
| costco wrote:
| Warning: long somewhat related story that is basically
| humblebragging, but the summary is that bypassing Twitter
| ratelimits is not very hard.
|
| I didn't feel like playing around with Twitter's annoying
| certificate pinning so I just uploaded the Twitter APK to
| Corellium, turned on what they call the "network monitor", opened
| the Twitter app since it lets you use Twitter without signing in.
| I clicked around, searched and viewed tweets. Then I looked at
| the requests in the log and saw it has a similar guest token
| process to the website but with a few differences. Anyways, if
| you recreate these requests, with one IP address you can generate
| a few OAuth tokens with no expiry per day. These tokens are for
| unauthenticated users so obviously they have no write privileges
| but that's not what was needed here. So if you have a proxy
| provider with a large pool of IPs where you can buy like 1GB of
| bandwidth you can use a very small percent of your bandwidth
| allowance and get thousands of tokens/secrets easily, all with
| their own separate rate limits. It doesn't even matter what IP
| you end up using the tokens on. Then I followed
| https://docs.google.com/document/d/1xVrPoNutyqTdQ04DXBEZW4ZW...
| and the fact that /statuses/lookup.json still allows you to
| return 100 (!) tweets at once to reconstruct something close to
| what the 50% Twitter firehose would look like. And Twitter
| doesn't even block datacenter IP addresses! Was going to display
| the data at https://firehose.lol but the fact that it required a
| few hundred requests a second made me feel bad so I didn't end up
| running the program for more than a few minutes at a time and
| shut it down.
|
| Looking at (a fraction of) the Firehose for a few minutes was
| interesting, originally I accidentally forgot to not display
| tweets labelled possibly_sensitive so I saw some pretty salacious
| material for a few seconds. Lots of Chinese gambling ads even
| though Twitter is blocked there, dubious investment promoters,
| accounts with usernames like FirstnameLastname3781264872 who
| would tweet three random words at each other every couple of
| seconds, and a handful of funny tweets.
| matteoraso wrote:
| Nice. This isn't nearly as efficient, but a simpler way to
| bypass the ratelimit is to use archive.md, which is immune to
| the ratelimit. It's useful if you don't have an account and
| just want to see a few tweets here and there.
| linusg789 wrote:
| web.archive.org and ghostarchive.org also work.
|
| you can also use the Googlebot user agent to see the page,
| despite it being a different format
| sylware wrote:
| It means anonymous access has been restored. Nitter does not use
| twitter API which is a goner anyway (but was restored for a
| little while).
|
| Twitter should provide a noscript/basic (x)html interop www
| portal.
| naillo wrote:
| Funny how the effect of the rate limit has barely affected bots
| and scrapers _at all_ but severely damaged the userbase of the
| site.
| malermeister wrote:
| tinfoil hat time: what if that was the point all along? twitter
| was a vital space for organizing protests - think arab spring,
| occupy wall street.
|
| one of the richest guys in the world (who's also very anti-
| union, btw!) buys it up in a time where inequality is getting
| worse and worse and social fabrics are starting to tear and
| makes it unusable.
|
| no more space for organizing. one fewer threat to capital.
| paxys wrote:
| He owns the site. If this is truly his intention then all he
| has to do is turn off the main switch. There's really no
| reason to give him the benefit of doubt and twist the
| narrative into "Musk is a genius and everything is going
| according to plan". The simpler theory is likely to be the
| correct one - he has never run a social network before and
| has no idea how to stop bleeding users and cash so is
| desperately throwing ideas at the wall hoping something
| sticks.
| tedunangst wrote:
| I don't believe the theory, but if you wanted to cut off
| protests, you'd want to leave the site up but severely
| degraded to delay migration away.
| klardotsh wrote:
| Given the dude paid $44B for a site clearly worth
| significantly less than that, and then promptly ran it into
| the ground, your "tinfoil hat time" answer honestly seems
| like the only rational answer. It checks out on more levels
| than any of his actions have.
| coldpie wrote:
| Yes, it seems plausible. It was also funded in large part
| by the Saudis, American banks, and other wealthy
| individuals who are all strongly incentivized to hinder
| activist communication networks. Now everyone who used
| Twitter has to re-form their networks elsewhere. Mission
| accomplished.
| https://www.aljazeera.com/news/2022/10/28/saudis-kingdom-
| hol...
| rtsil wrote:
| He would be really stupid if that is his thinking, because as
| we just saw these users would just move to a competitor as
| long as there is one. Twitter doesn't have a monopoly on
| short conversation-based social media.
| malermeister wrote:
| A competitor that "isn't for news, politics or negativity"
| [0]. In other words, a place that won't let the same thing
| happen.
|
| [0] https://www.theverge.com/2023/7/7/23787334/instagram-
| threads...
| rtsil wrote:
| > news, politics or negativity
|
| They said they won't boost these, not that these are
| forbidden.
| ryantgtg wrote:
| Very anecdotally, bot activity on my accounts was increasing in
| the last month and then it completely stopped after that recent
| rate limiting stuff.
| paxys wrote:
| I feel the same way. Don't know if it's because of the
| algorithm or something else but Twitter these days seems to
| be 80% bots shilling crypto and/or simping for Musk.
| MicropenisMike wrote:
| It's true for a lot of cases.
|
| Adding almost any restriction only hurts casual users, and
| attackers are rarely casual users.
| ilikehurdles wrote:
| Anyone remember threads? Whatever happened to that app?
| sangnoir wrote:
| Last I heard, it had 100M sign-up by its 5th day, nbd.
| barbazoo wrote:
| Didn't they release a beta last week?
| zirgs wrote:
| Still not available in the EU. Probably because of privacy
| issues.
| meepmorp wrote:
| So, is the AI Scrapocalypse no longer a critically important
| issue?
| brokenbyclouds wrote:
| I'm skeptical that was ever the rain for the rationing.
| cmrdporcupine wrote:
| Probably he loudly threatened to fire his workshop elves so
| they worked through the nights and slept on old mattresses in
| the office so they could Make Twitter Scale Again and now all
| is well in Load Balancer land?
| I_am_tiberius wrote:
| This seems like Elon distributed free API tokens to specific 3rd
| party vendors?
| Macha wrote:
| What reason could he possibly have to give them to Nitter,
| which has at its primary purpose undermining their conversion
| and engagement attempts?
|
| The old token was the twitter web token. I suspect the new one
| is one of the mobile clients. Maybe new tweetdeck. Though
| probably the iOS client token makes the most sense, being the
| hardest to rotate on a whim with app store review.
| I_am_tiberius wrote:
| Ok thanks.
| tgv wrote:
| They don't have indivual tokens? Well, let's not give them
| ideas.
| Macha wrote:
| The clients currently work logged out (exempting rate
| limits) and it's not like you can upload a per user copy of
| the app to play store/app store, so that root of trust
| needs to start somewhere which is what the nitter team can
| extract.
| jakear wrote:
| Hmmm... I was thinking you could limit based on the
| unique identifierForVendor[1], but without a way to
| verify that a given id is legit this would be easily
| circumvented. An API whereby Apple cryptographically
| signs a vendor/device-specific ID so you can effectively
| rate limit without needing any personal info whatsoever
| would be nice.
|
| 1: https://developer.apple.com/documentation/uikit/uidevi
| ce/162...
| Macha wrote:
| A device specific ID would be a fingerprinting mechanism
| that would conflict with the privacy goals though? It
| would then have to be resettable, like the ad identifier
| already is.
| jakear wrote:
| They already have a device specific ID, this isn't asking
| for a new ID. See the link above. It's not considered a
| fingerprinting mechanism because it's only specific to a
| device/app-vendor pair, so can't be shared to fingerprint
| across apps.
|
| The problem is you can't use it for rate limiting because
| a bad actor could just generate a random ID and use that.
| That's why an endpoint for validating a given ID was
| issued for a particular vendor is required for a privacy-
| preserving anonymous rate limiting implementation.
| INTPenis wrote:
| Is this really permanent? I'd love to know more about this bearer
| token.
|
| Because in the other github issue thread it seemed like every
| time they found a way around Twitter's safeguards, it was
| shutdown.
|
| It seems like they've literally hard coded a token into the
| source code. Meaning thousands of nitter-instances, thousands of
| users, around the world, will use the same token.
|
| And potentially so will the AI companies.
|
| So I just don't see how this can work.
| jonnycomputer wrote:
| I'm curious that an "unofficial" API has been allowed to
| continue working, however intermittently, at all. I appreciate
| using Nitter, but something about it doesn't add up to me.
| DeathArrow wrote:
| It seems the old API is working again.
| hospitalJail wrote:
| I think this is why NewPipe doesn't work most of the time.
|
| Or if it does work, its absurdly slow.
| roumenguha wrote:
| Did you mean invidious?
| lolinder wrote:
| NewPipe has been working for me for years with _occasional_
| breaking changes on YouTube 's side that required an urgent
| update. I watch pretty much everything at 2x speed 1080p and
| have no issues with buffering.
| ajot wrote:
| I generally watch things on 720p, and it's been crashing
| frequently. Setting the resolution to 1080 usually solves
| the problem, but transfers the problem to my wonky internet
| connection.
| cmeacham98 wrote:
| My guess is that this token is used in the official Twitter
| web/mobile app - making it hard for Twitter to just straight up
| disable/ban.
| NelsonMinar wrote:
| that's what happened last time we went around this merry-go-
| round, back in January or so. Those keys were working for
| many months.
|
| https://www.reddit.com/r/fossdroid/comments/10b0krt/comment/.
| ..
| soraminazuki wrote:
| I wouldn't bet on it. After all, this is Elon Musk we're
| talking about here. When given a choice, he'll choose the
| most disruptive option.
| foderking wrote:
| "the most entertaining outcome is most likely"
| rompic wrote:
| RSS does not work?
| zedeus wrote:
| It's disabled on nitter.net to reduce load while this new
| solution is being tested. Check the instance list for updated
| instances, most of them have RSS enabled:
| https://github.com/zedeus/nitter/wiki/Instances
| browningstreet wrote:
| Thanks for your work on this.
| nicolas-siplis wrote:
| Hey there, just wanted to thank you because you also fixed my
| Twitter Spaces downloader app[0]! After the API changes the
| default bearer token I was using (same as yours) stopped working,
| but after changing the same way you all's back to normal :D
|
| 0: https://github.com/Chiplis/moonbird
| tough wrote:
| Hey thanks I didn't know this app you built existed, will try
| it out.
| sheepscreek wrote:
| This is the most impressive project in Nim I've seen yet.
| Rewriting any major front-end, complete with working
| authentication and handling idiosyncrasies of the private API is
| a herculean task. For context, Twitter would have a team of two
| dozen or more supporting what this does, effectively. Kudos to
| the author for accomplishing this feat!
| unsupp0rted wrote:
| > For context, Twitter would have a team of two dozen or more
| supporting what this does
|
| Of course, because if Nitter goes down nobody bats an eye.
| matheusmoreira wrote:
| Nobody? If Nitter goes down Twitter might as well not even
| exist to me.
| nunez wrote:
| Same.
|
| I had my Privacy Redirect plugin redirect Twitter to
| 0.0.0.0 instead of nitter.lacontrevoie.fr after it got
| killed.
| black_puppydog wrote:
| this is exactly my experience for the last week. let's see
| whether I feel the need to update my instance with this
| patch. for now I'm just like "oh, another twitter link, let
| me completely ignore it."
| matheusmoreira wrote:
| I'll either ignore it or wait until somebody else
| accesses it and copies the text or takes a screenshot. If
| nobody does, it's because it's not important enough and I
| probably don't need to know about it.
| JeremyNT wrote:
| Yeah, and I immediately realized Nitter was back when I saw
| 100+ unread tweets in my RSS feed.
|
| I just marked them all as read and moved on. I guess this
| Nitter outage may be the thing that finally pushes me to
| find the people I follow on other platforms.
| Macha wrote:
| And the flagship instance of Nitter is unusable more often
| than not - the fact that it's working today is likely an
| artifact of people having given up given the week long total
| outage.
| zedeus wrote:
| Maybe you don't use it very often? For months before the
| recent chaos it was very reliable. The fact that it's
| working today is only because I fixed it, Twitter didn't
| change anything.
| uka wrote:
| It works great. Don't pay attention to the philosophers.
|
| I quit twitter when I discovered it. Thank you.
| Macha wrote:
| To be clear, this is not intended to be a dig at the
| nitter team, I always assumed the software was more their
| focus than nitter.net
| Macha wrote:
| Timelines are usually blank, to the point that I have
| removed it from the rotation of instances in my browser
| redirector extension. I assumed this was a result of
| being rate limited by Twitter.
|
| I will admit that since I did remove it from the
| rotation, I don't see it that often these days, mostly
| when others link to it and then I navigate to a timeline.
| tourmalinetaco wrote:
| I've had timelines/tweets fail to load, but a refresh
| fixed it quickly. If this was a consistent problem did
| you consider opening an issue to see if they could
| improve the website?
| jonwz wrote:
| Any chance you can re-enable rss?
| skulk wrote:
| The frontend uses Karax, which is my favorite frontend/SPA
| library in any language. It is an absolute joy to use, even if
| it's a bit rough around the edges.
|
| https://github.com/karaxnim/karax
| maxlin wrote:
| Awesome, now my light personal-use scraper works again. Didn't
| even take a restart!
|
| Hope Twitter soon lets go of the temporary login restriction too.
| Given that this isn't completely blocked without a login, I'd
| expect that to be not far from now on. From what I've collected,
| I hope Twitter'd start selling dumps of their public data for a
| bit of a win/win with AI companies and Twitter itself.
| costco wrote:
| I think this exists:
| https://developer.twitter.com/en/docs/twitter-api/enterprise...
| but it is rumored to be absurdly expensive. Unless prices come
| down to at least the same order of magnitude as scraping I
| suspect people will be willing to risk the legal uncertainty
| involved with scraping.
| eterps wrote:
| Displaying a timeline with multiple accounts seems still broken,
| but other than that it works fine.
| zedeus wrote:
| Search, which is required for the multi-timeline feature, isn't
| implemented right now. I have a fix in the works using a legacy
| endpoint.
| lewantmontreal wrote:
| We weren't supposed to talk about fight club
| snarkyturtle wrote:
| I mean, nitter has pages indexed on Google so it's not exactly
| secret
| al1r4d wrote:
| Alhamdulillah
| linusg789 wrote:
| Not sure why you're being downvoted (Alhamdulillah - "thank
| god" in Arabic)
| joos3 wrote:
| This is absolutely amazing, as you still can't view profiles or
| replies on Twitter without logging in.
|
| I've been a nitter user for 4 years now and will be as long as it
| works.
| DavideNL wrote:
| Note that you can use RSS too - just append `/rss` to the
| url...
| synthoidzeta wrote:
| To the Nitter URL? Not working for me
| DavideNL wrote:
| Yea, there's also an icon/link in the upper right corner of
| the page, just use that.
|
| It's disabled on some instances...
| tourmalinetaco wrote:
| Unfortunately RSS has been disabled on the main instance.
| It had been available before, but presumably given the
| rate limiting has been disabled.
| linusg789 wrote:
| The alternate instances may still have them.
___________________________________________________________________
(page generated 2023-07-10 23:01 UTC)