[HN Gopher] Twitter shut off API access; users volunteering thei...
___________________________________________________________________
Twitter shut off API access; users volunteering their own data for
an open API
Author : OmarShehata
Score : 106 points
Date : 2024-09-18 16:23 UTC (6 hours ago)
(HTM) web link (omarshehata.substack.com)
(TXT) w3m dump (omarshehata.substack.com)
| toomuchtodo wrote:
| https://www.community-archive.org/
|
| https://github.com/TheExGenesis/community-archive
| ranger_danger wrote:
| > cheaply as possible (put everything in S3
|
| yikes.
| OmarShehata wrote:
| feedback is very much needed
| diggan wrote:
| For reference, this is the GitHub issue:
| https://github.com/TheExGenesis/community-archive/issues/72
|
| Cheapest is to stay away from anything related to the big
| clouds, especially Amazon with their "premium bandwidth"
| charges.
|
| I'd probably spin up a storage box from Hetzner (or similar
| service for dedicated servers) as a first step, as you get
| unmetered connection. It should last you a pretty long while
| as most content will be fetched from caches.
| evv wrote:
| To be charitable I hope the author refers to the S3 API here,
| of which there are several competitors and open source
| implementations.
|
| Many companies offer "object storage" which is compatible with
| the S3 API
|
| And there are some open source projects such as Minio which
| offer the same (and I forget what are the others?)
| rasengan wrote:
| The trend of shutting down / charging steeply for API access has
| fundamentally changed the internet.
| flykespice wrote:
| Back to the old web scrapping you go.
| CaptainFever wrote:
| Yep. Short-sighted companies don't realize that API was a
| truce, to save money and increase security (e.g. giving
| user/pass to third-party clients -> OAuth) on both sides. But
| anything that can be displayed, can be scraped...
| https://www.eff.org/issues/analog-hole
| add-sub-mul-div wrote:
| I honestly don't know what to think of this time. On one hand
| it's sad in principle to see Reddit and Twitter lock down to
| the point they have.
|
| But on the other hand they'd both already become cesspools by
| that time, and I was still visiting them daily. And now I've
| quit them both which is a good thing.
| gary_0 wrote:
| The whole point of the Internet up until about 2019 was that it
| was so cheap to host information it was basically free. If your
| site scaled up, you covered the hosting costs with ads or
| donations or something. The expensive part was finding content,
| so "user generated content" sites had to entice users to post
| stuff. That resulted in an implicit social contract with the
| users; these sites lived in fear of the users taking their
| content elsewhere.
|
| Now the Web is expensive for some reason, and users have become
| so dependent on a small number of sites as a communications
| medium that the megacorporations running them feel like they
| have infinite leverage, and the social contract of user-
| generated content is completely forgotten.
| bcrosby95 wrote:
| Hosting information isn't expensive, trying to justify your
| billion dollar valuation is.
| throw0101a wrote:
| Many moons ago Twitter used to have RSS (Atom?) feeds for each
| user so you could use any old news aggregator to keep up to date.
| pavlov wrote:
| But that's too sensible and user-friendly. It would be
| obviously weird and wrong if they inserted ads and Musk's
| tweets into someone's RSS feed. Yet showing those unwanted
| insertions is the whole purpose of the company now.
| sunaookami wrote:
| Twitter shut off RSS access over 10 years ago along with
| their popular 1.1 API.
| pavlov wrote:
| I know. Because it wasn't compatible with their aim of
| showing ads.
|
| And nothing has changed after Twitter went private under
| Musk, except that the reasons for stuffing unwanted content
| into users' timelines are slightly different.
| xNeil wrote:
| I don't think that's fair - getting back the
| chronological timeline itself made the purchase a good
| one in my eyes, alongside the For You feed, which seems
| to work wonderfully (for me). Not to forget you can
| literally turn off ads if you want. I seem to be a real
| minority, but I do genuinely believe Musk has done a
| great job with Twitter (or X).
| spondylosaurus wrote:
| You could get SMS notifications whenever a user tweeted, too.
| Imagine signing up for that today!
| sahila wrote:
| That makes more sense to disable as SMS is expensive. This is
| still fairly easy to set up yourself using a tool like
| IFTTT[1] or similar.
|
| [1] https://ifttt.com/search/query/twitter
| WorldMaker wrote:
| You could also for a long time send SMS (if you registered
| your number) to 40404 to tweet to Twitter. It's still in my
| address book from the era when I was tweeting T9 from dumb
| phones or using voice-to-SMS to tweet on early smart phones.
| crystal_revenge wrote:
| Social media used to be about content sharing, but it's clear
| that it's far more profitable to keep your users on your site
| at all times and keep the content they create in house rather
| than linked somewhere else.
|
| HN is probably the last true aggregator site left that I know
| of.
|
| It's partly why the web is so much worse. There's really no
| reason to create content outside of a walled platform is it is
| getting increasingly difficult to find an audience for it. Even
| blogging is an uphill battle since more and more social media
| sites penalize link sharing (they want you to create the
| content on their platform and leave it there).
|
| That's why it's not surprising that these APIs are disappearing
| since the fundamental model has changed.
| criticalfault wrote:
| Users should just get off this continued tragedy and API access
| wouldn't be an issue
| OmarShehata wrote:
| A lot of users/communities are stuck because of network effects
| & the long history of posts (they still reference them, like
| living wikis)
|
| These pipeline tools can help people migrate if they want
| without losing that history & network (my prediction is once
| people see what's possible with an open API, that'll further
| motivate user migration, or for twitter to open up again)
| mschuster91 wrote:
| > the long history of posts (they still reference them, like
| living wikis)
|
| As someone with a multitude of cat photo threads, 100x this.
|
| While I'm around on Twitter, Bluesky and Mastodon, the "post
| history" aspect is my most pressing issue with Mastodon.
| Like, when you move your account, you cannot take your old
| posts to the new server - you're straight out of luck if your
| instance decides to call it quits for whatever reason. And
| even if the instance _doesn 't_ give up, being unable to view
| old(er) photos is something I very commonly notice on a bunch
| of Mastodon servers.
| yodsanklai wrote:
| To me, Twitter is by far the most toxic social network. It has
| the power to turn the most interesting and smartest people into
| bitter trolls. Maybe some people get value from it, but it
| requires serious self discipline to not get dragged into stupid
| arguments.
| gspencley wrote:
| I've tried to "get" Twitter since the early days. I've
| created a couple of Twitter accounts over the years to
| support various creative endeavours but I've never found
| myself getting much value out of it as a user.
|
| I heard that it's original use-case was that back 2008 there
| were no good ways to do group messaging over SMS on a cell
| phone. That's a problem and solution that I can understand.
|
| But as a broader social network? I don't get it.
|
| Things that bother me to the point that they are deal
| breakers:
|
| - The character limits (there is nothing worse in life than
| reading through a x/N self-reply to read something long ...
| I'd rather file a tax return)
|
| - Showing me posts from pages I don't follow in the feed (in
| 99% of the cases I'm aware of that page/profile already and
| have chosen not to follow it because I don't like them)
|
| - Ads in the news feed
|
| Some have said that they like Twitter for getting news. I
| have way better options for that.
|
| I know that I'm not representative of the typical person,
| generally speaking, but Twitter is one of those things where
| I really can't understand why _anyone_ likes it and uses it,
| let alone why it is so popular. And it 's not that I'm hating
| on something I don't know anything about ... I've honestly
| tried to use it and get value out of it, but I've never found
| anything of value on offer.
| rchaud wrote:
| > but Twitter is one of those things where I really can't
| understand why anyone likes it and uses it, let alone why
| it is so popular.
|
| I think it's a carryover of general nostalgia for how the
| Internet "used to be", i.e. before every site with a few
| million users decided to start experimenting with
| algorithms to max out user engagement and ad impressions,
| leading to the hell we see everywhere, from IG to Tiktok.
|
| Yes, there was a time when Twitter wasn't a toxic mess.
| It's the last social network that was popular before
| smartphones took off.
| gspencley wrote:
| Interesting. That might explain why I dislike it so much.
| As a middle aged person who grew up with an early
| iteration of the world wide web for a decade to a decade
| and a half before Twitter even existed, both smart phones
| and social media mark a turning point for me from what
| the Internet "used to be."
|
| Twitter, in my mind (and maybe this is perception and not
| reality), ushered in infinite scroll and short bites of
| information. Twitter is to forums what TikTok is to
| documentaries. I see Twitter and the "mobile revolution"
| going hand in hand (something that left me behind because
| I still dislike using a smart phone, generally, and
| rarely do compared to most other people).
|
| But I guess if you're a great deal younger than me, and
| you grew up with an Internet where Twitter just always
| existed, then it might represent some earlier version of
| the Internet that is drastically different from what you
| consider to be "contemporary" (though, putting the TikTok
| mention aside, I'm still not sure what that view of the
| contemporary Internet is if Twitter is what we're
| comparing it to).
|
| I guess I'm just so old that I still see Twitter as a
| relatively new phenomenon. Very different from the
| nostalgia that I feel for what the world wide web used to
| be when I was young.
| smileybarry wrote:
| It's gotten _dramatically_ worse since premium started
| rewarding payouts for engagement /views, alongside zero
| moderation. Now there's an actual incentive to be a jerk, a
| spammer, and make repeated bad faith arguments / "just asking
| questions" -- people reply, your tweet is seen by more
| people, you get paid a share of ad revenue.
|
| Then you have the regurgitated messages in replies (AI or
| otherwise) endlessly copied and distorted. Recently it's been
| 5+ pages until I see a real reply, while I have 3000+
| accounts blocked by now. It's getting harder and harder to
| find the real person who wrote that witty reply, and not the
| endless blue check accounts who copy-pasted it to farm
| engagement on their visibility-boosted tweets.
|
| Genuinely getting more and more unusable.
| wcarss wrote:
| Why not, um... stop using it?
|
| Mastodon has a lot of great people, and a lot less of all
| this stuff.
| CaptainFever wrote:
| Not parent, but I did use Mastodon for about a year. I
| ended up moving back to Twitter because:
|
| 1. I just couldn't vibe with the culture there. From my
| POV, Mastodon is made out of pearl-clutchers and
| politics.
|
| 2. So much drama. The FediSearch drama. The Raspberry Pi
| incident. It's just so tiring and you feel like you need
| to walk on eggshells all the time.
|
| 3. So much drama. You would just pray that your admin
| didn't get into a spat with another admin and get
| defederated. You could get a solo server, but that costs
| money and you might get blocked by a large server's admin
| anyway.
|
| 4. So much drama. Pray that your server doesn't shut
| down, because you can't import your posts elsewhere.
| Solo, yes, but costs money.
|
| At least with Twitter, the rules are sort of well known,
| and you can follow anyone there unless they block you
| personally.
|
| I heard BlueSky is good, though. Haven't tried it yet.
| Nostr was also another one, to get around the admin drama
| issue, but it doesn't seem very popular.
| numpad0 wrote:
| Tangential but there's my disappointment to AI in there:
| they haven't found a way to build convincingly human bots,
| let alone a procedural content creation system.
|
| Both AI believers and doomers insist generative AI could
| surpass humans by every single metric that matters in
| social media. All we've got is first Nigerian, then Indian,
| and now increasingly Pakistani spammers desperately reply
| bombing trending posts made by 10 years alpha users, as if
| whoever behind it completely failed to do it and has been
| wasting gullible human spammer candidates faster than
| Aperture Science.
| rchaud wrote:
| > they haven't found a way to build convincingly human
| bots, let alone a procedural content creation system.
|
| The 'they' being the corporations that hired thousands of
| people from the very countries you're associating with
| spammers , to review, correct and train AI models?
| brightball wrote:
| I never did much with Twitter until I took over the Carolina
| Code Conference, but it's been by far the easiest place to
| engage with different tech communities. Just the "Follow All"
| button that appears after I follow something like Gleam or
| Roc is hugely helpful.
|
| I probably spend more time on Twitter now than I ever did
| before.
| strictnein wrote:
| This obviously varies person to person, but I don't find it
| to be that. I carefully maintain who I am following (about
| 600 people and companies, mainly in the infosec / natsec
| space) and when I start to see bad behavior/content I'm quick
| to unfollow people. Also, I only use the Following tab, never
| the For You tab.
| mistermann wrote:
| > It has the power to turn the most interesting and smartest
| people into bitter trolls.
|
| There is tremendous value in having access to large
| quantities of data demonstrating the various failure modes of
| relatively intelligent minds. Nothing is being (publicly)
| done with this data so far, but this will not last forever.
|
| This is something that has always bothered me about the
| /r/SSC community (and others): the "no culture war topics"
| constraint + moderators locking threads when the Rationalists
| start collectively behaving irrationally.
| asdff wrote:
| People should do a lot of things. They shouldn't smoke and they
| should work out, but here we are in 2024 where the phillip
| morris stock has outperformed ibm over the last 5 years and the
| obesity crisis shows no signs of letting up. Knowing something
| is bad is the first step of course but clearly not enough to
| drive a real behavioral change, and we aren't even fully in
| agreement as a society that social media is harmful like how
| cigarettes or obesity are known to be harmful.
| jrm4 wrote:
| Funny, just now I've been playing around with the various tweet
| deleters and trying to get something working; presently I think
| I'm about to settle on something involving a basic screen macro
| recorder thing, like one of the iterations of AHK.
|
| I'm somewhat surprised that this space feels relatively dormant
| compared to the more complex stuff out there.
|
| (APIs suck)
| kypro wrote:
| > What if I could ask patio's archive: "what are some good books
| to read about [topic]" or "what advice would you give to someone
| trying to get a job at Stripe"
|
| Or what if I could ask: "Given Omer Shehata's Twitter history,
| formulate a phishing scam that he would be likely vulnerable to".
|
| The problem I see with here is that there are far more bad actor
| use cases for identifiable user data than good. In my opinion the
| main reason most social networks have stopped doing public by
| default and now do private by default is because not doing so
| opens them up to Cambridge Analytica type scandals where people
| don't realise what they're signing up for.
|
| Personally if you do this, I would be very clear with your users
| that by submitting their data it will be made available publicly
| in an identifiable form. And that even if they revoke their data
| from your service it's possible for their data will continue to
| be archived by others, possibly for malicious reasons.
| theexgenesis wrote:
| Dev here.
|
| Cognitive security vulnerabilities like this are the thing I'm
| most concerned about. I think it's right to be very upfront
| about risks like these, and I'm even considering if we want to
| walk back the fully public thing and make it private / invite-
| only instead.
| abdullahkhalids wrote:
| To play the devil's advocate. If you were running a large public
| forum, and you knew that many companies had started to scrape all
| data off your site, and were going to cumulatively make billions
| off that data, and some of those billions will come from
| polluting your forum with crap content, would you continue
| running your site in the open?
|
| What is the game theory here? Twitter cooperates and OpenAI
| defects, and we call that a win?
| KerrAvon wrote:
| This is a problem for literally every website that isn't
| completely paywalled. Twitter is not special in this regard.
| miki123211 wrote:
| Twitter was "special" because they actually _had_ an API.
|
| No other social media site (besides Reddit) had one. Facebook
| kind of tried, although theirs was always a lot more limited,
| and got shut down when it turned out Cambridge Analytica used
| it for widespread election fraud[1].
|
| Both Twitter and Reddit did basically the same thing at
| roughly the same time, Twitter just had the misfortune to be
| under the control of Elon Musk, so the move was perceived as
| ideological.
|
| [1] As it later turned out, Cambridge Analytica was basically
| a "nothing burger", they claimed to be able to accomplish a
| lot while, in reality, accomplishing very little. The damage
| to interoperability was already done, though.
| shadowgovt wrote:
| Indeed. CA's impact on election outcomes was likely
| negligible; Antonio Martinez's "Chaos Monkeys" makes the
| case that Trump using CA was more indicative of the overall
| notion that Trump's team was spending money to try a bit of
| everything (online and offline) and the Clinton campaign
| wasn't. Their tactical failure was believing they could
| redirect the money to down-ticket races because Clinton /
| Trump was such an obvious matchup that they didn't need to
| spend to win.
|
| What CA did show was that Facebook's statements about
| protecting user privacy were fundamentally incompatible
| with the way their API worked, so they had to shut it down
| because the alternative would have been to just sort of...
| Let it hang in the air that it wasn't hard for a third-
| party to build a system to completely bypass user intent in
| scoping their information.
|
| (I had the misfortune of trying to write a Facebook app
| about fifteen years prior, and that was my takeaway at the
| time also... "Do people, like, realize that their whole
| process for protecting scraping the social network via
| third-party app integration is the honor system?" Turns out
| people didn't).
| radarsat1 wrote:
| > Do people, like, realize that their whole process for
| protecting scraping the social network via third-party
| app integration is the honor system?
|
| That sentiment goes back pretty far in Facebook's history
|
| [1] https://www.yahoo.com/news/mark-zuckerberg-branded-
| early-fac...
| abdullahkhalids wrote:
| To continue playing the devil's advocate:
|
| 1. Experts can comment, but I think the value of multi person
| conversational data from forums is uniquely valuable and in
| short supply relative to just blogposts/news stories on the
| internet.
|
| 2. The absolute economic value of the entire corpus of
| Twitter is much more valuable than any single boatforum.com
| like website. So Twitter has a much more incentive to lock
| itself down than boatforum.com.
| xNeil wrote:
| Yes, and Twitter isn't reacting specially in this regard. See
| Reddit hiking API prices by an absurd amount.
| ChocMontePy wrote:
| Fact Check: Reddit didn't hike API prices by an absurd
| amount.
|
| That was was the story spread far and wide by the Apollo
| app developer that was believed by the gullible and angry
| Reddit masses.
|
| But Reddit actually set a reasonable API price, as
| evidenced by the fact that a year and a half later there
| are still five 3rd party apps running on reasonable
| subscriptions:
|
| Infinity For Reddit
|
| Nara For Reddit
|
| Narwhal 2
|
| Now for Reddit
|
| Relay For Reddit
| OmarShehata wrote:
| Alternatively, build a private invite only dataset of specific
| communities. Scale horizontally instead of having one single
| central dataset! That's still a huge win for user ownership.
|
| It doesn't have to be one company controlling who gets access,
| the users can decide this
| asdff wrote:
| The problem is that the users and their capital are not
| organized enough to suggest or demand for this. On the other
| side of the table we have the "establishment" which may not
| be formally organized but has enough shared incentives among
| itself where the outcomes aren't any different as if it were
| indeed formally organized and pooling resources.
|
| In this sense we have this intractable push and pull going on
| with just about any community of sufficient size. We have the
| users who might want or should want some privacy and respect
| and other such benefits, and then we have the people who
| actually invest and build these platforms, who are
| incentivized to deliver other things than what the users best
| interest might be. Either you empower users to have more
| money to be able to roll their own solutions, or you try and
| set up a world where the incentives of capital perfectly
| match the needs of the individual or collective of
| individuals, which is probably impossible to do.
| numpad0 wrote:
| You having a problem with companies making billions? Why not
| just regulate that, instead of the latter "...and cumulatively
| make billions" part?
|
| If companies making billions isn't the part that's problematic,
| the billions part can be left out, and the real problem can be
| discussed instead.
| rurp wrote:
| It's tragic that the LLM craze OpenAI kicked off is threatening
| to ruin one of the greatest common goods ever invented in the
| open internet. But hey, at least a handful of giant
| corporations and investors are making money, so I guess that
| counts as a win.
| bravetraveler wrote:
| Practically speaking, _' some'_ of the money is the same or
| just as good as _' all'_ of the money in terms of a functioning
| economy and not pure capitalism.
|
| People pirate software, yet people still develop and sell it.
|
| Yes, it's not the most profitable thing done alone. Only for
| those who find the nice combination or feedback loop of 'fit',
| demand, improvement, and expansion.
|
| If you can make billions off the thing, you can presumably
| handle some GeoIP/rate limiting... or simply, _not caring_.
| Anything that falls through the cracks is categorically
| insignificant to your grand nature.
|
| To justify it, if one must, consider it a trial. As your
| friendly neighborhood dealer would say: _" The first taste is
| free"_.
| bravetraveler wrote:
| Neat boundaries on the Town Square
| nunobrito wrote:
| It has been difficult to rescue data from Twitter even before
| purchase. On our case it was relevant because this is online
| digital history for the people in my country.
|
| The only thing we can is motivate more people to use open
| platforms like NOSTR where API or data/identity handling is
| completely different.
| danielodievich wrote:
| The only useful thing on Twitter that I ever saw was the lovely
| and tender Dog Rates https://x.com/dog_rates. You could read
| anonymously and be all aww and schucks about all those good dogs.
| They've thankfully stopped engaging with this cesspool that it
| became and moved somewhere else, Instagram perhaps? Somewhere
| where I can't read without an account, so I don't read it
| anymore.
| vlod wrote:
| Here's a few more in my feed (no affiliation)
|
| - https://x.com/contextdogs (out of context dogs)
|
| - https://x.com/_B___S (B&S)
|
| - https://x.com/buitengebieden (Buitengebieden)
| KomoD wrote:
| > They've thankfully stopped engaging with this cesspool that
| it became and moved somewhere else
|
| No, they haven't. They're still actively posting on Twitter.
|
| It's just that if you are viewing a profile logged out you get
| the most popular tweets instead of the most recent ones.
| bangaladore wrote:
| Here's a thought: someone "trustworthy" should maintain a Chrome
| extension or Tapermonkey script that automatically scrapes data
| from various social media sites in a fully anonymized fashion. As
| people browse Twitter, Reddit, or XYZ, the posts/comments are
| sent to some aggregation system. It might be against TOS, but
| certainly far less than scraping, and you couldn't tell, as it's
| the user driving what gets scraped.
|
| I don't use Twitter often, but I'd run something like that if
| there were strong anonymity guarantees. Seems like a win-win for
| everyone.
|
| Does anything like this exist today?
| kllrnohj wrote:
| Not quite the same, but https://returnyoutubedislike.com/ is in
| a similar vein of an extension crowd-sourcing a return of data.
| yojo wrote:
| Reminds me a little of RECAP (https://free.law/recap), an
| automated scraper/saver/sharer for PACER (the US court
| electronic records system).
|
| Obviously the content is very different, but the technology is
| basically doing what you're talking about, minus anonymizing
| the data.
| bangaladore wrote:
| Good find. Yeah, pretty much exactly like this.
| nicbou wrote:
| I was thinking of that exact strategy for two scraping problems
| I have. It would be a good way to gently scrape Berlin.de for
| appointments, and Immobilienscout24 for new flats.
|
| From what I can tell, it would work fine so long as the user is
| actively looking at those pages.
| bangaladore wrote:
| Yeah, something generic to work for any use case would be
| nice, but privacy becomes more difficult as you need to
| tailor the situation to each site to maintain privacy (i.e.
| only pull the information on the apartment/flat listing, or
| public tweet or reddit comment, etc...)
| allkindsof wrote:
| Isn't Twitter just a bunch of Elon alt accounts at this point?
| fb03 wrote:
| At this point, just nope out of it and use something like Bluesky
| mrkramer wrote:
| Elon is trying really hard to destroy Twitter, isn't he?
| xyst wrote:
| switched to mastodon, bluesky long ago.
| urda wrote:
| I've been very happy with Mastodon.
| molticrystal wrote:
| Twitter was originally a microblogging service that had rss feeds
| to syndicate things or monitor the microblogs of people/companies
| that were interesting, it has gone way far off into the fields.
|
| Same journey reddit is making, starting after it prepared to go
| public.
___________________________________________________________________
(page generated 2024-09-18 23:01 UTC)