[HN Gopher] Show HN: I reverse engineered X to Read Threads with...
___________________________________________________________________
Show HN: I reverse engineered X to Read Threads without Any Account
as Articles
Hi fellow hackers, I'm Ved, I came across the pain point of having
a reader that could provide threads as articles after scrolling
them many times and missing the context of the original info
shared. To solve my itch I decided to build a thread reader
without Ads(all existing sol. were full of them), since X/Twitter
API was very pricey, I did some reverse engineering to convert
rendered X.com/twitter pages to an API and host it in a VM (which
obv. was dead cheap). All of your data is saved in LocalStorage.
Including your comments and saved threads(there is a known issue
with Linux and MacOS browsers i think). Code is open-sourced at :
https://github.com/vednig/unlaceapp App is available at :
https://unlace.app There are two views inbuilt in app
https://unlace.app/?url=https://x.com/ianwcrosby/status/1872...
https://unlace.app/thread/ianwcrosby/status/1872724231999381...
Reverse Engineered X API is available at
https://xapi.betaco.tech/x-thread-api?url=https%3A%2F%2Fx.co...
Meanwhile you can enter any twitter thread url on
https://unlace.app to get a clean and read friendly article.
Here's the stack(if anyone's interested): - NextJS - FastAPI -
Selenium That's all folks, looking towards a productive feedback
session.
Author : vednig
Score : 32 points
Date : 2024-12-30 08:24 UTC (14 hours ago)
| garciasn wrote:
| Interesting and much needed--I absolutely refuse to go to Twitter
| links anymore; why bother, they don't usually work unless you're
| logged in. So, I clicked some of the links and, at this time,
| they don't load in a reasonable amount of time, I assume due to
| the Hug of Death or whatever it is we call that here.
|
| This one loaded:
| https://unlace.app/thread/ianwcrosby/status/1872724231999381...,
| but oddly it's duplicating the thread.
| jimmySixDOF wrote:
| I had the same experience and suggest a loading progress bar of
| some sort. I also didn't see any of the Selenium process on
| their Github so can't see exactly how the tweets are processed
| for plain text cleaning. The Twitter/x API is so priced out of
| the market these types of work arounds are necessary
| inconveniences so thanks to the dev for putting this together!
| vednig wrote:
| backend server's code is now available at
| https://github.com/vednig/x-api , you can have a look there
| for more info.
| vednig wrote:
| Backend server's source code is now located at
| https://github.com/vednig/x-api ,currently the script waits for
| complete page to load and then extracts info to send out a
| response.
|
| Since each new request takes high time and processing power, it
| was only logical to store responses and cache them for timely
| usage.
|
| There's also a bit of a workaround required to get videos from
| X.com as they only load their direct URL when focused hence
| parsing html couldn't work, so I had to log network traffic to
| ensure I get their HLS links too.
| derdi wrote:
| > I absolutely refuse to go to Twitter links anymore; why
| bother, they don't usually work unless you're logged in.
|
| Replace "x.com" by "xcancel.com" (or some other still working
| Nitter instance) and you're set.
| https://xcancel.com/ianwcrosby/status/1872724231999381790
| vednig wrote:
| Xcancel is also a great service, it's a complete X
| replacement.
| garciasn wrote:
| > still working Nitter instance
|
| This is the issue with Nitter instances; they seem too
| ephemeral for reliable use.
| vednig wrote:
| Repetition occurs due to m3u8 requests I'm logging for videos,
| which require twice the reload of the page, it can be avoided
| by updating some code.
|
| And since it's highly cached based on unique url requests, it
| persists, I'm thinking of having a purge cache option too, I'd
| have to think how to implement it so as to prevent abuse as
| well.
| toomuchtodo wrote:
| Great work! Can I chip in to support archiving unrolled threads
| to the Internet Archive and archive.today (it's a trivial HTTP
| request to do so)? I asked threadreaderapp.com for this to radio
| silence.
| vednig wrote:
| Sure, service is free for all(it's highly cached, so it
| shouldn't be a problem), code is open source and if it helps
| that would be great.
|
| Here are few things to consider:
|
| - First unique request for X might take time (time increase is
| directly proportional to number of comments on a post)
|
| - If Internet Archive and Archive.today have a limit for
| crawled request this may affect them (only first time)
|
| - Links to profile in Unlace are currently not 100% accurate
| all the time
|
| I'm currently looking for ways to resolve these issues, in the
| server. Meanwhile feel free use it as you may, there's no T&C
| or Privacy Policy in service because there's nearly zero data
| collection and infinite use cases to explore(star trek
| reference).
| matcha-video wrote:
| Very cool! I'm very impressed that Twitter hasn't detected your
| selenium usage, it seems like they're really trying to push
| people towards their paid API.
|
| I made a similar tool, except it's a browser extension that lets
| you download threads as markdown (then it's up to you to dump
| them into a DB or train a model on them or whatever), I'd love to
| get your feedback
| https://chromewebstore.google.com/detail/thread-to-markdown/...
| (firefox version also available)
|
| Edit: Example output using tweet thread mentioned elsewhere in
| this post https://sharetext.io/c04b7c82
| vednig wrote:
| Selenium usage is similar to browsing patterns of real person.
| (I hope they don't ban my account, if they see this)
|
| Your extension somehow crashed on my system while downloading a
| thread(i couldn't see download button), I was able to get 1 md
| file downloaded though and I'd say it does it's job nicely, in
| the highest manner.
|
| Feature Request: Make download button compatible with Dark
| Mode(currently it stands out loud)
| matcha-video wrote:
| You're a true user! I just put up a small change that should
| improve reliability, let me address dark mode as well.
|
| May I ask what you use the markdown for after its downloaded?
| Would you modify the output at all to make it better fit your
| use case?
| vednig wrote:
| I'd use it somewhere in Github or Dev.to where there is
| support for MD, though I'm opening it using VS Code which
| doesn't has direct MD support. Or I'd like to archive it
| for some reason.
|
| I downloaded this tweet, for example, this can give you an
| idea about potential usecases
| https://x.com/ThePrimeagen/status/1873778477448782035
| matcha-video wrote:
| Dark mode is now supported as of v0.3.4 :) Thanks for
| your feedback
| vednig wrote:
| Status Update: Server is down, because of excessive requests. I'm
| trying to get it back up.
| vednig wrote:
| Status Update2: Server is back up and running
| vednig wrote:
| Status Update: Live Stus is now available at
| https://stats.uptimerobot.com/e01zgAwwLr
| HelloUsername wrote:
| https://unlace.app/?url=https://x.com/ianwcrosby/status/1872...
|
| Doesn't load with Javascript disabled
|
| https://unlace.app/thread/ianwcrosby/status/1872724231999381...
|
| Does load with Javascript disabled.
|
| Why one yes, the other no?
| vednig wrote:
| First one is rendered client side while second one is server
| side rendered returned completely, for SEO purposes etc.
|
| Both use nearly the same logic. You can view them in the repo
|
| 1. https://github.com/vednig/unlaceapp/blob/main/app/page.tsx
|
| 2.https://github.com/vednig/unlaceapp/blob/main/app/thread/%5B.
| ..
___________________________________________________________________
(page generated 2024-12-30 23:02 UTC)