[HN Gopher] Show HN: Download Twitter data without API keys
___________________________________________________________________
Show HN: Download Twitter data without API keys
In April last year I started thinking about using Twitter in a
smarter way. I wanted to do analytics on my tweets and find out
more about people following me on Twitter. What kinds of things do
people who follow me like and retweet? I decided to dig into the
data and find out. When I went to try and download Twitter data in
raw form I found I quickly got bogged down writing API wrangling
code and fiddling with API keys. I just wanted to crunch some data
but here I was wrangling Twitter's API. This was such a frustrating
experience it suddenly looked like an opportunity to me. Was there
room for a product here? A product which does one simple thing:
help people extract Twitter data painlessly without writing any
code. I did some research and discovered some tools that purported
to do this, but none of them worked well for my use-case and all
were badly designed and/or expensive. I decided to take a shot at
it. I worked on this as a side project in a "calm company"
fashion. Each week for 26 weeks I would put aside one day to chip
away at the features. I tried not to think about how the issue
tracker was filling up more and more. Several times I pared back
the feature set to try to really focus on the core use-case. When
my first user reached out and engaged I knew I might be onto
something. I kept posting progress on Twitter and a few more people
started to use it each week. Some of them came back, hinting at
possible user retention. I hired a writer to write some articles to
help with SEO and I kept working towards and MVP that I could use
to test the market. Finally the day arrived where all of the
critical issues in my issue tracker were closed. That meant it was
launch day. That was yesterday. So here I am releasing this on
Hacker News to you, dear reader. God speed little micro-SaaS, may
the winds of fortune be at your back.
Author : chr15m
Score : 41 points
Date : 2022-02-03 04:28 UTC (1 days ago)
(HTM) web link (tweetfeast.com)
(TXT) w3m dump (tweetfeast.com)
| rank0 wrote:
| I like the spirit of this, but the monetization strategy rubs me
| the wrong way. You're basically reselling a UI wrapper for the
| twitter API.
|
| Nevertheless, good luck and I hope they don't shut you down too
| fast.
| ugjka wrote:
| 3... 2... 1... cease & desist
| [deleted]
| ryankrage77 wrote:
| Charging for web scraping feels scummy somehow. Can't quite put
| my finger on why, since I wouldn't feel guilty about scraping
| twitter myself.
| michaelmior wrote:
| I'm not too familiar with the Twitter terms of service, but I
| would be surprised if selling access to their data wasn't a clear
| violation.
| PeterBarrett wrote:
| Well they're not selling access to data, the data is freely
| available. They're selling a service that packages that data
| for you.
| vasco wrote:
| Data which probably was obtained through the API which from
| my interpretation of
| https://developer.twitter.com/en/developer-
| terms/agreement-a... is not permitted.
| throwaway2016a wrote:
| If obtained through scraping vs the API it also isn't
| allowed via the regular ToS https://twitter.com/en/tos and
| their robots.txt
|
| So either way sounds like a ToS violation. Which Twitter is
| usually pretty aggressive about shutting down.
| altdataseller wrote:
| I can't see how this is true as there are tons of companies
| selling access to data obtained from the Twitter API such
| as social media monitoring companies (Sprinklr, Meltwater)
| simonw wrote:
| Those companies have business development deals in place
| with Twitter that authorize them to do this.
| altdataseller wrote:
| There are plenty of other non-enterprisey companies that
| sell Twitter data as well*
|
| *I was a founder of one of them.
| michaelmior wrote:
| That's true in a sense. But I would still be paying them
| money to receive data from Twitter. So in a sense, yes, I am
| paying for Twitter data. In any case, I have a hard time
| seeing how this wouldn't violate the ToS.
| nojito wrote:
| You can't package and redistribute tweets only tweet ids.
| cyral wrote:
| As a counterpoint, stock data is freely available (through
| online websites, your broker/app, etc), yet it costs
| thousands of dollars per month to actually license and
| acquire such data through the proper channels. Trying to
| build a business based on scraping such data (or even using
| your broker's free personal-use API) will get you in trouble
| pretty quick by the exchanges.
| antisthenes wrote:
| > I just wanted to crunch some data but here I was wrangling
| Twitter's API.
|
| > I worked on this as a side project in a "calm company" fashion.
| Each week for 26 weeks I would put aside one day to chip away at
| the features.
|
| So you wanted to crunch some data...and instead of spending 3-4
| hours setting up the API properly...you instead spent 26 days
| writing an unofficial hack?
|
| It seems like what you really wanted was to spend 26 days on a
| coding hobby, not actually crunching some data.
| meigwilym wrote:
| Great idea (despite the dubious legality).
|
| Perhaps a section on _what_ can be done with the data? Your
| potential clients are probably the type of users who don 't know
| why they'd need such a tool. And an explanation of the sentiment
| analysis?
| donkarma wrote:
| twint does this for free
| throwawayrev3 wrote:
| It's last commit was on March 2021. Is the project abandoned?
| torbTurret wrote:
| That's not even that long ago? What?
| throwawayrev3 wrote:
| Idk. I just genuinely wanted to know. Because there are
| many open issues. There is even an issue asking whether
| it's abandoned.
|
| I was thinking because it's twitter and it doesn't use an
| API, twitter might introduce breaking changes and Twint has
| to catch up often. Just like NewPipe, yt-dlp etc does for
| YouTube.
| [deleted]
| spupe wrote:
| What is the advantage of your implementation versus Twint, which
| is free?
| throwawayrev3 wrote:
| Twint's last commit was on March 2021.
| iKlsR wrote:
| What assurances can you give that you will be around in a year
| because I think this could get crippled easily regardless of how
| you are doing the fetching on the backend (scraping, api,
| nitter). I'd love something like this to not have to wrangle with
| the api myself but it's giving uneasy vibes for longevity.
|
| Edit: I tried it, got the following for "tweets from a search
| result"
|
| Error: Request failed with code 429
|
| Request exceeds account's current package request limits. Please
| upgrade your package and retry or contact Twitter about
| enterprise access.
| mritchie712 wrote:
| This will be gone in days if anyone uses it, it's clearly
| against the terms.
| nannal wrote:
| So they really were trying to sell access to the content you
| could fetch with a twitter api key?
| new_stranger wrote:
| > Up to 10k rows of data.
|
| uh, how is 10k tweets (1 second of data?) enough?
___________________________________________________________________
(page generated 2022-02-04 23:02 UTC)