[HN Gopher] Twitter improves API usage for researchers
___________________________________________________________________
Twitter improves API usage for researchers
Author : jansenmac
Score : 109 points
Date : 2021-01-27 13:13 UTC (9 hours ago)
(HTM) web link (blog.twitter.com)
(TXT) w3m dump (blog.twitter.com)
| thih9 wrote:
| I wonder how are they going to enforce their rules, e.g. non-
| commercial use. I assume this will require some monitoring to be
| effective. Large scale Twitter API access is typically pricey,
| malicious actors might try to buy or steal researcher's
| credentials to cut costs.
| cocktailpeanuts wrote:
| I recently tried to sign up for Twitter API and the process is
| nothing like what it used to be. You have to give them a lot of
| information to even qualify, such as what you're going to use
| it for. It used to be that those were just some fields you need
| to fill out and you could sign up immediately. But nowadays the
| application process requires a direct approval from their team,
| which means they're monitoring every API account like Apple
| does with their app store. And if you like about your usage you
| are probably liable
| williesleg wrote:
| Twitter is officially in the shitter.
| blindm wrote:
| I always wondered: how many of these tweets are just Justin
| Bieber fandom type posts, bots, spam, or other dross? Twitter is
| infamous for its bad signal to noise ratio. These researchers
| need to write algos to filter out all the noise
| sigil wrote:
| A lot! I ran named entity recognition on the Twitter garden
| hose back in 2012. The top entities were Bieber and the Jonas
| brothers.
| adolph wrote:
| Are there researchers out there using bot accounts to
| prospectively experiment with social media researchers?
| edent wrote:
| What's wrong with that? If you wanted to investigate, say, the
| rise of The Beatles - wouldn't you love to have access to the
| random thoughts of their fans in the 1960s?
|
| Similarly, if you're researching bots and spam and how they
| manipulate people & markets - this is still useful.
| guerrilla wrote:
| Is it me or is Twitter making a lot of announcements this month?
| jansenmac wrote:
| Twitter makes it easier for Researchers to use tweets and the
| Twitter API for research.
| tester34 wrote:
| basing on those UI screens, then why research is always
| associated with academia?
|
| there's a lot of strong people especially in CS who do not work
| with academia and still work on interesting stuff
| anonymousDan wrote:
| Because it's easy for Twitter to draw the line between
| commercial and non-commercial use?
| leephillips wrote:
| Absolutely. The limitation to people associated with academic
| institutions is pretty old fashioned (and also, from another
| point of view, modern).
| adolph wrote:
| Maybe it has to do with control. Student==likely low impact.
| Faculty==likely uncontroversial. API==looks like transparency.
| artembugara wrote:
| We've been running our startup [1] in this "media research
| industry" for just a while. We're on the classic media use case
| side.
|
| It is true that the vast majority of "research" is done by non-
| academics. Lots of companies doing market research want to mine
| media data.
|
| Still, I believe that this "social media research" is a bit
| overvalued. There was this wave of "social media is the primary
| source where information appear". But now many realized how
| freaking difficult to separate this data from the noise comparing
| to traditional news published by journalists.
|
| Also, take a look on this article [2] about how Dataminr sells
| insights from Twitter data to foreign governments (2017). Seems
| like just a way to punish the opposition channels.
|
| [1] https://newscatcherapi.com/
|
| [2] https://www.theverge.com/2017/1/27/14412014/dataminr-
| twitter...
| beefman wrote:
| But not for suspended accounts
|
| https://www.reuters.com/article/us-twitter-product/twitter-g...
| waheoo wrote:
| Could have saved you some work, the research results are that
| Twitter is very liberal, less conservative and nobody has seen a
| libertarian for days.
| wlesieutre wrote:
| I'd think libertarians would love twitter, deplatforming is the
| free market at work and the government has no right to make
| them do business with anyone they choose not to
| peytn wrote:
| And one might think liberals, putting liberty above order,
| would be aghast at silencing opponents to maintain order, and
| yet here we are. Our political theater has gotten pretty
| weird. The whole thing is looking more and more like
| unprincipled tribes vying for power.
| pjanoman wrote:
| Silence is a strong word -- I'm certainly still able to
| hear news from Donald Trump, even if he was banned from
| Twitter.
| coding123 wrote:
| As someone that doesn't try I still hear news _about_
| Trump but no longer _from_ Trump. That 's a big (welcome)
| change.
| sojournerc wrote:
| You're right about Trump. Most people don't have the
| platform he has.
|
| Many dissenting voices have been removed from the
| conversation, and you wouldn't even know they are
| missing.
| croon wrote:
| Except I keep hearing about it constantly, everywhere,
| including Twitter. And yet I haven't seen a single one
| being banned for discussing fiscal policy, less
| regulation, conservative views on social programs etc.
| "Conservative voices [being] silenced" are almost always
| some variation of spamming evidently real-world damaging
| conspiracy theories or clear ToS violations.
| sojournerc wrote:
| I didn't say conservative - I said dissenting. Although I
| imagine many conservatives have learned to self-censor so
| they can remain part of the conversation there.
|
| How about asserting that men and women are different
| biologically? Is that a conspiracy theory?
|
| Or is it simply a dissenting viewpoint from the group-
| think of Twitter that was silenced...
|
| https://thefederalist.com/2018/11/25/twitter-permanently-
| ban...
| croon wrote:
| > In late 2018, Twitter changed its policy on hateful
| conduct and harassment to officially prohibit
| intentionally calling a trans person by the wrong
| pronouns or using their pre-transition names.[62]
| Beginning in August 2018, Murphy stated that her Twitter
| account was locked more than once after she tweeted about
| issues involving trans women.[63] Twitter permanently
| suspended Murphy's account in late November 2018, after
| she referred to Jessica Yaniv, a trans woman, as
| "him".[64][65][66] On February 11, 2019, Murphy filed a
| lawsuit against Twitter in response to her banning.[67]
| The suit was dismissed in early June, but Murphy stated
| that she intended to file an appeal.[68][69]
|
| Clear ToS violation. Which proves my point.
|
| Sidenote: The Federalist itself got its account banned
| from twitter for spreading covid misinformation.
| sojournerc wrote:
| This thread started with:
|
| > And one might think liberals, putting liberty above
| order, would be aghast at silencing opponents to maintain
| order, and yet here we are.
|
| You aren't liberal, that's OK, and Twitter isn't a
| liberal platform, and definitely has a political bias
| towards the progressive left.
|
| I'd rather make decisions for myself than let Twitter
| tell me what is or isn't misinformation.
|
| > Sidenote: The Federalist itself got its account banned
| from twitter for spreading covid misinformation.
|
| Thanks for proving my point
| croon wrote:
| > Twitter isn't a liberal platform, and definitely has a
| political bias towards the progressive left.
|
| [citation needed] seeing as your previous example fell
| flat.
|
| > > Sidenote: The Federalist itself got its account
| banned from twitter for spreading covid misinformation.
|
| > Thanks for proving my point
|
| I fail to see your point. They spread misinformation that
| would get people killed, and were banned for it.
|
| I've not in my 30 years on the internet spent more than a
| passing moment on a BBS, forum, social media that let
| anything go, nor would I want to as I've seen what it
| devolves into.
|
| You have not displayed any basis for your assertion that
| the moderation enforcement on twitter is biased.
|
| What point did I prove for you?
| andrewzah wrote:
| "silencing opponents"
|
| Nobody here is silenced. You do not have a right to a
| twitter account. People can, and do, make accounts
| elsewhere.
|
| Literally anything Trump does can and is covered by the
| media; this is exactly the opposite of being "silenced".
| 5560675260 wrote:
| Twitter should be judged by the way it governs its platform.
| And from libertarian perspective it's governed poorly. Sure,
| under current laws they can get away with deplatforming in
| the way they do now, but there is nothing commendable or
| desirable about it for libertarians specifically.
| jijji wrote:
| you would think from the amount of republican/conservative
| accounts they have banned that they have some AI or parser
| dedicated to banning these types of voices
| Cthulhu_ wrote:
| Where's your sources, numbers, methodology? I mean anyone can
| make a kneejerk statement based on their perception (read:
| bubble), but that's not science.
| ceejayoz wrote:
| You should follow different people.
| minimaxir wrote:
| It's a shame access to this API is limited to academic
| institutions, as many social media/misinformation researchers are
| now independent or affiliated with journalistic institutions.
| trident5000 wrote:
| Misinformation and troll farms on Twitter start with
| unrestricted API usage to the masses.
| Nextgrid wrote:
| Misinformation & troll farms appear to do just fine with the
| current restricted API.
|
| What evidence is there that read-only API access will make it
| significantly easier for them (enough to outweigh the other
| upsides)?
| trident5000 wrote:
| So you're telling me API's are not useful while at the same
| time telling me that API's are useful. Trolls want API
| access for the same reason researchers want access. You
| could argue research is the first step to becoming an
| effective troll.
| Nextgrid wrote:
| I don't see API access as being necessary for a small-
| scale trolling operation, and large scale operations have
| enough resources to work around the lack of API by
| scraping.
|
| I'm not saying that API access is completely useless, I
| was raising the question of whether the _potential_ (and
| relatively small) benefit to trolling outweighs the major
| upsides of API access being available for all.
| adolph wrote:
| In what research contexts is API usage valid instead of scraping
| a view more similar to what people experience? If the Twitter
| site and API are retrospectively cleared of removed/suspended
| accounts with large impact, how does that affect retrospective
| studies?
|
| Are there ethical implications of working with Twitter to gather
| data? Despite Twitter TOS, legal, IRB ok, are there informed
| consent issues in studying the artifacts of social media use?
| dharmab wrote:
| One easy example is language, for example tracking the spread
| of new words or other language constructs. You don't care how
| the site looks, you care about the text that was previously
| input.
| fnord123 wrote:
| Until now, none I think. The API only gave a partial view while
| scraping offered all tweets for a particular search term. The
| scraper had to be clever to juke the anti scraping systems but
| you would get a more complete data set than using the API.
|
| And the streaming API was terrible. Even if there was no data
| on the stream you could consume tens of gigabytes of bandwidth
| a day. Dreadful.
___________________________________________________________________
(page generated 2021-01-27 23:01 UTC)