Post ASvDpXBS4p2tpEqqAa by Tedspence@indieweb.social
(DIR) More posts by Tedspence@indieweb.social
(DIR) Post #ASv2FL7cLWDiUmb29o by ken@toots.schwanksta.com
2023-02-21T23:05:18Z
0 likes, 0 repeats
Does anyone on here work on the Twitter API or know someone? Because it looks like with recent Twitter API changes, @ProPublica can't track deleted tweets from politicians anymore w/ Politwoops (projects.propublica.org/politwoops/). The API we used for years has been deprecated, and the v2 API simply...does not seem to report deletions correctly. The new method to ingest tweets also seems very limited. And I'm not sure we actually understand what the paid model would be!
(DIR) Post #ASv2XcfppLne90JrI8 by en3py@infosec.exchange
2023-02-21T23:08:35Z
0 likes, 0 repeats
@ken @ProPublica sounds more like a feature rather than a bug. Being able to delete a tweet and leave no trace. No tweet? That's untold.
(DIR) Post #ASv2dVXzEjOlmMSFEW by JizzelEtBass@kolektiva.social
2023-02-21T23:09:39Z
0 likes, 0 repeats
@ken @ProPublica I know that the #NaziBirb app disabled API integration across the board...
(DIR) Post #ASv2fmx1V6CmKqFjKi by ken@toots.schwanksta.com
2023-02-21T23:10:05Z
0 likes, 0 repeats
@en3py @ProPublica I’m not sure how a third-party app would know how to remove your tweet but then again I guess they’re getting rid of those too
(DIR) Post #ASv2poCPiMVH6PEI2S by kg6hxm@social.makerforums.info
2023-02-21T23:11:52Z
0 likes, 0 repeats
@kenCan you just check links to see if they return valid tweets?@ProPublica
(DIR) Post #ASv2ty2369VmkyMehc by ken@toots.schwanksta.com
2023-02-21T23:12:38Z
0 likes, 0 repeats
@kg6hxm @ProPublica we track a loooot of tweets :)
(DIR) Post #ASv32NgTbmQ2dEHnY8 by kg6hxm@social.makerforums.info
2023-02-21T23:14:09Z
0 likes, 0 repeats
@kenYeah... I figured that, but a solution is a solution until a better solution comes around. I wouldn't trust anything that Musk has control over anymore for reliability.@ProPublica
(DIR) Post #ASv385aLoPrMEk8u9I by ken@toots.schwanksta.com
2023-02-21T23:15:12Z
0 likes, 0 repeats
@kg6hxm @ProPublica unfortunately given the limitations on how often you can check URLs it just doesn’t scale to what we need to do
(DIR) Post #ASv3Ib17nv5acIfylM by sgurvey@journa.host
2023-02-21T23:17:01Z
0 likes, 0 repeats
@ken I don't think the new Twitter wants you to be able to track those things.
(DIR) Post #ASv3LLyuABv0qvx7Xk by kg6hxm@social.makerforums.info
2023-02-21T23:17:35Z
0 likes, 0 repeats
@kenBummer. I was going to offer some PowerShell code that could pull/ping/report for ya, even multi threaded that could take a while. Maybe base it off the account level and pull all active tweets and see what dropped out?@ProPublica
(DIR) Post #ASv3rV0a8WSw7H46M4 by colo_lee@universeodon.com
2023-02-21T23:23:23Z
0 likes, 0 repeats
@ken @ProPublica pretty sure no one knows what the supposed v2 paid model will be ...
(DIR) Post #ASv3y0Uzy5FfWJOaDQ by Green_Footballs@mastodon.social
2023-02-21T23:24:34Z
0 likes, 0 repeats
@ken @ProPublica Yes, the streaming API is no more. Most of the basic API functions still work but it’s anybody’s guess for how long, and the non-streaming API probably won’t work for ProPublica - way too limited.And they still haven’t announced what the API paywall will look like. Elon Musk is just leaving developers to twist in the wind.
(DIR) Post #ASv3yPujTXguLMczIW by ManyRoads@mstdn.social
2023-02-21T23:24:38Z
0 likes, 0 repeats
@ken @ProPublica Perhaps @ProPublica should endorse and adopt an opensource API/ platform. Might I suggest @Mastodon
(DIR) Post #ASv40iMdqiOE92mHVw by otheorange_tag@mstdn.social
2023-02-21T23:25:03Z
0 likes, 0 repeats
@ken @ProPublica No, but doesnt library of congress and wayback machines do this? May be time to screen scrape and mouseclick. I used to do this to autoblock magas etc when I couldn't be bothered to learn someones transient apis
(DIR) Post #ASv4MPrfJLNh8kdY6S by ken@toots.schwanksta.com
2023-02-21T23:28:59Z
0 likes, 0 repeats
@ManyRoads @ProPublica @Mastodon we clearly like Mastodon, but unfortunately we must track where the politicians post
(DIR) Post #ASv4a0O2pjaEB0GITI by davidaugust@mastodon.online
2023-02-21T23:31:23Z
0 likes, 0 repeats
@ken @ProPublica As twitter’s systems degrade (digital & more), their behavior will become more erratic, inconsistent & unpredictable. It’s sand, ever shifting/unstable. No reason to believe Twitter understands what paid model will be. Unpleasant but relying on any Twitter system to analyze Twitter introduces errors & corrupted data. A scraping system, admittedly more demanding to deploy, to load public profiles & capture, analyze them without further use of Twitter is most stable option.
(DIR) Post #ASv5L4vxkGXTtCZaSW by DarryB@mastodon.world
2023-02-21T23:39:53Z
0 likes, 0 repeats
@ken @ProPublica I see what's going wrong here Ken. You're using Twitter, that's your problem bud.
(DIR) Post #ASv5ZAzk0WsQk13wgq by ManyRoads@mstdn.social
2023-02-21T23:42:29Z
0 likes, 0 repeats
@ken @ProPublica @Mastodon I thought that was likely the case. Perhaps some folks here can instigate an outreach effort to get Tech and Press sites discussing the value of OpenSource & the Fediverse (Mastodon)...
(DIR) Post #ASv9BxMytnzhdj7HiS by cxarli@ehr.scot
2023-02-22T00:23:05Z
0 likes, 0 repeats
@ken @ProPublica @parkermolloy Have you considered that dear Mr. Musk might not wish deleted tweets to be trackable?
(DIR) Post #ASv9uTZ8YXf4Eg1xqa by ken@toots.schwanksta.com
2023-02-22T00:31:10Z
0 likes, 0 repeats
@cxarli @ProPublica @parkermolloy yes
(DIR) Post #ASv9y3Kc2Jx41uwV1s by shoq@mastodon.social
2023-02-22T00:31:48Z
0 likes, 0 repeats
@ken @ProPublica cc @Green_Footballs
(DIR) Post #ASvCNdqVc5ukQRBvIe by sara@hci.social
2023-02-22T00:58:48Z
0 likes, 0 repeats
@ken @ProPublica have you tried the compliance API? I think it cost money to use (even pre-Musk twitter). The entire purpose of the UI is to update external databases of tweet data to reflect users’ revised preferences, including and even especially when tweets are made private or deleted
(DIR) Post #ASvDSOIwvHOJcwK480 by sara@hci.social
2023-02-22T01:00:09Z
0 likes, 0 repeats
@ken @ProPublica oops, API not UI
(DIR) Post #ASvDSOweXf9Nc4tm0u by ken@toots.schwanksta.com
2023-02-22T01:10:54Z
0 likes, 0 repeats
@sara @ProPublica that is the one we have attempted to use, but it does not return deletions anymore. Any tweets deleted in the last few months don’t show up.
(DIR) Post #ASvDpXBS4p2tpEqqAa by Tedspence@indieweb.social
2023-02-22T01:15:04Z
0 likes, 0 repeats
@ken @ProPublica maybe time to just write a web scraper instead?
(DIR) Post #ASvE6ZcwFIUOKHphce by Tinu@mastodon.social
2023-02-22T01:18:06Z
0 likes, 0 repeats
@ken @ProPublica You should be able to find organizers on Twitter under #SaveA11yBots - those bots are still working. Our argument was accessibility. Some of us said all non-profit bits should be allowed to remain free.
(DIR) Post #ASvEmNfj5Iq9PX4WpM by moira@mastodon.murkworks.net
2023-02-22T01:25:42Z
0 likes, 0 repeats
@ken @ManyRoads @ProPublica @Mastodon I've been on my reps to join Mastodon, but a lot more people need to be doing that.
(DIR) Post #ASvLEmNfPXyVsQY8ki by VoteBlue4evr@universeodon.com
2023-02-22T02:38:05Z
0 likes, 0 repeats
@ken @ProPublica That was probably the point.
(DIR) Post #ASvLWCPnrvOvXnr61Y by jbaert@mastodon.social
2023-02-22T02:41:13Z
0 likes, 0 repeats
@ken @ProPublica what 2.0 API calls are you using?
(DIR) Post #ASvNN7stXpDonJGR4C by dansinker@omfg.town
2023-02-22T03:01:56Z
0 likes, 0 repeats
@ken @ProPublica *chef's kiss*
(DIR) Post #ASvOZSU6sYz2787DpA by fern@mindly.social
2023-02-22T03:15:24Z
0 likes, 0 repeats
@ken @ProPublica a bit of a cursed workaround, but perhaps you can work with the Internet Archive? Any chance they could have a special cron for scraping the birdplace more frequently? Also, A) It feels hella gross that we would have to resort to these measures to hold our elected officials - our EMPLOYEES - accountable for their wordsB) personally speaking, I am a huge fan of ProPublica and am so grateful for the work that you do.
(DIR) Post #ASvTV8w47UIA60x9ay by michi@newsie.social
2023-02-22T04:10:39Z
0 likes, 0 repeats
@ken @ProPublica Why pay for bad service? Musk has made no secret he does not support free press.
(DIR) Post #ASveRa7SCzV4k4mgee by emmatonkin@mstdn.social
2023-02-21T23:35:20Z
0 likes, 0 repeats
@kg6hxm @ken @ProPublica Reckon you have a good point there. Might be that the nearest thing to reliability now does not involve use of a developer API. Perhaps a distributed approach (yes, it's incredibly silly, but so is voluntarily breaking a service anyway...)?
(DIR) Post #ASveSWQbPxHPaGBnYO by kg6hxm@social.makerforums.info
2023-02-22T04:35:47Z
0 likes, 0 repeats
@emmatonkinI was able to make a parallel processing PowerShell script with "n" threads work through a list reliably. The hardest part would be in asking for the smallest web page size to get the information needed.I might have to try making this.@ken @ProPublica
(DIR) Post #ASvh3oBoXlH9IEBBqq by DrOtto@mastodon.online
2023-02-22T06:42:33Z
0 likes, 0 repeats
@ken @ProPublica A scam. The paid model would be a scam.
(DIR) Post #ASvhum5J6HMk72OMds by williampietri@sfba.social
2023-02-22T06:52:07Z
0 likes, 0 repeats
@ken @sara @ProPublica That is the one I was going to suggest too. I have some academic pals that are sweating related issues; DM me if you want to be put in touch. Also glad to ask in the ex-Twitter Slacks if that's helpful.
(DIR) Post #ASviChXOkCyNDt4zAm by en3py@infosec.exchange
2023-02-22T06:55:24Z
0 likes, 0 repeats
@ken @ProPublica we already have a massive proliferation and duplication of data. Some third pary apps probably wouldn't even care. If they take data realtime via API they don't even have to care.
(DIR) Post #ASvxky4pG0Jng0VqNM by botvolution@mastodon.sdf.org
2023-02-22T09:49:39Z
0 likes, 0 repeats
@ken @ProPublica I'm guessing you used to use the v1.1 streaming filtered endpoint and track all the accounts of interest?v2 streaming e/p is certainly a mess. Everything being done at the other place seems to be aimed at making objective analysis of activity there impossible.I suspect finding mitigations for each change will prove increasingly fruitless.(I don't have any actually useful suggestions; but I'll have a think)
(DIR) Post #ASwAUgkZD04Si2bX4C by ken@toots.schwanksta.com
2023-02-22T12:12:25Z
0 likes, 0 repeats
@dansinker @ProPublica thought it was worth a shot dan!
(DIR) Post #ASwT5U7AwpA5PXsIkK by ken@toots.schwanksta.com
2023-02-22T15:40:46Z
0 likes, 0 repeats
@williampietri If anyone knows how to get it working, would be happy to talk to them!
(DIR) Post #ASwlgGVAK1lN52mewK by williampietri@sfba.social
2023-02-22T19:09:04Z
0 likes, 0 repeats
@ken Fab. I'll let you know. If they API's no good, also glad to chat about converting it to manual checking. I have done a bunch of crawling over the years.
(DIR) Post #ASxkyfJ6vdN8QiJbbE by youfoundryan@macaw.social
2023-02-23T06:35:55Z
0 likes, 0 repeats
@ken @ProPublica (Former search API TL here) Are you hydrating individual tweets or using one of the compliance APIs?
(DIR) Post #ASyL5yz3laRXpgj0tM by ken@toots.schwanksta.com
2023-02-23T13:20:38Z
0 likes, 0 repeats
@ProPublica the batch compliance API. We’ve sent it tweets we *know* were deleted and get back no deletion events!
(DIR) Post #ASyPHhTH6U8goGhVVQ by lahosken@hachyderm.io
2023-02-23T14:07:32Z
0 likes, 0 repeats
@ken I would ask @travisbrown , though I confess I'm not sure whether his track-big-changes-on-twitter thingies use the API or the Internet Archive or something completely different
(DIR) Post #ASyTusHXF9ATRt92f2 by ken@toots.schwanksta.com
2023-02-23T14:59:28Z
0 likes, 0 repeats
@jbaert the batch compliance API to get deletion events for our very large database of tweet IDs
(DIR) Post #ASyl1rOOe9ZjwFGrY0 by msgbi@mastodon.social
2023-02-23T18:11:10Z
0 likes, 0 repeats
@ken @ProPublica @charles_ex @aloa5 need need 18 days