Post AXKfqqV760TH2GOHCK by sam@queernerds.social
(DIR) More posts by sam@queernerds.social
(DIR) Post #AXKeZhQVTs8I0zUoFs by mjg59@nondeterministic.computer
2023-07-03T23:08:13Z
0 likes, 0 repeats
Hitting Twitter with a Googlebot user agent now returns "Unverified web crawlers are not allowed"
(DIR) Post #AXKerCHU1AbQdCcQKG by Viss@mastodon.social
2023-07-03T23:10:20Z
0 likes, 0 repeats
@mjg59 i wonder what would happen if you figured out the user agent that twitter itself uses to do things like fetch cards and whatnot and used that against it :D
(DIR) Post #AXKf5T7qBfk0vZuifI by dango_@mas.to
2023-07-03T23:11:15Z
0 likes, 0 repeats
@mjg59 bold strategy, ngl
(DIR) Post #AXKfUmPTcp7XydESUC by nickzoic@aus.social
2023-07-03T23:14:32Z
0 likes, 0 repeats
@mjg59 surprisingly polite response, for Twitter.
(DIR) Post #AXKfqqV760TH2GOHCK by sam@queernerds.social
2023-07-03T23:19:25Z
0 likes, 0 repeats
@mjg59 Google pay $8
(DIR) Post #AXKg7NyeOo8RUh5q2i by mjg59@nondeterministic.computer
2023-07-03T23:22:22Z
0 likes, 0 repeats
Although there's still material being indexed, so it's possible that they're still allowing it from specific IP ranges
(DIR) Post #AXKhyR0jwz6uClvj6m by FritzAdalis@infosec.exchange
2023-07-03T23:46:59Z
0 likes, 0 repeats
@mjg59What if you use Spider Man's user agent?
(DIR) Post #AXKiOEpjqf0Bxz5X60 by natanbc@uwu.social
2023-07-03T23:51:08Z
0 likes, 0 repeats
@mjg59 wonder if GCP ranges bypass that
(DIR) Post #AXLPpQonh1eoz6rXzU by flameeyes@mastodon.social
2023-07-04T07:58:38Z
0 likes, 0 repeats
@mjg59 so they caught up with me circa 2010.https://flameeyes.blog/2010/05/01/so-you-think-you-can-crawl/