[HN Gopher] Disrupting the largest residential proxy network
       ___________________________________________________________________
        
       Disrupting the largest residential proxy network
        
       Author : cdrnsf
       Score  : 223 points
       Date   : 2026-01-28 22:46 UTC (3 days ago)
        
 (HTM) web link (cloud.google.com)
 (TXT) w3m dump (cloud.google.com)
        
       | xyzzy_plugh wrote:
       | > These efforts to help keep the broader digital ecosystem safe
       | supplement the protections we have to safeguard Android users on
       | certified devices. We ensured Google Play Protect, Android's
       | built-in security protection, automatically warns users and
       | removes applications known to incorporate IPIDEA SDKs, and blocks
       | any future install attempts.
       | 
       | Nice to see Google Play Protect actually serving a purpose for
       | once.
        
         | trollbridge wrote:
         | Yeah, it serves the purpose of blocking this kind of proxy
         | traffic that isn't in Google's personal best interests.
         | 
         | Only Google is allowed to scrape the web.
        
           | a456463 wrote:
           | Yup exactly. Google must be the only one allowed to scrape
           | the web. Google can't have any other competition. Calling it
           | in "user's best interest" is just like their other marketing
           | cons: "play integrity for user's security" etc
        
           | viraptor wrote:
           | Have you got any proof of Google scraping from residential
           | proxies users don't know about, rather than from their
           | clearly labelled AS? Otherwise you're mixing entirely
           | different things into one claim.
        
             | idiotsecant wrote:
             | I don't think parent post is claiming that Google is using
             | other people's networks to scrape the web only that they
             | have a strong incentive to keep _other_ players from doing
             | that.
        
               | viraptor wrote:
               | No, there are other scrapers that Google doesn't block or
               | interact with. You can even run scraping from GCP. This
               | has nothing to do with "only Google is allowed to
               | scrape". They even host apps which exist for scraping
               | data, like https://play.google.com/store/apps/details?id=
               | com.sociallead...
        
             | misir wrote:
             | That's the whole point. Websites that try to block scraping
             | attempts will let google scrape without any hurdle because
             | of google's ads and search network. This gives google some
             | advantage over new players because as a new name brand you
             | are hardly going to convince a website to allow scraping
             | even if your product may actually be more advantageous to
             | the website (for example assume you made a search engine
             | that doesn't suck like google, and aggregates links instead
             | of copying content from your website).
             | 
             | Proxies in comparison can allow new players to have some
             | playing chance. That said I doubt any legitimate & ethical
             | business would use proxies.
        
           | vachina wrote:
           | This is demonstrably false by the success of many scrapers
           | from AI companies.
        
             | Nextgrid wrote:
             | LLMs aren't a good indicator of success here because an LLM
             | trained on 80% of the data is just as good as one trained
             | on 100%, assuming the type/category of data is distributed
             | evenly. Proxies help when you do need to get access to 100%
             | of the data including data behind social media loginwalls.
        
           | 1vuio0pswjnm7 wrote:
           | "Only Google is allowed to scrape the web."
           | 
           | If I'm not mistaken, the plaintiffs in the US v Google
           | antitrust litigation in the DC Circuit tried to argue that
           | website operators are biased toward allowing Google to crawl
           | and against allowing other search engines to do the same
           | 
           | The Court rejected this argument because the plaintiffs did
           | not present any evidence to support it
           | 
           | For someone who does not follow the web's history, how would
           | one produce direct evidence that the bias exists
        
             | SkiFire13 wrote:
             | > For someone who does not follow the web's history, how
             | would one produce direct evidence that the bias exists
             | 
             | Take a bunch of websites, fetch their robots.txt file and
             | check how many allow GoogleBot but not others?
        
             | 1vuio0pswjnm7 wrote:
             | Common Crawl provides gzipped robots.txt collections
        
           | miki123211 wrote:
           | Google does not use residential proxies.
           | 
           | This does nothing against your ability to scrape the web the
           | Google way, AKA from your own assigned IP range, obeying
           | robots.txt, and with an user agent that explicitly says what
           | you're doing and gives website owners a way to opt out.
           | 
           | What Google doesn't want (and I don't think that's a bad
           | thing) is competitors scraping the web in bad faith, without
           | disclosing what they're doing to site owners and without
           | giving them the ability to opt out.
           | 
           | If Google doesn't stop these proxies, unscrupulous parties
           | will have a competitive advantage over Google, it's that
           | simple. Then Google will have to decide between just giving
           | up (unlikely) or becoming unscrupulous themselves.
        
             | ryanjshaw wrote:
             | > This does nothing against your ability to scrape the web
             | the Google way
             | 
             | I thought that Google has access to significant portions of
             | the internet that non-Google bots won't have access to?
        
               | morkalork wrote:
               | Their crawler has known IPs that get a white-glove
               | treatment by every site with a paywall for example
        
         | direwolf20 wrote:
         | Does it also block unwanted traffic from Google apps or does it
         | have a particular hatred for companies that interfere with
         | Google's business model?
        
           | tgsovlerkhgsel wrote:
           | Play Protect blocks malicious apps, not network traffic, so
           | no, it obviously doesn't interfere with Google's apps.
           | 
           | AFAIK it also left SmartTube (an alternative YouTube client)
           | alone until the developer got pwned and the app trojanized
           | with this kind of SDK, and the clean versions are AFAIK again
           | being left alone. No guarantee that it won't change in the
           | future, of course, but so far they seem to not be abusing it.
        
             | direwolf20 wrote:
             | Does malicious mean interfering with Google's business
             | model, or does it include intrusive advertising?
        
               | ThePowerOfFuet wrote:
               | malicious [?] intrusive.
        
               | direwolf20 wrote:
               | but intrusive advertising is malicious
        
       | kotaKat wrote:
       | I'm actually a little shocked seeing that there was a _WebOS_
       | variant of the residential proxying SDK endpoint. Does that mean
       | there might be a bit more unchecked malware lurking behind the
       | scenes in the LG ecosystem?
       | 
       | Personally I'm surprised they didn't have a Samsung option.
        
         | wincy wrote:
         | I keep my brand new LG C5 totally disconnected from the
         | internet and use my Apple TV for movie watching. I'm not going
         | to trust a company like LG to secure their devices.
        
           | xnx wrote:
           | > trust a company like LG to secure their devices.
           | 
           | They have an interest in securing their devices so they can
           | sell proxy service themselves.
        
         | dewey wrote:
         | Why would webOS App Store be any different than the iOS or
         | Android App Store which also have monetization frameworks for
         | bandwidth sharing.
        
       | whartung wrote:
       | My understanding is that routing through residential IPs is a
       | part of the business of some VPN providers. I don't know how
       | above board they are on this (as in notifying customers that this
       | may happen, however buried in the usage agreement, or even
       | allowing them to opt out).
       | 
       | But, my main point, is that the whole business is "on the up and
       | up" vs some dark botnet.
        
         | nielsbot wrote:
         | FTA
         | 
         | > While operators of residential proxies often extol the
         | privacy and freedom of expression benefits of residential
         | proxies, Google Threat Intelligence Group's (GTIG) research
         | shows that these proxies are overwhelmingly misused by bad
         | actors
        
           | direwolf20 wrote:
           | Google's definition of a "bad actor" is someone who wants to
           | use Google without seeing the ads. Or Kagi. Or an AI other
           | than Gemini.
        
         | kawsper wrote:
         | Oxylabs sells proxies for scrapers, I suppose you can use the
         | socks-proxy as a VPN, and they claim to use Honeygain.
         | 
         | Honeygain is a platform where people sell their residential
         | internet connection and bandwidth to these companies for money.
         | 
         | For comparison Honeygain pays someone 10 cents per GB, and
         | Oxylabs sells it for $8/GB.
        
           | aussieguy1234 wrote:
           | That takes buying low and selling high to a whole new level
        
           | moduspol wrote:
           | We need a better market. I'd sell for $7/GB.
        
         | Bratmon wrote:
         | > I don't know how above board they are on this
         | 
         | Saying you don't know something in the comments of an article
         | that explains that thing is a bold strategy
        
         | mhitza wrote:
         | Mullvad seems to be one of those VPN providers. [1] Though I
         | very much doubt they would sneakily make end-users devices exit
         | nodes. Though, as a historical side note, let's not forget
         | Skype used to make users computers act as a relay as well
         | during its more decentralized days.
         | 
         | [1] Using the website mentioned by user Rasbora
         | https://news.ycombinator.com/item?id=46837806
        
       | samsullivan wrote:
       | The need for proxies in any legitimate context became obsolete
       | with starlink being so widespread. Throw up a few terminals and
       | you have about 500-2k cgnat IP addresses to do whatever you like.
        
         | JDye wrote:
         | 2k IPs is not enough to do most enterprise scale scraping.
         | Starlink's entire ASN doesn't seem to have enough V4 addresses
         | to handle it even.
        
           | chatmasta wrote:
           | The actual secret is to use IPv6 with varied source IPs in
           | the same subnet, you get an insane number of IPs and 90% of
           | anti-scraping software is not specialized enough to realize
           | that any IP in a /64 is the same as a single IP in a /32 in
           | IPv4.
        
             | cferry wrote:
             | > any IP in a /64 is the same as a single IP in a /32 in
             | IPv4
             | 
             | This is very commonly true but sadly not 100%. I am
             | suffering from a shared /64 on which a VPS is, and where
             | other folks have sent out spam - so no more SMTP for me.
        
           | fc417fc802 wrote:
           | If they're CGNAT then unless Starlink actively provides
           | assistance to block them it won't matter.
           | 
           | As someone who wants the internet to maintain as much anarchy
           | as possible I think it would be nice to see a large ISP that
           | actively rotated its customer IPv6 assignments on a tight
           | schedule.
        
       | londons_explore wrote:
       | We need _more_ residential proxies, not less.
       | 
       | I've had enough of companies saying "you're connecting from an
       | AWS IP address, therefore you aren't allowed in, or must buy
       | enterprise licensing". Reddit is an example which totally blocks
       | all data to non-residential IP's.
       | 
       | I want exactly the same content visible no matter who you are or
       | where you are connecting from, and a robust network of
       | residential proxies is a stepping stone to achieving that.
        
         | xg15 wrote:
         | Also, nevermind the tech companies building their own proxy
         | networks, such as Find My or Amazon Sidewalk.
        
           | a456463 wrote:
           | Agreed. With things people paid for and using our wifi data
           | to build their "positioning dbs" that you can't block or turn
           | off on your phone, without "rooting" your own device.
        
           | enneff wrote:
           | How is Find My a proxy network?
        
             | direwolf20 wrote:
             | In the literal sense. Your traffic is proxied through
             | devices belonging to unwilling strangers.
        
               | enneff wrote:
               | By "your traffic" you mean device location reports? Or
               | something else?
        
               | DANmode wrote:
               | The data that powers the app tracking your devices, shown
               | on your devices, yes.
               | 
               | (What else?)
        
               | enneff wrote:
               | I don't know. I wouldn't have thought of myself as
               | proxying other people's traffic by carrying my iPhone
               | around. (For one thing, it's my own phone that initiates
               | all the activity- it monitors for Apple devices, the
               | devices don't reach out to my phone.) I can see how you
               | could frame it that way, though. I just thought they
               | might be referring to something else that I didn't know
               | about.
        
               | MBCook wrote:
               | I remain skeptical. I can understand how one would might
               | see it that way, but I think it's stretching the word
               | proxy too far.
               | 
               | Devices on Apple's Find My aren't broadcasting anything
               | like packets that get forwarded to a destination of their
               | choosing. I would think that would be a necessity to call
               | it "proxying".
               | 
               | They're just broadcasting basic information about
               | themselves into the void. The phones report back what
               | they've picked up.
               | 
               | That doesn't fit the definition to me.
               | 
               | I absolutely don't mind the fact that my phone is doing
               | that. The amount of data is ridiculously minuscule. And
               | it's sort of a tit for tat thing. Yeah my phone does it,
               | but so does theirs. So just like I may be helping you
               | locate your AirTag, you would be helping me locate mine.
               | Or any other device I own that shows up on Find My.
               | 
               | It's a very close to a classic public good, with the only
               | restriction being that you own a relevant device.
        
               | DANmode wrote:
               | > aren't broadcasting anything like packets that get
               | forwarded to a destination of their choosing
               | 
               | Protocol insists the data only goes back to owner device
               | or Apple server.
        
               | fc417fc802 wrote:
               | Yes. It's "edge routing" that happens to be restricted to
               | a single operator.
        
         | direwolf20 wrote:
         | You can run one, something like ByteLixir, Traffmonetizer,
         | Honeygain, Pawns, there are lots more, just google "share my
         | internet for money"
         | 
         | What will you be proxying? Nobody knows! I haven't had the
         | police at my house yet.
         | 
         | Seems a great way to say "fuck you" to companies that block IP
         | addresses.
         | 
         | You may see a few more CAPTCHAs. If you have a dynamic IP
         | address, not many.
        
           | dist-epoch wrote:
           | How much can you make if you run all of them at the same
           | time?
           | 
           | Doesn't the ISP detect them?
        
             | direwolf20 wrote:
             | like $3 a month
             | 
             | and why would they
        
         | ndiddy wrote:
         | If you look at the article, the network they disrupted pays
         | software vendors per-download to sneakily turn their users into
         | residential proxy endpoints. I'm sure that at least _some_ of
         | the time the user is _technically_ agreeing to some wording
         | buried in the ToS saying they consent to this, but it 's
         | certainly unethical. I wouldn't want to proxy traffic from
         | random people through my home network, that's how you get legal
         | threats from media companies or the police called to your
         | house.
        
           | londons_explore wrote:
           | > that's how you get legal threats from media companies or
           | the police called to your house.
           | 
           | Or residential proxies get so widespread that almost every
           | house has a proxy in, and it becomes the new way the internet
           | works - "for privacy, your data has been routed through
           | someone else's connection at random".
        
             | Imustaskforhelp wrote:
             | > Or residential proxies get so widespread that almost
             | every house has a proxy in, and it becomes the new way the
             | internet works - "for privacy, your data has been routed
             | through someone else's connection at random".
             | 
             | Is this a re-invention of tor, maybe I2P?
        
               | rolph wrote:
               | IP8 address tumbler? to wit, playing the shell game, to
               | obstruct direct attribution.
        
               | chii wrote:
               | > Is this a re-invention of tor
               | 
               | in a way, yes - the weakness of tor is realistically the
               | lack of widespreadness. Tor traffic is identifiable and
               | blockable due to the relatively rare number of exit nodes
               | (which also makes it dangerous to run exit nodes, as you
               | become "liable").
               | 
               | Engraining the ideas of tor into regular users' internet
               | usage is what would prevent the internet from being
               | controlled and blockable by any actor (except perhaps
               | draconian gov't over reach, which while can happen, is
               | harder in the west).
        
           | dataviz1000 wrote:
           | They provide an SDK for mobile developers. Here is a video of
           | how it works. [0] They don't even hide it.
           | 
           | [0] https://www.youtube.com/watch?v=1a9HLrwvUO4&t=15s
        
             | ndiddy wrote:
             | Of course they're pitching it like everything's above
             | board, but from the article:
             | 
             | > While many residential proxy providers state that they
             | source their IP addresses ethically, our analysis shows
             | these claims are often incorrect or overstated. Many of the
             | malicious applications we analyzed in our investigation did
             | not disclose that they enrolled devices into the IPIDEA
             | proxy network. Researchers have previously found
             | uncertified and off-brand Android Open Source Project
             | devices, such as television set top boxes, with hidden
             | residential proxy payloads.
        
               | direwolf20 wrote:
               | If popup ads that open the play store are ethical, this
               | is ethical.
        
               | calgoo wrote:
               | I love how its the "evil" Open Source project devices,
               | and "other app stores" that are the problem, not the 100s
               | of spyware ridden crap that is available for download
               | from the Play store. Would be interesting to know how
               | many copies of the SDK was found and removed from their
               | own platform.
        
         | BoredPositron wrote:
         | I still "run" a small ISP with a few thousand residential ips
         | from my scraping days. The requirements are laughable and costs
         | were negligible in the early 2000s.
        
         | Aurornis wrote:
         | > I want exactly the same content visible no matter who you are
         | or where you are connecting from
         | 
         | The reason those IP addresses get blocked is not because of
         | "who" is connecting, but "what"
         | 
         | Traffic from datacenter address ranges to sites like Reddit is
         | almost entirely bots and scrapers. They can put a tremendous
         | load on your site because many will try to run their queries as
         | fast as they can with as many IPs as they can get.
         | 
         | Blocking these IP addresses catches a few false positives, but
         | it's an easy step to make botting and scraping a little more
         | expensive. Residential proxies aren't all that expensive, but
         | now there's a little line item bill that comes with their
         | request volume that makes them think twice.
         | 
         | > We need more residential proxies, not less
         | 
         | Great, you can always volunteer your home IP address as a
         | start. There are services that will pay you a nominal amount
         | for it, even.
        
           | megous wrote:
           | Okay. So what does ten million requests cost, then? Like... a
           | dollar? Is it a dollar? Is it two dollars if they splurge?
           | 
           | Because if the deterrent here is a line item so small it
           | shows up as 'miscellaneous vibes' on a balance sheet, that's
           | not a barrier. That's a tip jar.
        
             | Aurornis wrote:
             | Residential proxies are often priced by Gigabyte and the
             | pricing is several orders of magnitude higher than your
             | estimate.
        
         | tokyobreakfast wrote:
         | > I've had enough of companies saying "you're connecting from
         | an AWS IP address
         | 
         | I run a honeypot and the amount of bot traffic coming from AWS
         | is insane. It's like 80% before filtering, and it's 100%
         | illegitimate.
        
           | ghxst wrote:
           | Most of them abuse the ip pool attached to lambda from my
           | experience.
        
           | jstanley wrote:
           | > it's 100% illegitimate.
           | 
           | Based on what?
           | 
           | I think perhaps you merely meant to say that more than 99% of
           | it is illegitimate?
        
         | JDye wrote:
         | I live in the UK and can't view a large portion of the internet
         | without having to submit my ID to _every_ site serving anything
         | deemed "not safe the for the children". I had a question about
         | a new piercing and couldn't get info on it from Reddit because
         | of that. I try using a VPN and they're blocked too. Luckily, I
         | work at a copmany selling proxies so I've got free proxies
         | whenever I want, but I shouldn't _need_ to use them.
         | 
         | I find it funny that companies like Reddit, who make their
         | money entirely from content produced by users for free (which
         | is also often sourced from other parts of the internet without
         | permission), are so against their site being scraped that they
         | have to objectively ruin the site for everyone using it. See
         | the API changes and killing off of third party apps.
         | 
         | Obviously, it's mostly for advertising purposes, but they love
         | to talk about the load scraping puts on their site, even suing
         | AI companies and SerpApi for it. If it's truly that bad, just
         | offer a free API for the scrapers to use - or even an API that
         | works out just slightly cheaper than using proxies...
         | 
         | My ideal internet would look something like that, all content
         | free and accessible to everyone.
        
           | Aurornis wrote:
           | > that they have to objectively ruin the site for everyone
           | using it. See the API changes and killing off of third party
           | apps.
           | 
           | Third party app users were a very small but vocal minority.
           | The API changes didn't drop their traffic at all. In fact,
           | it's only gone up since then.
           | 
           | The datacenter IP address blocks aren't just for scrapers,
           | it's an anti-bot measure across the board. I don't spend much
           | time on Reddit but even the few subreddits I visited were
           | starting to become infiltrated by obvious bot accounts doing
           | weird karma farming operations.
           | 
           | Even HN routinely gets AI posting bots. It's a common
           | technique to generate upvote rings - Make the accounts post
           | comments so they look real enough, have the bots randomly
           | upvote things to hide activity, and then when someone buys
           | upvotes you have a selection of the puppet accounts upvote
           | the targeted story. Having a lot of IP addresses and
           | generating fake activity is key to making this work, so
           | there's a lot of incentive to do it.
        
             | JDye wrote:
             | I agree that write-actions should be protected, especially
             | now when every other person online is a bot. As for read-
             | actions, I'll continue to profit off those being protected
             | too but I wouldn't be too bothered if something suddenly
             | changed and all content across the internet was a lot
             | easier to access programmatically. I think only harm can
             | come from that data being restricted to the huge
             | (nefarious) companies that can pay for that data or
             | negotiate backroom deals.
        
             | direwolf20 wrote:
             | Reddit's traffic is almost exclusively propaganda bots.
        
           | 201984 wrote:
           | Fix your government.
        
             | JDye wrote:
             | Thanks lad. Will get right on it.
        
               | ThePowerOfFuet wrote:
               | Scrapping First-past-the-Post is probably a good start.
               | 
               | Good luck!
        
           | what wrote:
           | Have you considered that it's because a new industry popped
           | up that decided it was okay to slurp up the entire internet,
           | repackage it, and resell it? Surely that couldn't be why
           | sites are trying to keep non humans out.
        
           | ErroneousBosh wrote:
           | > I live in the UK and can't view a large portion of the
           | internet without having to submit my ID to _every_ site
           | serving anything deemed "not safe the for the children".
           | 
           | Really? Because I live in the UK and I've never been asked
           | for my ID for anything.
        
         | a456463 wrote:
         | This blog post from the company that used promise "don't be
         | evil", one that steals water for data centers from vilages and
         | towns via shady deals, whose whole premise it stealing other
         | people's stuff and claiming it as their own and locking them
         | out and selling their data.. Who made them the arbiter of the
         | internet? No one!!!
         | 
         | They just stole this and get on their high horse to tell people
         | how to use internet? You can eff right off Google.
        
         | crtasm wrote:
         | I'm reading reddit.com from a Tor node, they also have a .onion
         | domain you could use.
        
           | Jblx2 wrote:
           | Anyone know how to create a usable reddit account from the
           | .onion domain?
        
             | phyzome wrote:
             | I've tried it, and my account was shadowbanned a few hours
             | after I created it. It's very obnoxious.
        
               | cluckindan wrote:
               | Reddit bots shadowban almost everyone who post before
               | they have enough comment karma. Nothing to do with Tor or
               | VPN.
        
               | phyzome wrote:
               | I didn't try posting, I tried commenting.
        
         | nine_k wrote:
         | There's a company that pays you to keep their box connected to
         | your residential router. I assume it sells residential proxy
         | services, maybe also DDoS services, I don't know. It's aptly
         | named Absurd Computing.
        
         | yuliyp wrote:
         | The end game of that is no useful content being accessible
         | without login, or needing some sort of other proof-of-
         | legitimacy.
        
           | supertrope wrote:
           | Amazon.com now only shows you a few reviews. To see the rest
           | you must login. Social media websites have long gated the
           | carrots behind a login. Anandtech just took their ball and
           | went home by going offline.
        
           | Nextgrid wrote:
           | That's already the case (irrespective of residential proxies)
           | because content only serves as bait for someone to hand over
           | personal information (during signup/login) and then _engage_
           | with ads.
           | 
           | Proxies actually help with that by facilitating mass account
           | registration and scraping of the content without wasting a
           | human's time "engaging" with ads.
        
       | direwolf20 wrote:
       | All of this sounds legal, so on what basis did they get them shut
       | down?
        
         | SOTGO wrote:
         | I haven't looked at any court documents, but the WSJ article
         | from Wednesday reported that "Last year, Google sued the
         | anonymous operators of a network of more than 10 million
         | internet-connected televisions, tablets and projectors, saying
         | they had secretly pre-installed residential proxy software on
         | them... an Ipidea spokeswoman acknowledged in an email that the
         | company and its partners had engaged in "relatively aggressive
         | market expansion strategies" and "conducted promotional
         | activities in inappropriate venues (e.g., hacker forums)...""
         | 
         | There was also a botnet, Kimwolf, that apparently leveraged an
         | exploit to use the residential proxy service, so it may be
         | related to Ipidea not shutting them down.
        
           | direwolf20 wrote:
           | Google does much worse in Google-branded devices and apps,
           | like the wifi location data harvesting.
        
             | toofy wrote:
             | neat, so let's stop them too.
             | 
             | the answer is stop all the bad actors, not "well jimmy does
             | it!"
        
       | scirob wrote:
       | so that only google and anthropic are allowed to scrape the web.
       | No one else may have workarounds
        
         | a456463 wrote:
         | Exactly. This is just google building a "moat" around their
         | shady business.
        
           | cvalka wrote:
           | 100%
        
         | mrweasel wrote:
         | Anyone could scrape the net, then modern scrapes came along
         | with their shitty code and absolutely no respect. The reason
         | why so many of us block or throttle scrapers is because they
         | miss behave. They don't back off, they try to by-pass caches
         | and if they crash a site they don't adjust, they will just
         | pound it the ground again when it's back. We managed to talk to
         | one large AI company would didn't really want to fix anything,
         | but told us that they'd be fine with us just rate limiting
         | them, as if we somehow owed them anything. They just get a
         | stupid low rps now, even if we'd let them go faster, if they'd
         | just fix they bot.
         | 
         | Some sites don't want you scraping, but it's their content,
         | their rules. We don't really care, but we have to due to the
         | number and quality of the bots we're seeing. This is in my mind
         | a 100% self-imposed problem from the scrapers.
        
       | progbits wrote:
       | I'm surprised by the negative takes...
       | 
       | Yes, proxies are good. Ones which you pay for and which are
       | running legitimately, with the knowledge (and compensation) of
       | those who run them.
       | 
       | Malware in random apps running on your device without your
       | knowledge is bad.
        
         | bdcravens wrote:
         | Many are "compensated" (in the way of software they didn't pay
         | for), so the real question is that of disclosure (in which case
         | many software vendors check the box in the most minimal way
         | possible by including it as fine print during the install)
        
           | happyopossum wrote:
           | No, the question is not just disclosure. People have their
           | bandwidth stolen, and sometimes internet access revoked due
           | to this kind of fraud and misuse - disclosure wouldn't solve
           | that
        
             | the_fall wrote:
             | Also, as a website owner, these residential proxies are a
             | _real_ pain. Tons and tons of abusive traffic, including
             | people trying to exploit vulnerabilities and patently
             | broken crawlers that send insane numbers of requests, and
             | no real way to block it.
             | 
             | It's just nasty stuff. Intent matters, and if you're
             | selling a service that's used only by the bad guys, you're
             | a bad guy too. This is not some dual-use, maybe-we-should-
             | accept-the-risks deal that you have with Tor.
        
             | bigfatkitten wrote:
             | If they're lucky. Sometimes people have their doors kicked
             | in by armed police.
        
         | CodeMage wrote:
         | Getting rid of malware is good. A private for-profit company
         | exercising its power over the Internet, not so much. We should
         | have appropriate organizations for this.
        
           | UqWBcuFx6NV4r wrote:
           | Okay. You get right on that. In the meantime, would you
           | rather they did nothing? What do you actually want, in
           | concrete terms?
        
           | vachina wrote:
           | The proxies is the reason why you get spam in your Google
           | search result, spam in your Play store (by means of fake good
           | reviews), basically spam in anything user generated.
           | 
           | It directly affects Google and you, I don't see why they
           | should not do this.
        
             | Nextgrid wrote:
             | Spam in Google search results is due to Google happily
             | taking money from the spammers in exchange for promoting
             | their spam, or that the spam sites benefit Google
             | indirectly by embedding Google Ads/Analytics.
             | 
             | I don't see any spam in Kagi, so clearly there _is_ a way
             | to detect and filter it out. Google is simply not doing so
             | because it would cut into their profits.
        
               | miki123211 wrote:
               | The reason you don't see spam in Kagi is because nobody
               | is targeting Kagi specifically.
               | 
               | They can probably get away with a lot of stupid rules
               | that would backfire if anybody tried to cater to them
               | specifically.
        
               | Nextgrid wrote:
               | "SEO spammers being more advanced than multi-billion-
               | dollar search conglomerate" is a myth. Spam sites have an
               | obvious objective: display ads, shill affiliate links or
               | sell products. All these have to be visible, since an ad
               | or product you can't see/buy is worthless. It is trivial
               | to train a classifier to detect these.
               | 
               | But let's play devil's advocate and say you are right and
               | spammers are successfully outsmarting Google - well, Kagi
               | does use Google results via SerpAPI by their own
               | admission, meaning they too should have those spam
               | results. Yet they somehow manage to filter them out with
               | a fraction of the resources available to Google itself
               | with no negative impact on search quality.
        
         | throwoutway wrote:
         | > Malware in random apps running on your device without your
         | knowledge is bad.
         | 
         | And ones that have all the indicators of compromise of Russia,
         | Iran, DPRK, PRC, etc
        
           | bigiain wrote:
           | Am I the only one cynically thinking that "Russia, Iran,
           | DPRK, PRC, etc" is the "But think of the chiiildren!!!"
           | excuse for doing this?
           | 
           | And when Google say
           | 
           | "IPIDEA's proxy infrastructure is a little-known component of
           | the digital ecosystem leveraged by a wide array of bad
           | actors."
           | 
           | What they really mean is " ... leveraged by actors
           | indiscriminately scraping the web and ignoring copyright -
           | that are not us."
           | 
           | I can't help but feel this is just Google trying to pull the
           | ladder up behind then and make it more difficult for other
           | companies to collect training data.
        
             | Craighead wrote:
             | No, what they're saying is what they said, what you're
             | implying reveals a strange bias. Web scraping through
             | residential proxies? Please think through your thoughts
             | more. There's much more effective and efficient ways to do
             | so. Multiple bad actors, like ransomware affiliates, have
             | been caught using residential proxy networks. But by all
             | means, don't let facts and cyber threat intelligence get in
             | the way.
        
               | bomewish wrote:
               | What are the much more effective and efficient ways --
               | since you said it ?
        
               | usefulposter wrote:
               | >let facts and cyber threat intelligence get in the way
               | 
               | Appeal to authority by way of invoking the megacorp-
               | branded "threat intelligence" capability (targeted PR
               | exercise).
        
               | JDye wrote:
               | Residential proxies aren't used for scraping? That
               | doesn't align well with my experience...
        
             | shit_game wrote:
             | >I can't help but feel this is just Google trying to pull
             | the ladder up behind then and make it more difficult for
             | other companies to collect training data.
             | 
             | I can very easily see this as being Google's reasoning for
             | these actions, but let's not pretend that clandestine
             | residential proxies aren't used for nefarious things. The
             | _vast_ majority of social media networks will ban - or more
             | generally and insiously - shadow ban accounts /IPs that use
             | known proxy IPs. This means that they are gating access to
             | their platforms behind residential IPs (on top of their
             | other various blackboxes and heuristics like
             | fingerprinting). Operators of bot networks thus rely on
             | residential proxy services to engage in their work, which
             | ranges from mundane things like engagement farming to
             | outright dangerous things like political astroturfing,
             | sentiment manipulation, and propaganda dissemination.
             | 
             | LLMs and generative image and video models have made the
             | creation of biased and convincing content trivial and
             | cheap, if not free. The days of "troll farms" is over, and
             | now the greatest expense for a bad actor wishing to
             | influence the world with fake engagement and biased
             | opinions is their access to platforms, which means accounts
             | and internet connections that aren't blacklisted or shadow
             | banned. Account maturity and reputation farming is also
             | feeling a massive boon due to these tools, but as an
             | independent market it also similarly requires internet
             | connections that aren't blacklisted or shadow banned.
             | Residential proxies are the bottleneck for the vast
             | majority of bad actors.
        
               | throwaway10948 wrote:
               | > The vast majority of social media networks will ban -
               | or more generally and insiously - shadow ban accounts/IPs
               | that use known proxy IPs. This means that they are gating
               | access to their platforms behind residential IPs (on top
               | of their other various blackboxes and heuristics like
               | fingerprinting)
               | 
               | Social media will ban proxy IPs, yet gleefully force you
               | to provide your ID if you happen to connect from the
               | wrong patch of land. I find it difficult not to support
               | any and all attempts to bypass such measures.
               | 
               | The fact is that there's now a perfectly legitimate use
               | for residential proxies, and the demand is just going to
               | keep growing as more websites decide to "protect their
               | content", and more governments decide to pass tyrannical
               | laws that force people to mask their IPs. And with
               | demand, comes supply, so don't expect them to go away any
               | time soon.
               | 
               | This really just sounds like a rehash of the argument
               | against encryption. "Bad people use it, so it should go
               | away" - never mind that there are completely legitimate
               | uses for it. Never mind that using a residential proxy
               | might be the _only_ way to get any privacy at all in a
               | future where everyone blocks VPNs and Tor, a future where
               | you may not even be able to post online without an ID
               | depending you where you live, a future which we 're
               | swiftly approaching.
               | 
               | It's already here, in fact. Imgur blocks UK users, but it
               | also blocks VPNs and Tor. The only way somebody living in
               | the UK can access Imgur is through a residential proxy.
        
               | ErroneousBosh wrote:
               | > The only way somebody living in the UK can access Imgur
               | is through a residential proxy.
               | 
               | And very little of value was lost.
               | 
               | > This really just sounds like a rehash of the argument
               | against encryption. "Bad people use it, so it should go
               | away" - never mind that there are completely legitimate
               | uses for it.
               | 
               | Except that almost everything that uses encryption has
               | some legitimate use. There are pretty much no legitimate
               | uses for residential proxies, and their use in flooding
               | the Internet with crap greatly outweighs that.
               | 
               | If I plumbed a 30cm sewage line straight into your living
               | room would you be happy with it? Okay, well, tell you
               | what, let's make it totally legit - I'll drop a tasty
               | ripe strawberry into the stream of effluent every so
               | often, how about that?
        
             | ErroneousBosh wrote:
             | > Am I the only one cynically thinking that "Russia, Iran,
             | DPRK, PRC, etc" is the "But think of the chiiildren!!!"
             | excuse for doing this?
             | 
             | Maybe. But until I dropped all traffic from pretty much
             | every mobile network provider in Russia and Israel, I'd get
             | up every morning to a couple of thousand new users of whom
             | a couple of hundred had consistently _within a few hundred
             | milliseconds_ created an account, clicked on the activation
             | link, and then posted a bunch of messages in every forum
             | category spreading hate speech.
        
         | vlovich123 wrote:
         | > Some users may knowingly install this software on their
         | devices, lured by the promise of "monetizing" their spare
         | bandwidth.
         | 
         | Sounds like they're targeting networks even if the users are ok
         | participating in, precisely what you're saying is ok.
         | 
         | As for malware enrolling people into the network, it depends if
         | the operator is doing it or if the malware is 3rd parties
         | trying to get a portion of the cash flow. In the latter case
         | the network would be the victim that's double victimized by
         | Google also attacking them.
        
           | wmf wrote:
           | Users are OK with acting as proxies because they don't
           | understand all the shady stuff their proxy is being used for.
           | Also consumer ISPs generally ban this.
        
             | chii wrote:
             | But then would you make the same arguments for running a
             | tor node (presumably, you don't know what shady stuff is
             | there, but you know there's shady stuff)?
        
               | jraph wrote:
               | That's totally something you should consider, even if you
               | decide for running the tor node anyway in the end.
        
               | Spooky23 wrote:
               | Running a tor node is pretty stupid from a liability
               | perspective, but at least you have more deniability and
               | you are making an informed choice.
               | 
               | These residential proxies are pretty much universally
               | shady. I doubt most of the users understand what they are
               | consenting to.
        
               | sitzkrieg wrote:
               | tor nodes are zero risk as long as they're not an exit
               | 
               | been running nodes since 2017 on two providers and zero
               | issues
        
             | iammrpayments wrote:
             | You could say the same about google's terms of service.
        
               | BrenBarn wrote:
               | A thousand times yes.
        
             | JasonADrury wrote:
             | Why would the users care either way?
        
               | jraph wrote:
               | Some people care about ethics, and try to avoid doing bad
               | stuff, or helping the bad stuff.
        
               | JasonADrury wrote:
               | Sure, but that only answers why _some_ users might care.
        
           | xhcuvuvyc wrote:
           | > These SDKs, which are offered to developers across multiple
           | mobile and desktop platforms, surreptitiously enroll user
           | devices into the IPIDEA network.
           | 
           | ?
        
             | vlovich123 wrote:
             | Here's an alternate spin
             | 
             | > These SDKs, which are offered to developers across
             | multiple mobile and desktop platforms.
             | 
             | > other actors then surreptitiously enroll user devices
             | into the IPIDEA network using these frameworks.
             | 
             | I'm not saying Google did the wrong thing, but it is one
             | private entity essentially handing out a death sentence on
             | its own. The only mitigating thing is that a) technical
             | disruptions were either on their own infra b) legal
             | judgements they then enforced with cooperation from others
             | like Cloudflare. But it's not clear what the legal
             | proceedings were actually like
        
         | riedel wrote:
         | I learn: proxy networks run by large corps are good. True
         | internet is bad. While I understand that often we are talking
         | about Malware/Worms etc that enable this. However, i find it
         | often disturbing to here often a lot of libertarian speech from
         | the tech scene, while on the other hand are feeling themselves
         | very comfortable to take over state power like policing efforts
         | to save the world.
        
         | mschuster91 wrote:
         | > Ones which you pay for and which are running legitimately,
         | with the knowledge (and compensation) of those who run them.
         | 
         | The problem is, it is _by default_ unethical to have
         | residential users be exit nodes for VPNs - unless these users
         | are lawyers or technical experts.
         | 
         | No matter what you do as a "residential proxy" company - you
         | cannot prevent your service being used by CSAM peddlers, and
         | thus you cannot prevent that your exit nodes aren't the ones
         | whose IP addresses show up when the FBI comes knocking.
        
       | IhateAI wrote:
       | How do you stop mobile proxies operating through similar
       | nefarious business models... CGNAT prevents you from easily
       | identifying the exit nodes.
        
         | UqWBcuFx6NV4r wrote:
         | Working with network operators.
        
           | Nextgrid wrote:
           | Network operators have zero reason to care, they get paid per
           | the GB for the bandwidth.
        
             | IhateAI wrote:
             | $5-9 a GB, its an infinite money glitch actually.
        
       | chatmasta wrote:
       | Why are they leaving Bright Data (aka Illuminati aka Hola VPN)
       | untouched? They are doing this exact scheme on an industrial
       | scale.
        
         | 7thpower wrote:
         | They have a robust KYC that appears to serve, at least in large
         | part, as a way to stay off the shit list of companies with the
         | resources to pursue recourse.
         | 
         | Source: went through that process, ended up going a different
         | route. The rep was refreshingly transparent about where they
         | get the data, why the have the kyc process (aside from
         | regulatory compliance).
         | 
         | Ended up going with a different provider who has been cheaper
         | and very reliable, so no complaints.
        
           | chatmasta wrote:
           | Yeah, they make you do a Skype interview (or probably Zoom
           | interview nowadays). You could call this KYC or collateral,
           | depending on your view of the company. It does limit the
           | nefariousness of their clientele but I doubt they do much, or
           | any, monitoring of actual traffic after onboarding (not for
           | compliance reasons, anyway).
        
           | walletdrainer wrote:
           | I've certainly never been asked to do KYC with Luminati after
           | using them for hundreds of terabytes over the years.
           | 
           | It's not like I'm using some bigco email address or given
           | them any other reason to skip KYC either.
        
             | ghxst wrote:
             | They do KYC when you want to unblock certain domains.
        
               | walletdrainer wrote:
               | Also not my experience, even though I've had to email
               | them for whitelisting.
               | 
               | It might just be because my account is very old?
        
               | 7thpower wrote:
               | Maybe, or more likely you're not trying to pull in
               | content that is considered high risk to them, such as
               | YouTube transcripts.
        
             | dewey wrote:
             | They probably would if they would see your username here!
        
               | walletdrainer wrote:
               | I think they should have requested KYC when I was
               | complaining about being unable to log into gmail, but I'm
               | not going to complain as long as the service works.
               | 
               | I don't use Luminati for anything illegal though, so it's
               | possible they just have some super amazing abuse
               | detection algorithms that know this.
        
       | ExpertAdvisor01 wrote:
       | Of course brightdata doesn't get touched.
        
       | walletdrainer wrote:
       | It's interesting that when Luminati, an Israeli company, does
       | this, it's fine.
       | 
       | When the Chinese do this? Very bad.
        
         | VladVladikoff wrote:
         | They are both bad. You are showing your own bias.
        
           | walletdrainer wrote:
           | Personally, I don't think either of them are actually
           | meaningfully bad. A bit naughty, maybe?
           | 
           | I do think the disparity in attention is fascinating. These
           | new Chinese players have been getting nonstop press while
           | everyone ignores the established giant.
        
           | calgoo wrote:
           | No, he is referencing Google going after the Chinese company,
           | not the Israel based one. That does not mean there is bias
           | with the commenter at all, just that the companies operate
           | differently and are treated differently. The country of
           | origin is important as Israel based companies are more
           | integrated into the western business world, and tend to at
           | least try to show an effort in keeping spam and other things
           | off their platforms. Now I do agree that they are both bad
           | companies that should not be allowed to operate the way they
           | do. I would say the same thing about the other 1000 scrapers
           | hitting websites everyday as well (including Google).
           | 
           | What they did not comment directly on, is how many apps /
           | games they might have actually removed from the Playstore
           | with the removal of the SDKs, which would be the actual
           | interesting data.
        
             | JasonADrury wrote:
             | FWIW a couple of years ago I was involved in a court case
             | where there was a subpoena sent to Luminati to figure out
             | whether or not a specific request had originated from their
             | network, lawyers Luminati replied that they do not keep any
             | logs whatsoever as they aren't required to do so under
             | Israeli law.
             | 
             | Hard to imagine any serious anti-abuse efforts by Luminati
             | if they don't monitor what their users are doing, but this
             | is probably a deliberate effort to avoid potential
             | liability arising from knowing what their users are doing.
        
       | brikym wrote:
       | I'll betcha Google uses a lot of residential proxies themselves
       | to scrape data and don't want competitors doing it.
        
         | arewethereyeta wrote:
         | I'll betcha your scraping for google simply by using Chrome
        
       | edg5000 wrote:
       | Residential proxies are the only way to crawl and scrape. It's
       | ironic for this article to come from the biggest scraping company
       | that ever existed!
       | 
       | If you crawl at 1Hz per crawled IP, no reasonable server would
       | suffer from this. It's the few bad apples (impatient people who
       | don't rate limit) who ruin the internet for both users and
       | hosters alike. And then there's Google.
        
         | BatteryMountain wrote:
         | Saying the quiet part out loud...Shhhs
        
         | Ronsenshi wrote:
         | One thing about Google is that many anti-scraping services
         | explicitly allow access to Google and maybe couple of other
         | search engines. Everybody else gets to enjoy CloudFlare
         | captcha, even when doing crawling at reasonable speeds.
         | 
         | Rules For Thee but Not for Me
        
           | chii wrote:
           | > many anti-scraping services explicitly allow access to
           | Google and maybe couple of other search engines.
           | 
           | because google (and the couple of other search engines)
           | provide enough value that offset the crawler's resource
           | consumption.
        
             | JasonADrury wrote:
             | That's cool, but it's impossible for anyone to ever build a
             | competitor that'd replace google without bypassing such
             | services.
        
           | ehhthing wrote:
           | You say this like robots.txt doesn't exist.
        
             | toofy wrote:
             | it almost sounds like they're saying the contents of
             | robots.txt shouldn't matter... because google exists? or
             | something?
             | 
             | implying "robots.txt explicitly says i can't scrape their
             | site, well i want that data, so im directing my bot to take
             | it anyway."
        
             | sitzkrieg wrote:
             | so many things flat out ignore it in 2026 let's be real
        
           | ErroneousBosh wrote:
           | Why are you scraping sites in the first place? What
           | legitimate reason is there for you doing that?
        
             | digiown wrote:
             | Dunno, building a Google competitor? How do you think
             | Google got started?
        
             | Ronsenshi wrote:
             | Just today I wanted to get a list of locations of various
             | art events around the city which are all located on the
             | same website, but which does not provide a page with all
             | events happening this month on a map. I need a single map
             | to figure out what I want to visit based on distance I have
             | to travel, unfortunately that's not an option - only option
             | is to go through hundreds of items and hope whatever I
             | picked is near me.
             | 
             | Do you think this is such a horrible thing to scrape? I
             | can't do it manually since there are few hundred locations.
             | I could write some python script which uses playwrite to
             | scrape things using my desktop browser in order to avoid
             | CloudFlare. Or, which I am much more familiar with, I could
             | write a python script that uses BeautifulSoup to extract
             | all the relevant locations once for me. I would have been
             | perfectly happy fetching 1 page/sec or even 1 page/2
             | seconds and would still be done within 20 minutes if only
             | there was no anti-scraping protection.
             | 
             | Scraping is a perfectly legal activity, after all. Except
             | thanks to overly-eager scraping bots and clueless/malicious
             | people who run them there's very little chance for anyone
             | trying to compete with Google or even do small scale
             | scraping to make their life and life of local art
             | enthusiasts easier. Google owns search. Google IS search
             | and no competition is allowed, it seems.
        
               | ErroneousBosh wrote:
               | If you want the data, why not contact the organisation
               | with the website?
               | 
               | Why is hammering the everloving fuck out of their website
               | okay?
        
         | megous wrote:
         | I'd still like the ability to just block a crawler by its IP
         | range, but these days nope.
         | 
         | 1 Hz is 86400 hits per day, or 600k hits per week. That's just
         | one crawler.
         | 
         | Just checked my access log... 958k hits in a week from 622k
         | unique addresses.
         | 
         | 95% is fetching random links from u-boot repository that I
         | host, which is completely random. I blocked all of the
         | GCP/AWS/Alibaba and of course Azure cloud IP ranges.
         | 
         | It's almost all now just comming of a "residential" and
         | "mobile" IP address space from completely random places all
         | around the world. I'm pretty sure my u-boot fork is not that
         | popular. :-D
         | 
         | Every request is a new IP address, and available IP space of
         | the crawler(s) is millions of addresses.
         | 
         | I don't host a popular repo. I host a bot attraction.
        
           | kstrauser wrote:
           | I've been enduring that exact same traffic pattern.
           | 
           | I used Anubis and a cookie redirect to cut the load on my
           | Forgejo server by around 3 orders of magnitude:
           | https://honeypot.net/2025/12/22/i-read-yann-espositos-
           | blog.h...
        
             | plagiarist wrote:
             | Aha, that's where the anime girl is from. What sort of
             | traffic was getting past that but still thwarted by the
             | cookie tactic?
             | 
             | I guess the bots are all spoofing consumer browser UAs and
             | just the slightest friction outside of well-known tooling
             | will deter them completely.
        
               | kstrauser wrote:
               | Yep, that's why that's all over the place now. The cookie
               | thing is more of a first line of defense. It turns away a
               | lot of shoddy scrapers with nearly no resources on my
               | side. Anubis knocks out almost all of the remainder.
        
         | mrweasel wrote:
         | First of: Google has not once crashed one of our sites with
         | GoogleBot. They have never tried to by-pass our caching and
         | they are open and honest about their IP ranges, allowing us to
         | rate-limit if needed.
         | 
         | The residential proxies are not needed, if you behave. My take
         | is that you want to scrape stuff that site owners do not want
         | to give you and you don't want to be told no or perhaps pay a
         | license. That is the only case where I can see you needing a
         | residential proxies.
        
           | TZubiri wrote:
           | >The residential proxies are not needed, if you behave
           | 
           | I'm starting to think that somee users in hackernews do not
           | 'behave' or at least they think they do not 'behave' and
           | provide an alibi for those that do not 'behave'.
           | 
           | That the hacker in hackernews does not attract just hackers
           | as in 'hacking together features' but also hackers as in
           | 'illegitimately gaining access to servers/data'
           | 
           | As far as I can tell, as a hacker that hacks features
           | together, resi proxies are something the enemy uses. Whenever
           | I boot up a server and get 1000 log in requests per second
           | and requests for commonly exploited files from russian and
           | chinese IPs, those come from resi IPs no doubt. There's 2
           | sides to this match, no more.
        
           | tonymet wrote:
           | You can't get much crawling done from published cloud IPs.
           | Residential proxies are the only way to do most crawls today.
           | 
           | That said, I support Google working to shut these networks
           | down, since they are almost universally bad.
           | 
           | It's just a shame that there's no where to go for legitimate
           | crawling activities.
        
             | mrweasel wrote:
             | > You can't get much crawling done from published cloud
             | IPs.
             | 
             | Think about why that might be. I'm sorry, if you
             | legitimately need to crawl the net, and do so from a cloud
             | provide, your industry screwed you over with bad behaviour.
             | Go get hosting with a company that cares about who their
             | customers are, you're hanging out with a bad crowd.
        
               | tonymet wrote:
               | what industry is that? Every industry is on the cloud.
        
               | mrweasel wrote:
               | No, no they really aren't, but I was thinking the
               | "scraping industry" in the sense that that's a thing.
               | Getting hosting in smaller datacenters is simple enough,
               | but you may need to manage your own hardware, or VMs.
               | Many will help you get your own IP ranges and ASN, that's
               | going to go a long way, if you don't want to get bundled
               | in with the bad bots.
               | 
               | This differs obviously, but having an ASN in our case
               | means that we can deal you, contact you and assume that
               | you're better than random bot number 817.
        
               | tonymet wrote:
               | Scraping isn't an industry. There are legitimate and
               | illegitimate scraping pursuits.
               | 
               | There are lots of healthy / productive businesses in the
               | cloud and lots of scumbags, just like any enterprise.
               | 
               | I still have no idea about your point, by the way.
        
         | toofy wrote:
         | do we think a scraper should be allowed to take whatever means
         | necessary to scrape a site if that site explicitly denies that
         | scraper access?
         | 
         | if someone is abusing my site, and i block them in an attempt
         | to stop that abuse, do we think that they are correct to tell
         | me it doesn't matter what i think and to use any methods they
         | want to keep abusing it?
         | 
         | that seems wrong to me.
        
       | AugustoCAS wrote:
       | This was easy because it's a Chinese company.
       | 
       | The largest companies in this space that do similar this
       | (oxylabs, brighdata,etc) have similar tactics but are based in a
       | different location.
        
         | arewethereyeta wrote:
         | brighdata = Israel i think oxylabs = Lithuanian, child of
         | NordVPN
        
       | niedbalski wrote:
       | Thanks google for saving us. I guess this is the equivalent of
       | rival narcos fighting each other.
        
         | tclancy wrote:
         | But would make for a much less interesting dramatic series. I
         | bookmarked that link for the next time I have insomnia.
        
       | moffkalast wrote:
       | I see Google is doing their best to stamp out the competition.
        
       | g947o wrote:
       | > attackers can mask their malicious activity by hijacking these
       | IP addresses.
       | 
       | Sounds like "malicious activity" == "scraping activities that
       | don't come from Google"
        
       | arewethereyeta wrote:
       | The big players are not taken out. This is sand thrown at our
       | faces.
        
       | kingforaday wrote:
       | Google shows a samaple of the IOCs but Google Trust Services have
       | issued a number of the SSL certs for those domains that have not
       | been revoked (yet?).
       | 
       | Only looking at the:
       | 
       | - a8d3b9e1f5c7024d6e0b7a2c9f1d83e5.com
       | 
       | - af4760df2c08896a9638e26e7dd20aae.com
       | 
       | - cfe47df26c8eaf0a7c136b50c703e173.com
       | 
       | Looks like a standard MD5 hash domain pattern of which currently
       | there are:                 user@host:/data/domains/2026/01/30$
       | zgrep -iE '^[a-f0-9]{32}\.com$' com.txt_domains.gz | wc -l
       | 3005
       | 
       | If you look at some of the others (not listed in Google's IOC),
       | they tend to have a pattern with their SSL certs e.g.:
       | 
       | - 0e6f931862947ad58bf3d1a0c5a6f91f.com                 X509v3
       | Subject Alternative Name:
       | DNS:0e6f931862947ad58bf3d1a0c5a6f91f.com,
       | DNS:effc538138d9342c547c5df42b03d81e.com, DNS:gulfclouds.site,
       | DNS:xinchaobccgba.net
       | 
       | - 17e4435ad10c15887d1faea64ee7eac4.com                 X509v3
       | Subject Alternative Name:
       | DNS:0dcbdf154c39288c91feb076795715e1.com,
       | DNS:0e8843e8f10f20eeef59f0076e4feb83.shop,
       | DNS:1014a1fb60e1b91404682e572ede6b4f.com,
       | DNS:178281a79266d2faa3e578f23c8a361e.com,
       | DNS:17e4435ad10c15887d1faea64ee7eac4.com,
       | DNS:19f75b2642320e0606f5e38ce9fbcf17.com, DNS:1vxe.com,
       | DNS:292893d0b31941e1c0d8eb01235be4eb.com,
       | DNS:2b1e642f3a60130d1b2cf244891bef0d.info,
       | DNS:354542342b7d2ddb66c97240d0c770dc.com,
       | DNS:37d993ba8c9284bedad2a3177dfc44a6.info,
       | DNS:3857036aaeedf670bbcca926945b50dd.com,
       | DNS:3961f3fa3a6bacc5c4f28e81c60f4169.com,
       | DNS:3eb4b3a3f8722b60d6ba2de7dd5f2523.org,
       | DNS:42a17c71c0d6f2a6d7e135f8e869ab3f.com,
       | DNS:4edd3793da3080640431430a4da57a86.org,
       | DNS:4f5667d51451a2060067a97bcddf077f.info,
       | DNS:5006cc38aff1ebc7d1232037fd592c60.net,
       | DNS:54c35ec930f5b52fd9505778bb9c3f00.com,
       | DNS:60255ec5427c2ba9a80b9c7648dd62e9.com,
       | DNS:638d0e352728a04bb56ca102e54b8c9b.xyz,
       | DNS:69234f9b18c0b4d572dc553dbfdb8f52.com,
       | DNS:6934addf679d79a79f0bfc2ff090b104.com,
       | DNS:694b64c9b41c17a229d92156d14a4ffd4.com,
       | DNS:6eba8c4def89561e1cee02bb3c9b373d.info,
       | DNS:7050f8c6563ff47465932e3838dc06fd.com,
       | DNS:72ad0de0a556f763e0629c64c694df4c.com,
       | DNS:86f7020358afaf71baeee5782b6264e4.xyz,
       | DNS:88f2f20d26dcabeafd2f9d24e7ea4e50.com,
       | DNS:911f4bf053ee3dadae1ca6bfdf40a817.com
       | 
       | would there be any reason any of these would be legitimate?
        
       | Rasbora wrote:
       | I've helped multiple people remove residential proxy malware that
       | was turning their network into a brightdata exit node and they
       | had no idea / did not consent to it. Why is google selectively
       | targeting one provider while letting others operate freely?
       | 
       | You can check if your network is infected here:
       | https://layer3intel.com/is-my-network-a-residential-proxy
        
       | buddylw wrote:
       | This problem isn't going until we find a better solution to
       | scraping than using your IP address as a passport.
        
       | avastel wrote:
       | Since I was also tracking this proxy network as part of my side
       | project, I wrote a short blog post + give access to 16m+ proxy
       | IPs IoCs that belong to this proxy network:
       | https://deviceandbrowserinfo.com/learning_zone/articles/insi...
       | 
       | Note that even after the disruption, I'm still able to route
       | millions of requests/day through IP IDEA's network
        
       | nektro wrote:
       | objectively good news. thanks, google.
        
       ___________________________________________________________________
       (page generated 2026-01-31 23:01 UTC)