[HN Gopher] Bad bots account for most internet traffic? Analysis
___________________________________________________________________
Bad bots account for most internet traffic? Analysis
Author : LinuxBender
Score : 97 points
Date : 2023-11-16 16:16 UTC (6 hours ago)
(HTM) web link (www.securityweek.com)
(TXT) w3m dump (www.securityweek.com)
| kredd wrote:
| I believe it. Very insignificant data point but - I've worked for
| small scale startups and larger scale enterprises, and in both
| cases you can see bots and crawlers trying to do random stuff. I
| don't think it was actually %70+ of the traffic, but can see
| specific types of businesses getting even more hits from them.
| andai wrote:
| Would this be solved by just providing APIs? Scraping is a pain
| and people only do it when there's no API. (Though these days a
| lot of businesses offer unofficial 3rd party APIs by scraping.)
|
| It's also stupid and massively wastes computation and traffic on
| both ends.
|
| Then people would think "but then I'd have to pay money to offer
| a free API", but they're already doing that in a more expensive
| way via the web interface anyway.
| gabipurcaru wrote:
| > people only do it when there's no API
|
| Legitimate scrapers, maybe. Everyone else does it to circumvent
| the API limitations, by posing as real traffic. APIs imply API
| keys which can be traced and banned.
| yjftsjthsd-h wrote:
| And what makes a scraper legitimate?
| cheschire wrote:
| In the case of GP, I believe they're referring to
| legitimate intent of the bot owner.
| JohnFen wrote:
| Legitimate scrapers adhere to the websites terms of
| service, for starters.
| ben0x539 wrote:
| Well, that's what happens when you make the API more limited
| than what people can do with scraping (or with using the
| public "our own web client" API key).
| thejohnconway wrote:
| No, there's a huge number of bots out there just doing stupid
| hacky stuff like trying for wp/admin/. Vast numbers of what I'd
| characterise as script-kiddy type attempts. Looking at the raw
| logs of a webserver not behind something like CloudFlare really
| opened my eyes to this. There's an extraordinary number of
| garbage requests.
| pixl97 wrote:
| Yea, there was an HN thread complaining about WAF's a few
| days ago, but damn, if you're on the public internet having
| some kind of WAF cuts down the bullshit responses by a
| massive amount.
|
| Same thing goes for email. I had one particular domain I
| managed spam filtering on and for every 10,000 messages we
| received, 1 was delivered to users. The amount of crap
| requests on the internet is insane.
| kube-system wrote:
| And that's just requests on 80/443. Hook up an IDS to the WAN
| interface and, you'll see thousands of attempts per day to
| exploit vulnerabilities across other services too.
|
| My low-traffic personal projects get 90%+ malicious traffic.
| gumballindie wrote:
| I think a more efficient approach would be to prosecute those
| who do it - bandwidth costs money - and to implement mechanisms
| to deter them. A not insignificant number of ai workers think
| are entitled to using people's bandwidth, servers and
| intellectual property for free. Until a couple of such people
| are taken to task this wont stop.
| JohnFen wrote:
| > Would this be solved by just providing APIs? Scraping is a
| pain and people only do it when there's no API.
|
| Many sites intentionally make scraping as painful as possible
| because they don't want anyone scraping them.
| josefresco wrote:
| I manage a network of several hundred websites. About 2 years ago
| we started blocking bot/crawler user-agents. Some websites see
| thousands of these requests every day, skewing analytics and
| eating bandwidth.
|
| Tools like this help: https://github.com/mitchellkrogza/nginx-
| ultimate-bad-bot-blo...
| supriyo-biswas wrote:
| You're better off using a combination of IP blocklisting and
| ratelimiting, both of which use some sort of binary tree to
| speed up lookups, rather than using a mega-regex which your
| solution proposes, which, with tons of backtracking, is very
| likely slower than serving the request would have been.
|
| Not to mention that user agents are easily spoofable.
| kube-system wrote:
| If a bot's user agent doesn't look identical to a normal user,
| either the bot developer is dumb, or they are relatively well
| behaving.
|
| Anyone malicious who knows what they're doing will just send
| you a statistically common user agent string.
|
| Under most http libraries this is a one-liner:
| headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64;
| x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0
| Safari/537.36 Edg/119.0.0.0'}
| acheong08 wrote:
| Try JA3 fingerprinting. It's much more difficult to spoof than
| user agents
| nwoli wrote:
| Good bots account for the remaining 26%
| sandworm101 wrote:
| I do not believe. I do not believe that bots account for more
| traffic than all the streaming services, all the filesharing and
| all the porn crisscrossing the internet. I would believe 73% of
| requests, even 73% of page loads/scrapes, but 73% of _all
| traffic_ is too huge a number.
| rightbyte wrote:
| Bots might "watch" Youtube for ad money for the creators and
| Google, though.
|
| I mean, it would not surprise me at all if bot traffic on
| Youtube is larger as in bytes/s of video, than actual visitors.
|
| Remember that Google has no interest in disclosing how many bot
| views there are.
|
| There was this news article about Spotify banning some
| musicians accounts due to listen farming.
|
| https://www.musicbusinessworldwide.com/great-big-spotify-sca...
|
| Since the payouts are not tied to the individual watcher or
| listener, scams run rampant. I guess the temptation to
| manipulate the money flow is way higher than having a "fair"
| distribution of each subscribers money to the artists.
| kuroguro wrote:
| Most bots operate in the gray area of just getting the view
| count up and not messing with the ads. They can operate
| publicly and not have law enforcement raid their offices.
|
| Yt definitely makes an effort to prevent that. You can read
| up on how botguard's virtual machine works in some older
| writeups.
| adocomplete wrote:
| I think it's possible if for nothing else but the fact that a
| bot can easily do 10-100x or more visits than a human can on
| the web.
| Darmody wrote:
| I think that too.
|
| I watched today a couple hours of Youtube videos. How many bots
| are needed to match that much traffic?
|
| if we're talking about requests it makes sense. I just have to
| open my apache/nginx logs to see how many bots are attempting
| to get in.
| dualboot wrote:
| I'm not defending the validity of this article but it is very
| rare that Netflix, Youtube, or any other stuff backed by the
| big CDNs(Content Delivery Networks) cross actual metered
| transit connections(for the networks, not including what the
| end user has to pay for their servivces.)
| strangattractor wrote:
| I would definitely believe that Bots are responsible for %73 of
| the traffic. Not sure how one would necessarily characterize
| them as "bad" but definitely wasteful. Worked at site that
| published science. They collected page view metrics. Doing
| something simple like excluding page views that where done by
| IP addresses that continuously cycled through all articles
| resulted in over a %70 reduction in page views. Needless to say
| it was never deployed because the high view numbers made the
| authors feel good.
| wnevets wrote:
| bots are still a subset of the traffic for streaming, porn,
| etc. Whether its for ad fraud or faking popularity on streaming
| those bots are running 24/7.
|
| However I agree, on its face 73% sound ridiculous.
| blakesterz wrote:
| Here's the link to the PDF of the report:
|
| https://www.arkoselabs.com/wp-content/uploads/Breaking-Bad-B...
| yjftsjthsd-h wrote:
| I feel like this is trying very hard to push a particular
| narrative but not actually bothering to substantiate anything.
|
| From the top...
|
| > Bad Bots are increasing dramatically -- Arkose estimates that
| 73% of all internet traffic currently (Q3, 2023) comprises Bad
| Bots and related fraud farm traffic.
|
| Note that "Bad Bots" are never defined, and that 73% will never
| be explained, either to say how they mean it (73% of connections
| is _plausible_ , 73% of packets is not, not in a world with
| Netflix and YouTube), or how they would measure such a thing.
|
| Then we get into little gems like
|
| > Scraping, it must be said, is a legally murky area. It is not
| specifically illegal; but if it defies a website's published
| terms of use, it is certainly immoral.
|
| Which... that's a claim you can make, but if you just throw out
| terms like "certainly immoral" without bothering to substantiate
| them I get to laugh at you.
|
| The tendency to cast aspersions without substantiating them gets
| substantially more pronounced later:
|
| > This is a website you can use to make sure your bots aren't
| getting prevented by a website," [...] "You can purchase this
| software. It has enterprise support and so on. But it is purpose
| built to commit crime. That is what it does. And there are many
| other different websites like this, but they look like legitimate
| businesses. It is a good example of a product purpose built to
| commit fraud."
|
| I'll grant that businesses don't like people bypassing their
| protections, but that doesn't make it automatically illegal or
| fraud to scrape or even to programmatically interact with sites.
| I'm not naive enough to think that it's not being used for
| illegal stuff, but that doesn't mean you get to paint the tools
| with a single brush.
| JohnFen wrote:
| > Note that "Bad Bots" are never defined
|
| This stuck out to me as well. What are "bad bots" and how do
| they differ from "good bots", and how are the authors
| determining which is which?
| PartiallyTyped wrote:
| > world with Netflix and YouTube
|
| Not in a world where pron exists.
| yjftsjthsd-h wrote:
| Okay, fine, a world where _video streaming_ exists, in
| whatever particulars.
| egypturnash wrote:
| "The top five categories of Bad Bot attacks are fake account
| creation, account takeovers, scraping, account management, and
| in-product abuse."
|
| - second paragraph, first sentence
| yjftsjthsd-h wrote:
| That's a description, not a definition. If they said that
| they classified a bot as bad if it engaged in one of five
| described behaviors, that would be a definition, but that's
| not what they've done. (And if that _was_ the definition they
| were using, then I 'm going to call out scraping and account
| management as not sounding like malicious behavior)
| swatcoder wrote:
| The actual report from Arkose (linked elsewhere in the
| comments) is much more detailed in what they're measuring and
| reporting.
|
| That said, the language around the "73% of traffic" figure is
| still a little ambiguous but most plausibly means "73% of
| requests/sessions to the significant commercial properties we
| studied"
|
| Assuming that's what they mean, it's reasonable that bots are
| banging on major sites three times more often than legitimate
| users.
|
| I think the linked article is doing the sloppy sensationalizing
| and the report appears to be much less so.
| yunwal wrote:
| What percentage of those "bad bots" are actually just
| crawlers for search engines and other internet indexes that
| aren't actually bad?
| danShumway wrote:
| To build on the above, it's worth thinking about the transition
| that happens between those two quotes:
|
| > Scraping, it must be said, is a legally murky area. It is not
| specifically illegal
|
| There's already a jump happening here from "it's not
| specifically illegal" to "it's a gray area because it's
| immoral." Even if it was certainly and unambiguously immoral,
| that does not make it automatically a legally murky area. In
| fact, scraping seems to be reasonably established as
| specifically _legal_.
|
| So the article is already kind of edging away from reality,
| where they acknowledge "this is not a crime" but they want to
| blur that and say that it's legally dubious, even though what
| they really mean is that they believe it's immoral.
|
| Which, fine, I don't respect it but whatever. But then we jump
| to:
|
| > But it is purpose built to commit crime.
|
| Wait, no. Scraping is not specifically illegal. This is the
| emulator debate again; we have something that really is
| strongly implied to be legal and has won court cases before.
| It's not like it's an untested gray area, we know that there
| are scenarios where scraping websites against the permission of
| the website owner is not illegal.
|
| So the article pretends it's more of a gray area than it
| actually is, and then once it has thrown doubt on it, drops the
| pretense entirely and calls it illegal. Not all scraping is
| fraud and not all scraping is illegal, and it's pretty galling
| for the article to admit that scraping is not illegal and then
| to turn around and pretend that they didn't just admit that.
|
| > It is also a good example of crime-as-a-service.
|
| No, it's not. The article _admitted_ that scraping is not
| specifically illegal. Tools to help scrape websites are not
| "crime-as-a-service" because objectively (and there is legal
| precedent backing this up) scraping is not a crime.
|
| ----
|
| The propaganda strategy being employed here is to admit a piece
| of factual information (scraping is not illegal), however
| hesitantly, and then to pretend that it was never admitted.
|
| If anyone calls the propagandist out, they can point to their
| admission and say, "no, I acknowledged that it's not illegal,
| see". But then for every other part of the conversation, they
| just ignore that and act like it is illegal, even though _at
| best_ all they 've put forward is that they think it's more
| dubious than courts currently recognize.
|
| They concede the point only to the degree they're forced to and
| only to the degree that would prevent someone from saying that
| they're lying or ignoring reality. But as soon as the context
| changes, they go back to acting like the point was never
| conceded.
|
| This has the benefit of not only allowing them to make the same
| arguments they wanted to make before, but also sort of trains
| listeners to think of the original conceded point as more murky
| than it actually is. "Sure," the reader thinks, "it's
| technically not illegal, but at the point we're talking about
| crime-as-a-service and illegal tools, surely there's something
| legally dubious going on, this must be more of a gray area" --
| even though not only has the article not backed up that idea,
| it's actually admitted that the idea is wrong.
| yjftsjthsd-h wrote:
| Well said:)
|
| > There's already a jump happening here from "it's not
| specifically illegal" to "it's a gray area because it's
| immoral. Even if it was certainly immoral, that does not make
| it automatically a legally murky area. In fact, scraping
| seems to be reasonably established as specifically legal.
|
| Yeah, I didn't want to get distracted from my main arguments,
| but that stuck out to me too - in my non-lawyer amateur
| understanding, the LinkedIn case[0] means that in the US
| scraping very much is _explicitly legal_.
|
| [0] https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn
| danShumway wrote:
| Yeah, IANAL but that is my understanding as well. You have
| to love the boldness of reframing "a court has specifically
| ruled this legal" as "no one has ruled it illegal". It's
| technically true while also heavily implying that it's just
| untested and it could be ruled illegal at any time.
|
| It's very: "technically speaking, the Constitution doesn't
| explicitly require people to quarter troops in their own
| homes, but..."
|
| Not outright _wrong_ , just kind of missing a lot of
| context and quietly implying a lot of stuff that is wrong.
| acheong08 wrote:
| > Note that "Bad Bots" are never defined, and that 73% will
| never be explained, either to say how they mean it
|
| Arkose runs a bot detection service, OpenAI being one of their
| customers (check your requests on chat.openai.com). I'm
| guessing the 73% are the number of requests/sessions that fail
| their JavaScript fingerprinting.
|
| Not sure how reliable these numbers are as I have a Go library
| specifically for passing their fingerprinting in pure Go (No
| JS) and it has been working for a while without much change
| yjftsjthsd-h wrote:
| And in the opposite direction, I disbelieve stats about bots
| because of the number of web sites that treat my real browser
| driven by an actual human as a bot:)
|
| (My personal favorite was being blocked by cloudflare, and
| then getting an email from cloudflare telling me how great
| they are because they block so many (alleged) bots)
| actondev wrote:
| And this is why we can't have nice things. Now that we will/are
| starting to have AI-created content, and visits from bots,
| comments from AI bots.. we need some new metric I reckon
| klyrs wrote:
| I propose: time spent offline.
| sonicanatidae wrote:
| What is this, "offline", of which you speak?!?!
|
| You mean like wifi?
| kibwen wrote:
| I think "offline" means mocking your ethernet driver to
| produce packets confabulated by a locally-running copy of
| ChatGPT, so you can doomscroll the internet even without
| needing to pay Comcast for the privilege.
| h2odragon wrote:
| Naw, theres this great new game; "nonvirtual reality"
|
| its kinda expensive and dangerously addictive
|
| the resolution and refresh rate are _amazing_
| defanor wrote:
| Cloudflare refused to serve me the linked page after I enabled JS
| and passed the captcha, "The action you just performed triggered
| the security solution". Each time it happens, I wonder whether
| website owners then look at the statistics, see the reported
| blocks, and are reinforced in their belief that Cloudflare does
| something useful. But in this case I also wonder whether that
| number was derived from such blocked requests as well; whether
| they count me as a bad bot or just a bad person.
| bshowg wrote:
| From the PDF:
|
| _We analyzed tens of billions of sessions worldwide across
| industries, [...]_
|
| I think they mean: 73% of all _commercial_ sessions are
| malicious.
|
| Very sensationalist, at the end of the day they are selling
| something
| jackconsidine wrote:
| > The biggest increases in attacks from Q2 to Q3 are SMS toll
| fraud (up 2,141%),
|
| Toll Fraud is _rampant_ lately. I can tell because people have
| attempted it on nearly every consumer-facing project I 've worked
| on. Additionally, the ill-advised "text the app to your phone"
| gimmicks have disappeared (fraudsters have cleaned up on direct,
| public endpoints to send SMS).
|
| Shameless plug, but I've written a little about Toll Fraud and
| how to prevent it [0]
|
| [0] https://koptional.com/article/how-to-stop-twilio-toll-fraud/
| charlieyu1 wrote:
| I would confess that I had written a bad bot in the past,
| fetching data every time I runs it multiple times a day when the
| data source only updates once or twice a week. I didn't know
| better back then
| tripue wrote:
| The article mention sms tool fraud as the biggest increase.. We
| faced the same problem at Zenly and had to build our own anti-
| spam strategies to prevent it. We used multiple providers to
| improve our conversion rate and reduce cost. We are now building
| this as a service https://www.ding.live/ and are seeing huge
| improvements for our customers in term of cost savings and
| conversion rate.
| oriettaxx wrote:
| did anybody test the AWS feature to limit bots in application
| load balancer? it costs a lot, and we are not sure if it is worth
| oriettaxx wrote:
| does it make sense to limit incoming connection to HTTP/2 to stop
| (at least many) bots?
| renegat0x0 wrote:
| Is web scraping immoral? Silly statement. Is Google immoral in
| checking people contents? Assertions like that are on the other
| hand always bad ;-)
|
| Producing a RSS feed and clean scrapable page is always a good
| thing to do to boost your page recognition and popularity.
|
| You data does not have to be scrapable, but you page should
| always be bot friendly.
| Beijinger wrote:
| Interesting question:
|
| When Netflix was a cheap stock but it made up a significant part
| of internet traffic, I assumed it could be the next big thing.
|
| So how do you invest in "bad bots", whatever this is supposed to
| mean. Honestly, I don't trust the numbers.
| red_admiral wrote:
| Wasn't there a time where something like 95% of all e-mails were
| Viagra spam? If so then at 73% the bots still have some catching
| up to do.
| forgotmypw17 wrote:
| I help run some websites with many links. I estimate that more
| than 90%, possibly close to 99% of our traffic is crawler bots,
| even after blocking some of them based on behavior.
___________________________________________________________________
(page generated 2023-11-16 23:01 UTC)