[HN Gopher] Please make this tool "opt-in" by default
       ___________________________________________________________________
        
       Please make this tool "opt-in" by default
        
       Author : _Microft
       Score  : 18 points
       Date   : 2023-04-23 21:58 UTC (1 hours ago)
        
 (HTM) web link (github.com)
 (TXT) w3m dump (github.com)
        
       | [deleted]
        
       | iudqnolq wrote:
       | This tool seems designed to download from many domains at once
       | based on the readme section about optimizing DNS resolution.
       | 
       | Considering that wouldn't a simple optimization be to shard by
       | domain instead of arbitrarily?
        
       | cs702 wrote:
       | This exchange first made me laugh, and then it made me think:
       | 
       |  _edent: OK, so every time someone builds a tool, I have to
       | specifically opt out of it. Is that what you 're saying?
       | 
       | rom1504: Yes, for example if you want to ban the tool "Google
       | Chrome" from accessing your website, you can do so._
        
       | wmf wrote:
       | This is a scraping tool that can download a list of URLs very
       | fast. It's not an AI tool per se but the advertised use is to
       | download corpora for AI training. Traditionally, polite Web
       | spiders tried not to hit any one server too hard but when your
       | goal is to download a bunch of data from a small number of sites
       | fast you're inevitably going to be putting real load on each
       | site. I'm not sure there's a solution that can satisfy everyone.
        
         | rolph wrote:
         | * * *
        
         | gus_massa wrote:
         | I guess the "solution" is that sites detect abuse and ban the
         | ip for 24 hours. When enough users of this tool get banned,
         | they will fill the github issues tracker with complain and the
         | maintainers will have to learn how to write a polite spider.
         | 
         | We have the inverse problem in the university. We are all (many
         | thousands) behind a proxy, so some popular sites detect the
         | avalanche of request and add a captcha or other anti abuse tool
         | to everyone.
        
       | rolph wrote:
       | rom1504 /img2dataset
       | 
       | https://github.com/rom1504/img2dataset
       | 
       | https://github.com/rom1504/img2dataset#opt-out-directives
        
       ___________________________________________________________________
       (page generated 2023-04-23 23:01 UTC)