[HN Gopher] Automatically filing issues when tracked file conten...
       ___________________________________________________________________
        
       Automatically filing issues when tracked file content changes
        
       Author : simonw
       Score  : 10 points
       Date   : 2022-04-28 17:29 UTC (2 days ago)
        
 (HTM) web link (simonwillison.net)
 (TXT) w3m dump (simonwillison.net)
        
       | JNRowe wrote:
       | Slightly tangential to this post, but related to the series.
       | 
       | Have you had any pushback from the sources you've been scraping
       | with actions? I can't imagine how lightly hitting GitHub from
       | this action is a problem, but a few of the other scrapers you've
       | discussed are hitting resources in ways we've definitely seen
       | website operators get upset about before.
       | 
       | I ask because I've been playing with a couple of scrapers for
       | info I'm interested in, but I'm a little reluctant to toggle the
       | repo visibility switch. Partly, because I don't want the grief
       | and selfishly because I don't want the source to disappear.
        
         | simonw wrote:
         | Nothing at all so far.
         | 
         | I've been pretty careful to be polite though - the majority of
         | my scrapers work no more than once an hour, and the few I've
         | done that are more frequent than that don't run more than once
         | every ten minutes (GitHub Actions can't be scheduled more often
         | than that).
         | 
         | Those tend to be sites which I would expect to be able to
         | handle the traffic very easily.
         | 
         | When I'm operating sites myself I don't tend to even notice a
         | scraper that only hits a single page every few minutes. The
         | things that hurt me are crawlers that hit every page on a site,
         | potentially more than a request per second and with traffic
         | patterns that are certain to hit un-cached sources.
        
           | JNRowe wrote:
           | Excellent, thanks!
           | 
           | I'm purposely being careful too. _Big_ backoff on any error,
           | curl 's --time-cond from git commit timestamp, etag where
           | possible, etc.
           | 
           | I'm not particularly concerned about an actual DoS'ing issue,
           | more being on the end of one of _those_ stories that pops up
           | every so often( "website owner sics lawyers on F12").
        
             | simonw wrote:
             | Oh interesting - I hadn't considered using --time-cond.
             | 
             | I'm always a bit suspicious of If-Modified-Since because
             | I'm never sure which timezones will be in effect - so I
             | tend to use ETags instead, which achieve the same goal
             | without involving any clock time.
             | 
             | I actually wrote a tiny Python CLI tool for doing those,
             | designed to be used in CI:
             | https://pypi.org/project/conditional-get/
        
       ___________________________________________________________________
       (page generated 2022-04-30 23:01 UTC)