[HN Gopher] Crawling More Politely Than Big Tech
       ___________________________________________________________________
        
       Crawling More Politely Than Big Tech
        
       Author : pkghost
       Score  : 12 points
       Date   : 2025-01-01 22:57 UTC (4 days ago)
        
 (HTM) web link (cameronboehmer.com)
 (TXT) w3m dump (cameronboehmer.com)
        
       | pkghost wrote:
       | A few implementation details from building a hobby crawler
        
       | ndriscoll wrote:
       | If you have cache headers, why use HEAD? Are servers more likely
       | to handle HEAD correctly than including them on the GET?
        
       ___________________________________________________________________
       (page generated 2025-01-05 23:00 UTC)