[HN Gopher] Making 1000 HTTP requests per second in Python
       ___________________________________________________________________
        
       Making 1000 HTTP requests per second in Python
        
       Author : jonluca
       Score  : 24 points
       Date   : 2021-12-24 14:51 UTC (8 hours ago)
        
 (HTM) web link (blog.jonlu.ca)
 (TXT) w3m dump (blog.jonlu.ca)
        
       | No1 wrote:
       | If anyone is wondering why the threaded implementation was slower
       | 
       | > pool = ThreadPool(40)
        
         | AtlasBarfed wrote:
         | How is python doing a thread pool with the GIL?
        
         | akvadrako wrote:
         | Many people still don't understand you can run millions of
         | threads on Linux on a beefy server. The bottleneck with python
         | will always be the CPU doing something with those requests.
        
         | remram wrote:
         | To be fair they did try multiple concurrency factors for
         | asyncio and found that 60 was fastest, so this is not too far.
         | Hopefully.
        
       | ArtTimeInvestor wrote:
       | For what its worth, this would be the bash equivalent:
       | for i in {1..1000}          do             wget --quiet
       | "somehost/$i" -O "$i.html" &         done         wait
       | 
       | Takes 1.8 seconds here. But will obviously depend on the network
       | situation and server. And there probably is something faster than
       | wget.
        
       | axiosgunnar wrote:
       | Or switch to a language more suited for the job and you can add a
       | ,,k" to that 1000
        
         | kimi wrote:
         | Exactly - 1000 req/s on modern hardware sounds a lot like "Look
         | ma, I was able to slow it down!"
        
         | hackingforfun wrote:
         | What language would you use for web scraping?
        
           | remram wrote:
           | Go is sort of made for that, as is NodeJS. Rust and Java are
           | other languages I assume would make short work of a task like
           | this.
        
             | hackingforfun wrote:
             | In terms of NodeJS vs Python, specifically for web
             | scraping, would you choose NodeJS? If so, why?
             | 
             | I'm more familiar with NodeJS but I'm working with a team
             | that is leaning towards using Python for web scraping, so
             | that's why I'm asking. They said spinning up multiple
             | processes in Python is easier so at scale it will work
             | better.
             | 
             | I know you can use the Cluster module to have child
             | processes in NodeJS, but in my experience it's a bit of a
             | pain to use, although it's not always required to use
             | anyway, at least when only using NodeJS as a web server (as
             | long as you have multiple NodeJS instances, in case one
             | goes down). Web scraping is a bit different though.
             | 
             | Curious if you have any thoughts on this.
        
               | [deleted]
        
       ___________________________________________________________________
       (page generated 2021-12-24 23:01 UTC)