[HN Gopher] Topfew and Amdahl
       ___________________________________________________________________
        
       Topfew and Amdahl
        
       Author : zdw
       Score  : 29 points
       Date   : 2021-03-29 17:07 UTC (1 days ago)
        
 (HTM) web link (www.tbray.org)
 (TXT) w3m dump (www.tbray.org)
        
       | lsb wrote:
       | After you shard some of the computation out to different
       | processors to filter before single core aggregation, then you can
       | push down some pre aggregation to limit the communication between
       | parallel threads, and it keeps going!
       | 
       | Super interesting to see parallelism for reading logs this way,
       | like for when you have too many logs for one core and too few for
       | Spark
        
         | timbray wrote:
         | That turns out to be hard because in your typical log file you
         | have bursts of requests localized to a particular part of the
         | file that get something into the top-N without even appearing
         | in other segments. So essentially you can end up having to do a
         | complete N-way merge between the segments to make sure you
         | really find the top hits.
        
       | webmaven wrote:
       | _> I'm using a 3.2GB file containing 13.3 million lines of Apache
       | logfile, half from 2007 and half from 2020._
       | 
       | I'm not sure if this was deliberate, but I hadn't come across
       | this use of 'logfile' as an uncountable noun (eg. like 'water',
       | 'money', or 'data') before.
       | 
       | Having occasionally dug into logs over the years, it certainly
       | _feels_ right, though asking  "how much logfile?" strikes me as a
       | bit Doge, but I'm sure that will wear off.
       | 
       | Linguistic evolution in action, folks!
        
       ___________________________________________________________________
       (page generated 2021-03-30 23:03 UTC)