[HN Gopher] Using Awk and R to parse 25tb (2019)
       ___________________________________________________________________
        
       Using Awk and R to parse 25tb (2019)
        
       Author : xrayarx
       Score  : 62 points
       Date   : 2023-10-07 18:50 UTC (4 hours ago)
        
 (HTM) web link (livefreeordichotomize.com)
 (TXT) w3m dump (livefreeordichotomize.com)
        
       | _a_a_a_ wrote:
       | I'm a database guy so everything looks like a database problem to
       | me, but I'm not sure how this would fit in (as I'm completely
       | unfamiliar with the data used here). Can anyone more
       | knowledgeable than me suggest whether a database, on a
       | conventional server with some decent RAM and a bunch of SSDs
       | would have have worked and perhaps been cheaper?
       | 
       | (Edit: OK, SSDs in 2019 _might_ not have been affordable but
       | spinny disks were cheap and still pretty fast)
        
       | isoprophlex wrote:
       | Now that DuckDB has S3 support, i guess a linux box, DuckDB and
       | some light SQL'ing is all you need?
        
       | laurent_du wrote:
       | Querying by rsid is clearly a bad idea. You want to partition by
       | chromosome (and sample-id in this case) and sort by position.
       | When looking for a given snp, the parquet reader will go through
       | the metadata to only read the data page that contains the given
       | position. Unless your pages are huge, read time will be super
       | small (and cost-efficient since you don't fetch too much data).
       | Since the data is static I would want to try storing all the
       | sample data and metadata in arrays. (For non-static data you
       | can't do that because you won't be able to edit the arrays later
       | - you can only add new rows to the parquets.) I am not really
       | sure I understand what the author is doing, sounds like he wanted
       | to sort by position but he failed to do so and decided to bin
       | instead? I agree that Awk is very useful in this kind of
       | problems.
        
       | dang wrote:
       | Related:
       | 
       |  _Using AWK and R to parse 25TB_ -
       | https://news.ycombinator.com/item?id=20293579 - June 2019 (104
       | comments)
       | 
       | Recent and also related:
       | 
       |  _Exploratory data analysis for humanities data_ -
       | https://news.ycombinator.com/item?id=37792916 - Oct 2023 (38
       | comments)
        
       | xnx wrote:
       | Reminds me of "Command-line Tools can be 235x Faster than your
       | Hadoop Cluster (2014)"
       | https://news.ycombinator.com/item?id=30595026
        
       | corytheboyd wrote:
       | Awk is such a nice little tool! It doesn't even have to be an
       | archaic one liner akin to a stack overflow answer. You can write
       | well structured, easy to follow awk programs, that use variables,
       | sane conditional logic, matching functions, etc. You can do all
       | of this by referencing the man page that you already have, and
       | nothing else. It's a bit like something between bash and perl--
       | enough functionality to accomplish non-trivial file processing
       | tasks, but not a fully featured programming language. Which is
       | perfect when it's perfect.
        
       ___________________________________________________________________
       (page generated 2023-10-07 23:01 UTC)