[HN Gopher] Building a Distributed Data Warehouse Without Data L...
       ___________________________________________________________________
        
       Building a Distributed Data Warehouse Without Data Lakes
        
       Author : TheIronYuppie
       Score  : 19 points
       Date   : 2023-11-02 17:53 UTC (5 hours ago)
        
 (HTM) web link (blog.bacalhau.org)
 (TXT) w3m dump (blog.bacalhau.org)
        
       | TheIronYuppie wrote:
       | Heya all!
       | 
       | As always, we'd love to hear more if you have any
       | questions/comments/concerns.
       | 
       | aronchick (at) expanso (dot) io
       | 
       | Disclosure: I co-founded Bacalhau
        
       | slt2021 wrote:
       | I didnt get what this does, is it just running a containerized
       | workloads on a cluster? like a docker swarm?
       | 
       | what makes it better than docker swarm or kubernetes or nomad or
       | any other container orchestration tool ?
        
         | TheIronYuppie wrote:
         | Heya!
         | 
         | The idea is that this provides a more distributed friendly way
         | to do the above. So imagine you'd like to run DuckDB queries
         | over flat files spread in 7 different regions/zones/etc
         | 
         | Kubernetes is great, but requires using a central control
         | plane.
         | 
         | Swarm is also great but is more focused on job execution,
         | rather than targeting data, results, etc
         | 
         | We love both ecosystems (our cofounders and team are made up of
         | very early contributors to both), but we think there's room for
         | a compliment of something that is more resilient to poor
         | network conditions and allows for data first scheduling and in-
         | place execution of any arbitrary job (containerized or not).
         | 
         | Does that help?
         | 
         | Disclosure: I co-founded Bacalhau
        
         | FridgeSeal wrote:
         | It doesn't appear to be a container-orchestration tool at all.
         | 
         | It's a distributed OLAP database of sorts. More like a
         | snowflake/ ClickHouse/ Trino (nee Presto).
        
           | chatmasta wrote:
           | I think it's more generalized distributed compute, with OLAP
           | being one of the use cases. There are a lot of videos on
           | YouTube if you search "Bacalhau cluster" or similar,
           | including what appears to be their legacy channel [0] (which
           | I saw a year ago and found very impressive) and a newer one
           | [1], but also conference talks from other channels.
           | 
           | I haven't been following the project, but I remember finding
           | the original demo to be very impressive - I don't remember
           | the details off-hand, but IPFS was somehow involved. Seems
           | like a lot of work has taken place since then. Docs are here
           | [2]
           | 
           | [0] https://www.youtube.com/@bacalhau3295
           | 
           | [1] https://www.youtube.com/@bacalhauproject
           | 
           | [2] https://docs.bacalhau.org/
        
             | TheIronYuppie wrote:
             | Thank you very much! Yep you nailed it - OLAP is just one
             | of the use cases. We support LOTS of different examples
             | [1]. We also support NON-container workloads like arbitrary
             | exeuctors [2] and WASM [3]. Our goal is to just make it
             | just as easy to build complicated, distributed workloads as
             | it is with Swarm, Kubernetes, Nomad, etc, but adding in
             | cross network reliability and queueing.
             | 
             | Oh, one other thing - we do support IPFS [4] natively but
             | it's optional; we also support S3 [5], local disk, etc.
             | 
             | [1] https://docs.bacalhau.org/examples
             | 
             | [2] https://blog.bacalhau.org/p/distributed-compute-
             | platform-bac... <- look for pluggable executors (it's in
             | beta)
             | 
             | [3] https://docs.bacalhau.org/examples/workload-
             | onboarding/rust-...
             | 
             | [4] https://docs.bacalhau.org/running-node/storage-
             | providers/#ip...
             | 
             | [5] https://docs.bacalhau.org/references/job-
             | specification/input...
        
       ___________________________________________________________________
       (page generated 2023-11-02 23:02 UTC)