[HN Gopher] Show HN: ReverseETL - The open-source alternative to...
       ___________________________________________________________________
        
       Show HN: ReverseETL - The open-source alternative to Hightouch and
       Census
        
       Author : nagstler
       Score  : 52 points
       Date   : 2024-03-02 08:38 UTC (14 hours ago)
        
 (HTM) web link (github.com)
 (TXT) w3m dump (github.com)
        
       | volderette wrote:
       | Will this stay open source or will it end up as another limited
       | open core data product?
        
         | nagstler wrote:
         | This project will always remain open-source.
        
           | volderette wrote:
           | That sounds great thanks!
        
       | tootie wrote:
       | I understand Reverse ETL as a concept, but why does it require
       | different software to solve? Aren't most ETL/ELT tools designed
       | to move data from any source to any other source? We've got a
       | pretty vanilla requirement of pulling data from CRM to data
       | warehouse for reporting and then pushing rollups back to CRM. And
       | we use the same tool for both flows.
        
         | nagstler wrote:
         | there are many nuances to sending data from data store to biz
         | tools, that don't apply to ETL tools. For example, data
         | warehouses are designed to take as much data and in whatever
         | format, you can throw at them. Biz tools on the other hand have
         | very custom payload/API formats, rate limits, and more.
        
         | whynotmaybe wrote:
         | From Wikipedia : extract, transform, load (ETL) is a three-
         | phase process where data is extracted, transformed (cleaned,
         | sanitized, scrubbed) and loaded into an output data container.
         | The data can be collated from one or more sources and it can
         | also be output to one or more destinations.
         | 
         | Reverse is only meaningful when you're used to a specific view
         | that ETL can only load into a single destination. (Usually data
         | warehouse)
         | 
         | Just like the term "push notification" which are a specific
         | kind of communication from the server to a mobile device but in
         | fine, it's the device calling a service to know if there are
         | any notification waiting for them. A Message Queue in other
         | words...
        
       | t0mas88 wrote:
       | Cool! I'm working on something on the other side of the same
       | pipeline. Real time data ingestion and identity resolution,
       | delivering into a data lake.
       | 
       | Have you considered adding source connectors for S3 based data
       | lakes? For example Parquet files or Delta Lake? Maybe via AWS
       | Athena to make it similar to the Red Shift connector?
        
       ___________________________________________________________________
       (page generated 2024-03-02 23:01 UTC)