[HN Gopher] Show HN: Replicate App-Embedded DuckDBs into Postgre...
       ___________________________________________________________________
        
       Show HN: Replicate App-Embedded DuckDBs into PostgreSQL with
       SyncLite OSS
        
       Author : syncliteio
       Score  : 40 points
       Date   : 2024-09-02 14:30 UTC (8 hours ago)
        
 (HTM) web link (github.com)
 (TXT) w3m dump (github.com)
        
       | syncliteio wrote:
       | SyncLite Open Source (https://www.synclite.io/) ==>
       | 
       | - SyncLite Logger, a wrapper JDBC driver, supporting popular
       | embedded databases (SQLite, DuckDB, Apache Derby, H2, HyperSQL),
       | creates transactional sql logs for embedded dbs, ships them to
       | configured staging storage (
       | S3/MinIO/SFTP/Kafka/OneDrive/GoogleDriver etc)
       | 
       | - Several application instances each creating multiple embedded
       | databases/devices are synchronized in real-time on the staging
       | storage.
       | 
       | - SyncLite consolidator, a centralized tool, continuously
       | processes device logs, generates CDC logs, replicates and
       | consolidates staged devices into a wide range of industry leading
       | database, data warehouses and data lakes.
       | 
       | Look forward to feedback and suggestions.
        
       | matharmin wrote:
       | Is this Java/JVM-specific? How are changes captured on the
       | client?
       | 
       | From reading through the docs it seems like the CDC work is done
       | in the JDBC driver (SyncLite Logger), but I had to read quite far
       | to see that, and it's difficult to put together how it works and
       | in what types of applications this can be used. For example, can
       | I capture changes to a SQLite or DuckDB database in an Android or
       | iOS application?
       | 
       | I'm interested to see the approaches used, since there is very
       | little out there for capturing changes in embedded databases.
        
         | syncliteio wrote:
         | SyncLite Logger is JDBC wrapper/driver, can be consumed in Java
         | and Python applications:
         | 
         | Few code samples are shown here:
         | https://github.com/syncliteio/SyncLite and
         | https://github.com/syncliteio/SyncLite/tree/main/synclite-co...
         | 
         | A jsp-servlet sample web app is here:
         | https://github.com/syncliteio/synclite-sample-web-app
         | 
         | SyncLite logger transactionally captures the exact sql
         | statements (DDLs and DMls) as executed on the underlying
         | embedded databases, into sqllog files.
         | 
         | The log files for each device (i.e. database) are shipped to a
         | configured staging storage(Local FS/SFTP/S3/MinIO/Kafka etc).
         | 
         | SyncLite consolidator continuously looks for new devices and
         | new logs for each devices.
         | 
         | The received raw sql logs are not directly applicable on final
         | destination as they are logical statements, may involve complex
         | dependencies/joins e.g. insert into select <cols> from t1 JOIN
         | t2 ...
         | 
         | Consolidator applies the incoming sql logs transactionally to a
         | native embedded database (SQLite) and in the process generates
         | record level CDC logs by identifying exact changed records and
         | tables
         | 
         | The CDC logs are record-level INSERT, UPDATE, DELETE
         | statements, DDL statements : CREATE TABLE, ALTER TABLE, DROP
         | TABLE, RENAME TABLE.
         | 
         | The CDC log files as generated are applied to final destination
         | databases.
         | 
         | A summary of this is here :
         | https://www.synclite.io/synclite/sync-ready-apps
         | 
         | Thank you for the feedback and we are improving the
         | documentation.
        
           | matharmin wrote:
           | Thanks, that gives a good overview
        
       | eknkc wrote:
       | I've read the initial paragraphs of Readme and I have no idea
       | what it does. I think a technical no buzzword summary would be
       | good.
        
         | syncliteio wrote:
         | SyncLite at the very core offers two key capabilities :
         | 
         | 1. A holistic solution for Database Replication/Consolidation
         | for numerous embedded databases used in several instances of
         | edge/desktop applications, into centralized databases/dw/data
         | lake etc.
         | 
         | 2. A holistic data streaming solution offering a Kafka Producer
         | and SQL API designed for final destination data delivery.
        
           | underbiding wrote:
           | "holistics", "edge", "final destination data delivery" are
           | all buzz words.
        
       | klysm wrote:
       | The whole concept here feels backwards to me. Why are we writing
       | into an OLAP database and replicating to an OLTP one?
        
       | buremba wrote:
       | Synclite looks great for companies that are starting to build
       | from scratch. Ingest data in an embedded database and via CDC
       | move the data into a centralized database. I'm also trying out a
       | similar idea but only it's a bit reverse approach, from Cloud
       | Warehouse -> Embedded DuckDB to reduce the compute cost for BI
       | and embedded analytics use-cases. The combination of cloud and
       | embedded databases are the future IMO.
       | 
       | The tech such as Apache Iceberg and embedded DuckDB enables
       | querying Snowflake + BigQuery tables directly from your BI tool,
       | without any compute cost: https://github.com/buremba/universql
        
       ___________________________________________________________________
       (page generated 2024-09-02 23:00 UTC)