https://www.pgflo.io/

pg_flo logopg_flo
Examples GitHub
 Learn more about a cloud offering Signup

pg_flo

The easiest way to move and transform data between PostgreSQL
databases The project is in active development. Follow the GitHub
repo for more details 

Get Started

  * Real-time Streaming

    Stream data changes in near real-time using PostgreSQL Logical
    Replication.

  * Bulk Copy

    Parallelizable bulk copy for fast initial data synchronization.

  * Powerful Transformations

    Apply regex-based transformations, mask sensitive data, and
    filter based on column values before the changes reach the
    destination.

  * Flexible Routing

    Route data seamlessly between tables - whether to differently
    named tables or the same table with custom column mappings.

# Install using Docker
docker pull shayonj/pg_flo:latest

# Start NATS server
docker run -d --name pg_flo_nats \
  --network host \
  nats:latest

# Start replicator
docker run -d --name pg_flo_replicator \
  --network host \
  shayonj/pg_flo:latest \
  replicator

# Start worker with PostgreSQL sink
docker run -d --name pg_flo_worker \
  --network host \
  shayonj/pg_flo:latest \
  worker postgres

View detailed setup instructions

Transformation Rules

Mask Email Addresses Replaces email characters with asterisks,
preserving the domain

- type: transform
  column: email
  parameters:
    type: mask
    mask_char: "*"
  operations: [INSERT, UPDATE]


Format Phone Numbers Formats 10-digit phone numbers to (XXX) XXX-XXXX
format

- type: transform
  column: phone
  parameters:
    type: regex
    pattern: "^(\\d{3})(\\d{3})(\\d{4})$"
    replace: "($1) $2-$3"
  operations: [INSERT, UPDATE]


Table Routing & Column Mapping

Flexible Table Routing Route data between differently named tables
and columns

users:
  source_table: users
  destination_table: customers
  column_mappings:
    - source: id
      destination: customer_id
    - source: username
      destination: customer_name
  operations: [INSERT, UPDATE]

Multi-Table Routing Route multiple tables with different mappings
simultaneously

orders:
  source_table: orders
  destination_table: transactions
  column_mappings:
    - source: id
      destination: transaction_id
    - source: total_amount
      destination: amount
  operations: [INSERT, UPDATE, DELETE]

Use Cases

Secure Production <> Staging Sync

Continuously sync production data to staging, leveraging powerful
transformation rules to maintain data privacy and security practices.

Data Archiving and Analytics

Sync and transform data to separate databases for archiving, auditing
and analytics purposes.

How It Works

pg_flo leverages PostgreSQL's logical replication system to capture
and stream data changes. It uses NATS as a message broker to decouple
reading from the WAL through the replicator and worker processes,
providing flexibility and scalability. Transformations and
filtrations are applied before the data reaches the destination.

Learn More

(c) 2024 pg_flo. All rights reserved.

Made with [?] and  by shayon.dev