https://github.com/bruin-data/ingestr

Skip to content
Toggle navigation
 
Sign in

  * Product
      +  
        Actions
        Automate any workflow
      +  
        Packages
        Host and manage packages
      +  
        Security
        Find and fix vulnerabilities
      +  
        Codespaces
        Instant dev environments
      +  
        Copilot
        Write better code with AI
      +  
        Code review
        Manage code changes
      +  
        Issues
        Plan and track work
      +  
        Discussions
        Collaborate outside of code
    Explore
      + All features
      + Documentation
      + GitHub Skills
      + Blog
  * Solutions
    For
      + Enterprise
      + Teams
      + Startups
      + Education
    By Solution
      + CI/CD & Automation
      + DevOps
      + DevSecOps
    Resources
      + Learning Pathways
      + White papers, Ebooks, Webinars
      + Customer Stories
      + Partners
  * Open Source
      +  
        GitHub Sponsors
        Fund open source developers
      +  
        The ReadME Project
        GitHub community articles
    Repositories
      + Topics
      + Trending
      + Collections
  * Pricing

Search or jump to...

Search code, repositories, users, issues, pull requests...

Search
[                    ]
Clear

Search syntax tips

Provide feedback

We read every piece of feedback, and take your input very seriously.

[                    ] [ ] Include my email address so I can be
contacted
Cancel Submit feedback

Saved searches

Use saved searches to filter your results more quickly

Name [                    ] 
Query [                    ]

To see all available qualifiers, see our documentation.

Cancel Create saved search
Sign in
Sign up
You signed in with another tab or window. Reload to refresh your
session. You signed out in another tab or window. Reload to refresh
your session. You switched accounts on another tab or window. Reload
to refresh your session. Dismiss alert
{{ message }}
bruin-data / ingestr Public

  * Notifications
  * Fork 2
  * Star 346
  * 

ingestr is a CLI tool to copy data between any databases with a
single command seamlessly.

bruin-data.github.io/ingestr/

License

MIT license
346 stars 2 forks Branches Tags Activity
Star
Notifications

  * Code
  * Issues 1
  * Pull requests 0
  * Actions
  * Projects 0
  * Security
  * Insights

Additional navigation options

  * Code
  * Issues
  * Pull requests
  * Actions
  * Projects
  * Security
  * Insights

bruin-data/ingestr

This commit does not belong to any branch on this repository, and may
belong to a fork outside of the repository.
 main
BranchesTags
  
Go to file
Code

Folders and files

        Name                  Name          Last commit   Last commit
                                              message        date
Latest commit

 

History

56 Commits
 
.github/workflows     .github/workflows                    

docs                  docs                                 

ingestr               ingestr                              

resources             resources                            

.dockerignore         .dockerignore                        

.gitignore            .gitignore                           

Dockerfile            Dockerfile                           

LICENSE.md            LICENSE.md                           

Makefile              Makefile                             

README.md             README.md                            

package-lock.json     package-lock.json                    

package.json          package.json                         

pyproject.toml        pyproject.toml                       

requirements-dev.txt  requirements-dev.txt                 

requirements.txt      requirements.txt                     

View all files

Repository files navigation

  * README
  * MIT license

                              [ingestr]

  Ingest & copy data from any source to any destination without any
                                code

[demo]
                            [6874747073]
---------------------------------------------------------------------

Ingestr is a command-line application that allows you to ingest data
from any source into any destination using simple command-line flags,
no code necessary.

  *  copy data from your database into any destination
  *  incremental loading: append, merge or delete+insert
  *  single-command installation

ingestr takes away the complexity of managing any backend or writing
any code for ingesting data, simply run the command and watch the
data land on its destination.

Installation

 

pip install ingestr

Quickstart

 

ingestr ingest \
    --source-uri 'postgresql://admin:admin@localhost:8837/web?sslmode=disable' \
    --source-table 'public.some_data' \
    --dest-uri 'bigquery://<your-project-name>?credentials_path=/path/to/service/account.json' \
    --dest-table 'ingestr.some_data'

That's it.

This command will:

  * get the table public.some_data from the Postgres instance.
  * upload this data to your BigQuery warehouse under the schema
    ingestr and table some_data.

Documentation

 

You can see the full documentation here.

Community

 

Join our Slack community here.

Supported Sources & Destinations

 

      Database       Source Destination
Postgres                   
BigQuery                   
Snowflake                  
Redshift                   
Databricks                 
DuckDB                     
Microsoft SQL Server       
Local CSV file             
Oracle                     
SQLite                     
MySQL                      

More to come soon!

Acknowledgements

 

This project would not have been possible without the amazing work
done by the SQLAlchemy and dlt teams. We relied on their work to
connect to various sources and destinations, and built ingestr as a
simple, opinionated wrapper around their work.

About

ingestr is a CLI tool to copy data between any databases with a
single command seamlessly.

bruin-data.github.io/ingestr/

Topics

bigquery postgresql snowflake mssql data-integration data-pipeline 
data-ingestion copy-database ingestion-pipeline duckdb

Resources

Readme

License

MIT license
Activity
Custom properties

Stars

346 stars

Watchers

4 watching

Forks

2 forks
Report repository

Releases 2

 
Oracle & CSV Support Latest
Feb 27, 2024
+ 1 release

Packages 1

 

Languages

  * Python 97.9%
  * Makefile 1.4%
  * Dockerfile 0.7%

Footer

 (c) 2024 GitHub, Inc.

Footer navigation

  * Terms
  * Privacy
  * Security
  * Status
  * Docs
  * Contact
  * Manage cookies
  * Do not share my personal information

You can't perform that action at this time.