https://github.com/DataExpert-io/data-engineer-handbook Skip to content Navigation Menu Toggle navigation Sign in * Product + GitHub Copilot Write better code with AI + Security Find and fix vulnerabilities + Actions Automate any workflow + Codespaces Instant dev environments + Issues Plan and track work + Code Review Manage code changes + Discussions Collaborate outside of code + Code Search Find more, search less Explore + All features + Documentation + GitHub Skills + Blog * Solutions By company size + Enterprises + Small and medium teams + Startups By use case + DevSecOps + DevOps + CI/CD + View all use cases By industry + Healthcare + Financial services + Manufacturing + Government + View all industries View all solutions * Resources Topics + AI + DevOps + Security + Software Development + View all Explore + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Enterprise + Enterprise platform AI-powered developer platform Available add-ons + Advanced Security Enterprise-grade security features + GitHub Copilot Enterprise-grade AI features + Premium Support Enterprise-grade 24/7 support * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up Reseting focus You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} DataExpert-io / data-engineer-handbook Public * Notifications You must be signed in to change notification settings * Fork 2.7k * Star 18.3k This is a repo with links to everything you'd ever want to learn about data engineering 18.3k stars 2.7k forks Branches Tags Activity Star Notifications You must be signed in to change notification settings * Code * Issues 10 * Pull requests 10 * Actions * Projects 0 * Security * Insights Additional navigation options * Code * Issues * Pull requests * Actions * Projects * Security * Insights DataExpert-io/data-engineer-handbook main BranchesTags [ ] Go to file Code Folders and files Name Name Last commit message Last commit date Latest commit History 285 Commits bootcamp bootcamp .gitignore .gitignore README.md README.md books.md books.md communities.md communities.md interviews.md interviews.md newsletters.md newsletters.md projects.md projects.md View all files Repository files navigation * README The Data Engineering Handbook This repo has all the resources you need to become an amazing data engineer! Getting started If you are new to data engineering, start by following this 2024 breaking into data engineering roadmap If you are here for the 6-week free YouTube boot camp you can check out * introduction * software needed For more applied learning: * Check out the projects section for more hands-on examples! * Check out the interviews section for more advice on how to pass data engineering interviews! * Check out the books section for a list of high quality data engineering books * Check out the communities section for a list of high quality data engineering communities to join * Check out the newsletter section to learn via email Resources Great list of over 25 books Top 3 must read books are: * Fundamentals of Data Engineering * Designing Data-Intensive Applications * Designing Machine Learning Systems Great list of over 10 communities to join: Top must-join communities for DE: * EcZachly Data Engineering Discord * Data Talks Club Slack * Data Engineer Things Community Top must-join communities for ML: * AdalFlow Discord * Chip Huyen MLOps Discord Companies: * Orchestration + Mage + Astronomer + Prefect + Dagster + Airflow + Kestra + Shipyard + Hamilton * Data Lake / Cloud + Tabular + Microsoft + Databricks + Onehouse + Delta Lake * Data Warehouse + Snowflake + Firebolt * Data Quality + dbt + Gable + Great Expectations + Streamdal + Coalesce + Soda + DQOps + HEDDA.IO * Education Companies + DataExpert.io + LearnDataEngineering.com + AlgoExpert + ByteByteGo * Analytics / Visualization + Preset + Starburst + Metabase + Looker Studio + Tableau + Power BI + Apache Superset + Evidence * Data Integration + Cube + Fivetran + Airbyte + dlt + Sling + Meltano * Modern OLAP + Apache Druid + ClickHouse + Apache Pinot + Apache Kylin + DuckDB + QuestDB * LLM application library + AdalFlow + LangChain + LlamaIndex * Real-Time Data + Aggregations.io + Responsive + RisingWave + Striim Data Engineering blogs of companies: * Netflix * Uber * Databricks * Airbnb * Amazon AWS Blog * Microsoft Data Architecture Blogs * Microsoft Fabric Blog * Oracle * Meta * Onehouse Data Engineering Whitepapers: * A Five-Layered Business Intelligence Architecture * Lakehouse:A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics * Big Data Quality: A Data Quality Profiling Model * The Data Lakehouse: Data Warehousing and More * Spark: Cluster Computing with Working Sets * The Google File System * Building a Universal Data Lakehouse * XTable in Action: Seamless Interoperability in Data Lakes * MapReduce: Simplified Data Processing on Large Clusters Social Media Accounts Here's the mostly comprehensive list of data engineering creators: (You have to have at least 5k followers somewhere to be added!) Name [687474] [687474] [687474] [687474] [687474] YouTube LinkedIn X/Twitter Instagram TikTok Data with Zach Zach Wilson Zach (70k+) Wilson EcZachly (30k+) eczachly (150k+) @eczachly (70k+) (400k+) Shashank E-learning Shashank Mishra Bridge Mishra (100k+) (100k+) Seattle Seattle Ben SeattleDataGuy Data Guy Data Guy Rogojan (10k+) (100k+) (100k+) TrendyTech Sumit TrendyTech (100k+) Mittal (100k+) Darshil Darshil Darshil Parmar Parmar Parmar (100k+) (100k+) Andreas Andreas Andreas learndataengineering Kretz Kretz Kretz (5k+) (100k+) (100k+) ByteByteGo ByteByteGo Alex Xu alexxubyte (1m+) (100k+) (100k+) The Ravit The Ravit Show Show (100k+) Guy in a Guy in a Cube Cube (100k+) Adam Adam Marczak Marczak (100k+) nullQueries nullQueries (100k+) TECHTFQ by TECHTFQ by Thoufiq Thoufiq (100k+) SQLBI Marco SQLBI (100k+) Russo marcorus (10k+) (50k+) Azure Lib Deepak Azure Lib (10k+) Goyal (100k+) Advancing Advancing Simon Analytics Analytics Whiteley (10k+) (10k+) Kahan Data Kahan Data Solutions Solutions (10k+) Ankit Ankit Ankit Bansal Bansal Bansal (10k+) (50k+) Mr. K Talks Mr. K Talks Tech Tech (10k+) Li Yin Li Yin (10k+) Jaco van Jaco van Gelder Gelder (10k+) Joseph Joseph startdataeng Machado Machado (5k+) (10k+) Eric Roby Eric Roby (10k+) Simon Simon Spati Spati (10k+) Dipankar Dipankar Mazumdar Mazumdar (5k+) Daniel Daniel Ciocirlan Ciocirlan (5k+) Hugo Lu Hugo Lu (5k+) Tobias Tobias Macey Macey (5k+) Marcos Marcos Ortiz Ortiz (5k+) Julien Julien Hurault Hurault (5k+) Alex Alex The Alex @alex_the_analyst Freberg Analyst Freberg (10k+) (100k+) (100k+) Marc Marc Lamberti Lamberti (50k+) Chip Huyen Chip Huyen (250k+) Alex Merced Alex Alex Merced Data Merced @amdatalakehouse @alexmercedcoder (30k+) John Kutay John Kutay John Kutay @JohnKutay (5k+) Lakshmi Lakshmi Sontenam Sontenam (9.5k+) Hassaan Hassaan Akbar Akbar (5k+) Samuel Python Focht Basics (10k+) Constantin Constantin Lungu Lungu (10k+) Ijaz Ali Ijaz Ali (24K+) Subhankar Subhankar (5k+) Ankur Big Data Ankur Ranjan Show Ranjan (100k+) (48k+) Great Podcasts * The Data Engineering Show * Data Engineering Podcast * DataTopics * The Data Engineering Side Of Data * DataWare * The Data Coffee Break Podcast * The Datastack show * Intricity101 Data Sharks Podcast * Drill to Detail with Mark Rittman * Analytics Power Hour * Catalog & cocktails * Datatalks * Data Brew by Databricks * The Data Cloud Podcast by Snowflake * What's New in Data * Open||Source||Data by Datastax * Streaming Audio by confluent * The Data Scientist Show * MLOps.community * Monday Morning Data Chat * The Data Chief Great list of 20+ newsletters Top must follow newsletters for data engineering: * DataEngineer.io Newsletter * Joe Reis * Start Data Engineering * Data Engineering Weekly Glossaries: * Data Engineering Vault * Airbyte Data Glossary * Data Engineering Wiki by Reddit * Seconda Glossary * Glossary Databricks * Airtable Glossary * Data Engineering Glossary by Dagster Design Patterns * Cumulative Table Design * Microbatch Deduplication * The Little Book of Pipelines * Data Developer Platform Courses / Academies * DataExpert.io course use code HANDBOOK10 for a discount! * LearnDataEngineering.com * Technical Freelancer Academy Use code zwtech for a discount! * IBM Data Engineering for Everyone * Qwiklabs * DataCamp * Udemy Courses from Shruti Mantri * Rock the JVM teaches Spark (in Scala), Flink and others * Data Engineering Zoomcamp by DataTalksClub * Efficient Data Processing in Spark * Scaler * DataTeams - Data Engingeer hiring platform * Udemy Courses from Daniel Blanco Certifications Courses * Google Cloud Certified - Professional Data Engineer * Databricks - Certified Associate Developer for Apache Spark * Databricks - Data Engineer Associate * Databricks - Data Engineer Professional * Exam DP-203: Data Engineering on Microsoft Azure * Microsoft Fabric Analytics Engineer Associate * AWS Certified Data Engineer - Associate About This is a repo with links to everything you'd ever want to learn about data engineering Topics data awesome sql bigdata dataengineering apachespark Resources Readme Activity Custom properties Stars 18.3k stars Watchers 299 watching Forks 2.7k forks Report repository Releases No releases published Packages 0 No packages published Contributors 83 * @EcZachly * @liyin2015 * @Adesoji1 * @evil-in * @sspaeti * @damaradiprabowo * @kkuznets * @cnstlungu * @jsneedles * @Ben8t * @ijaz-lab * @subhankardata * @capkutay * @youcc + 69 contributors Languages * Makefile 52.1% * Shell 47.9% Footer (c) 2024 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact * Manage cookies * Do not share my personal information You can't perform that action at this time.