https://www.percona.com/blog/get-your-backup-to-half-of-its-size-introducing-zstd-support-in-percona-xtrabackup/ Percona Database Performance BlogPercona Database Performance Blog * Percona Live * About Us * Contact Us [ ] * Discover FEATURED [svg] Percona Platform A unified experience for developers and database administrators to monitor, manage, secure, and optimize database environments on any infrastructure. Learn More BY SERVICE BY PRODUCT + Support o MySQL Support o MongoDB Support o MariaDB Support o PostgreSQL Support o DBaaS Support o High Availability Support o Flexible Pricing o Support Tiers o Technical Account Managers + Managed Services o Percona Managed Database Services o Percona Advanced Managed Database Service + Consulting o Percona Cloud Cover o Percona Open Source Advance o Percona and Microsoft Azure Partnership + Policies + Training + Percona Monitoring and Management + MySQL Databsase Software o Percona Distribution for MySQL o Percona Server for MySQL o Percona XtraDB Cluster o Percona XtraBackup + MongoDB Database Software o Percona Distribution for MongoDB o Percona Server for MongoDB o Percona Backup for MongoDB + PostgreSQL Database Software + Open Source Database Tools o Percona Toolkit + Percona Kubernetes Operators + Software Downloads * Resources + 2020 Survey Results + Webinars + Solution Briefs + Datasheets + Case Studies + Technical Presentations + Videos + White Papers + Product Documentation * About + About Percona + Contact Us + Customers + Percona 15th Anniversary + In The News + Percona Live + Careers o Percona Lifestyle + Events * Blog * Community + Forums + Community Blog + PMM Community Contributions How To Get Your Backup to Half of Its Size - Introducing ZSTD Support in Percona XtraBackup Back to the Blog 17 Nov 2022 Marcelo Altmann2022-11-17T08:08:43-05:00 By Marcelo Altmann Insight for DBAs, MySQL, Percona Software MySQL, mysql-and-variants, Percona XtraBackup 2 Comments ZSTD Support in Percona XtraBackupHaving a backup of your database is like insurance, you have to pay a monthly price to ensure you have a service available when you need to. When talking about backups, the storage required to keep your backups is what comes into factor when talking about price, the bigger your backup, or the bigger the retention period, the more it will cost. Compressing your backups is a common practice to reduce this cost. Currently, Percona XtraBackup (PXB) has support for two compression algorithms: quicklz (which is an abandoned project and will soon be deprecated in PXB) and LZ4. Today we are glad to introduce support for a new compression algorithm in Percona XtraBackup 8.0.30 - Zstandard (ZSTD). Zstandard is a fast lossless compression algorithm, targeting real-time compression scenarios at zlib-level and better compression ratios. Usage A new value has been added to the xtrabackup -compress option. Passing -compress=zstd to PXB will make it use the new Zstandard algorithm to run compression. Compression can work in parallel by adjusting -parallel=X for parallel file copy and -compress-threads=x for parallel compression of those files. For decompression either on PXB or xbstream the usage remains the same. Users only need to pass -decompress and the tool will use the same algorithm used for compression. Please note that as in qpress and LZ4, you will require the ZSTD client to run the -decompress operation. Compress Shell [xtrabackup --backup ] 1 xtrabackup --backup --compress=zstd --parallel=16 --compress- threads=8 --target-dir=/backup Decompress Shell [xtrabackup --decompr] 1 xtrabackup --decompress --parallel=16 --target-dir=/backup Testing To test how ZSTD compares with LZ4 and uncompressed backups, we set up a test environment using an AWS EC2 instance c5.4xlarge (16 CPUs and 32G of RAM ) with data and backup going to the same disk partition, an EBS IO2 with 10K provisioned IOPS SSD. For each round of tests, we created a set of 12 tables with 40M rows resulting in a 109G dataset. We ran five rounds of tests creating a new database with the same amount of data on each round. Percona XtraBackup was invoked using -parallel=16 -compress-threads=8 . Size of resulting backup (full, LZ4, and ZSTD), time to run the backup (full, LZ4, and ZSTD), and time to decompress the resulting backup (LZ4 and ZSTD). The second round of tests explored how the two algorithms and uncompressed backups perform when uploading the data to S3. For this round, we ran two tests. The first one was to take the backup again and upload it to S3, time to complete was measured. The second part of the test was to download the backup from S3 and in the case of uncompressed backups, just store it on disk, and for compressed backup, we were first decompressing it and then storing it on disk. On all tests, ZSTD compression level of one was used. Results ZSTD Support in Percona XtraBackup time to decompress [svg] [svg] [svg] Summary As we can see from the test results above, ZSTD not only overcame LZ4 results on all tests, but it also brought backup size to half of its original size. When streaming is added to the mix is when we see the biggest difference between both algorithms, with ZSTD overcoming LZ4 with an even bigger margin. This can bring users and organizations a huge amount of savings in backup storage, either on-premises or especially in the cloud - where we are charged for each GB of storage we use. Learn more about Percona XtraBackup Related Author [svg] Marcelo Altmann Marcelo Altmann is a C++ Software Engineer working on MySQL related products. At Percona he has also worked as a Senior Support Engineer and a Tech Lead of the Support Team. Prior to joining Percona , he worked as a MySQL DBA at Ireland's CCTLD, and worked as a DBA/PHP developer in Brazil. He also blogs about other MySQL related stuff at his personal blog. --------------------------------------------------------------------- Share this post FacebookTwitterLinkedInEmail Subscribe Connect with Login Notify of [new follow-up comments ] [ ] [>] guest Label [ ] {} [+] [ ] [ ] Name* [ ] Email* [ ] Website [ ] [Post Comment] [ ] [ ] [ ] [ ] [ ] [ ] [ ] D[ ] Connect with guest Label [ ] {} [+] [ ] [ ] Name* [ ] Email* [ ] Website [ ] [Post Comment] [ ] [ ] [ ] [ ] [ ] [ ] [ ] D[ ] 2 Comments Oldest Newest Most Voted Inline Feedbacks View all comments Rob Wultsch Rob Wultsch 25 minutes ago The last few jobs I have been at we ditched compression in xtrabackup and used zstd pipelined. From reading the above it seems like xtrabackup is still writing the compressed file and then decompressing which is not good for ssd life or restore speed. Has thought been given to uncompressing directly (skipping writing the compressed files to the fs)? 0 Reply Marcelo Altmann Marcelo Altmann Author Reply to Rob Wultsch 10 seconds ago If I understand correctly, you are talking about streaming the backup to provision a new server rather than keeping your backup stored to restore in case a disaster happens on your database, right ? In that case, yes you can use streaming, something like: receiver side: nc -l 9999 | xbstream -x -decompress source side: xtrabackup -backup -compress=zstd ... | nc receiver_ip:9999 this will compress the backup over the WAN and will write it only once decompressed on receiver side. 0 Reply --------------------------------------------------------------------- How Can We Help? Maximize your application performance with our open source database support, managed services or consulting. Talk to an Expert Subscribe Want to get weekly updates listing the latest blog posts? Subscribe now and we'll send you an update every Friday at 1pm ET. Subscribe to our blog Categories * MySQL(3606) * Insight for DBAs(1858) * Percona Software(1811) * Percona Events(894) * MongoDB(655) * Insight for Developers(561) * Cloud(387) * Benchmarks(355) * Percona Live(350) * Webinars(311) * PostgreSQL(268) * Monitoring(225) * Percona Services(201) * MariaDB(161) * Security(154) * ProxySQL(136) * Hardware and Storage(111) * Database Trends(78) * Storage Engine(69) * Percona Announcements(27) * Percona Platform(2) Percona Blog RSS Feed Services * Support * Managed Services * Consulting * Training Products * MySQL Software * MongoDB Software * PostgreSQL Distribution * Percona Operators * Monitoring & Management Resources * Solution Briefs * White Papers * Webinars * Case Studies * Datasheets * Documentation More * Blog * Community Blog * Technical Forum Help About * Customers * Newsroom * About * Careers Contact Us * Sales & General Inquiries * (888) 316-9775 (USA) * (208) 473-2904 (USA) * +44 203 608 6727 (UK) * 0-808-169-6490 (UK) MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners. Proudly running Percona Server for MySQL * * * * * * Terms of Use | Privacy | Copyright | Legal | Security Center Copyright (c) 2006-2022 Percona LLC. wpDiscuz [ ] Insert *