https://discourse.nixos.org/t/s3-update-and-recap-of-community-call/28942 NixOS Discourse S3 Update and Recap of Community Call Announcements ron June 9, 2023, 3:57pm #1 We wanted to share the latest updates and highlights of the recent S3 Community Call. We discussed our next steps as LogicBlox, the longstanding sponsor of https://cache.nixos.org, is ending its sponsorship. More context can be found here - The NixOS Foundation's Call to Action: S3 Costs Require Community Support - Announcements - NixOS Discourse Small note - We are still working on a few of the short term options to receive more information and will share immediately. Meeting Overview Our agenda covered an introduction to the issue, a review of our financial situation, potential options, and an open discussion. Financials We have over 200k EUR in the Foundation bank account, some of which have prior obligations. We can sustain the current costs for at least one year, but the termination of the LogicBlox sponsorship represents a significant cost increase as discussed prior. Options Discussed A host of solutions were proposed, from Foundation covering the short-term costs while developing a long-term solution to reaching out to AWS to subsidize costs. Other options included Cloudflare R2 with a potential promise of sponsorship, exploring trusted vendors like Wasabi, Telnix, Storj, and considering self-hosting. We had guests from Storj and Cloudflyer present at the meeting who shared insights about their platforms, emphasizing their competitive cost structures, migration support, and potential for replacing Fastly. Key Discussions We discussed various topics from data storage, deduplication to the longer term options. The overall model we are placing is to split between the short term and long term. Short term will prioritize the matters we need to handle to have a solid resolution in place for at least the next 12 months. This will allow the Infra team and the general community to explore a longer term solution with optimizations and improvements which are out of scope for the timeframe we currently have. Key Priorities for the Short Term 1. 0 Risk - Provide and move forward with a resolution that proposes the least amount of change/risk. As an example, our preference would be to stay on AWS in the short term if feasible. 2. Costs - While the foundation made measures to always be in a financial state to support events such as this, we prefer to find a resolution that is cost-effective, so those funds can be used for more community related matters. Should it come to it, we have made sure to have the reserves and can support the current state for almost a full year. 3. Longevity/scalability - We'd prefer a resolution that can be potentially more aligned with the next 5 to 10 years. Proper SLA/ Guarantees so that we have time to migrate properly when we need to. With that in mind, we are still optimizing in the short term for the two priorities above. Short-term Solutions Discussed and General Update 1. Storj - During the call: Storj has offered an 80% discount for pricing until we double our usage and described their system as being similar to Tahoe-LAFS 2. Cloudflare R2 - The foundation had an initial call with the Cloudflare team on June 7th. We provided an overarching review of Nix, community, the current situation and dove into the opportunities to collaborate based on the Cloudflare OSS Program. 1. Next Steps: Cloudflare team is taking the discussion internal and will revert in the coming days. 3. AWS S3 - We have been in contact with AWS throughout the week and awaiting a response to meet with the OSS team. We expect further information in the coming days with the primary goal of exploring sponsorship. 4. Migrate billing to foundation You can view the full discussion/options in the call notes, discourse thread and GitHub issues in the links below. Long-term Solutions For long-term solutions, discussions ranged from self-hosting with community hardware, adopting more efficient data structures, to implementing deduplication methods. We believe that once the immediate problem is solved, we can provide a runway of approximately 1 year for the infra team to work on these longer-term solutions. Thanks to all the participants and contributors. The efforts and care from so many folks is nothing short of incredible. Please comment/add any further topics and items! Links: * GitHub - Short Term List/Issue Discussion * GitHub - Long Term List/Issue Discussion * Discussion on Discourse * Community Call Notes 12 Likes The NixOS Foundation's Call to Action: S3 Costs Require Community Support iFreilicht June 9, 2023, 7:27pm #2 Was Backblaze B2 one of the options you considered or did you reach out to them. They were shortly mentioned in the previous thread as an option due to 0 egress cost from S3 and competitive pricing: The NixOS Foundation's Call to Action: S3 Costs Require Community Support - #77 by hexa nektro June 9, 2023, 7:31pm #3 what about the option of deleting unreachable paths that would free up between 79-94% of the space, potentially in conjunction with the other solutions? # NixOS Foundation's Financial Summary: A Transparent Look into 2022 NixOS Foundation Some info on our AWS costs (primarily cache.nixos.org and releases.nixos.org): The S3 bucket for cache.nixos.org stores 667M objects, with 107 TiB in standard storage and 318 TiB in infrequent access storage, so 425 TiB in total. The S3 bucket for nix-releases (i.e. https://releases.nixos.org) is 25.1 TiB (mostly ISO images). The S3 storage costs for our sponsor are about $5500 per month, after a $2300 enterprise discount. The S3 transfer costs are about $900 per month, which is most... 2 Likes endgame June 9, 2023, 10:05pm #4 # iFreilicht: Was Backblaze B2 one of the options you considered Domen had a bad time using B2 for Cachix and had to migrate off it. So I wouldn't be surprised if people are nervous about it. # nektro: what about the option of deleting unreachable paths that would free up between 79-94% of the space AIUI (I don't have any decision-making capacity here), deletion is on the table but needs to be handled carefully, because this bucket might have the last copy of sources that no longer exist. I know this has already happened because I found some instances when testing fonts in nixpkgs for reproducibility. 1 Like * Home * Categories * FAQ/Guidelines * Terms of Service * Privacy Policy Powered by Discourse, best viewed with JavaScript enabled Hosted by Flying Circus.