https://github.blog/2022-11-02-github-availability-report-october-2022/ / Blog * Engineering * Product * Security * Open Source * Enterprise * Changelog * Community * Education * Company * Policy Free trial Contact sales Search by Keyword [ ] Search * Engineering * Enterprise GitHub Availability Report: October 2022 In October, we experienced four incidents that resulted in degraded performance across GitHub services. This report also sheds light into an incident that impacted Codespaces in September. GitHub Availability Report: October 2022 Author Jakub OleksyJakub Oleksy November 2, 2022 * * * In October, we experienced four incidents that resulted in significant impact and degraded state of availability to multiple GitHub services. This report also sheds light into an incident that impacted Codespaces in September. October 26 00:47 UTC (lasting 3 hours and 47 minutes) Our alerting systems detected an incident that impacted most Codespaces customers. Due to the recency of this incident, we are still investigating the contributing factors and will provide a more detailed update on cause and remediation in the November Availability Report, which we will publish the first Wednesday of December. October 13 20:43 UTC (lasting 48 minutes) On October 13, 2022 at 20:43 UTC, our alerting systems detected an increase in the Projects API error response rate. Due to the significant customer impact, we went to status red for Issues at 20:47 UTC. Within 10 minutes of the alert, we traced the cause to a recently-deployed change. This change introduced a database validation that required a certain value to be present. However, it did not correctly set a default value in every scenario. This resulted in the population of null values in some cases, which produced an error when pulling certain records from the database. We initiated a roll back of the change at 21:08 UTC. At 21:13 UTC, we began to see a steady decrease in the number of error responses from our APIs back to normal levels, changed the status of Issues to yellow at 21:24 UTC, and changed the status of Issues to green at 21:31 UTC once all metrics were healthy. Following this incident, we have added mitigations to protect against missing values in the future, and we have improved testing around this particular area. We have also fixed our deployment dashboards, which contained some inaccurate data for pre-production errors. This will ensure that errors are more visible during the deploy process to help us prevent these issues from reaching production. October 12 23:27 UTC (lasting 3 hours and 31 minutes) On October 12, 2022 at 22:30 UTC, we rolled out a global configuration change for Codespaces. At 23:15 UTC, after the change had propagated to a variety of regions, we noticed new Codespace creation starting to trend downward and were alerted to issues from our monitors. At 23:27 UTC, we deemed the impact significant enough to status Codespaces yellow, and eventually red, based on continued degradation. During the incident, it was discovered that one of the older components of the backend system did not cope well with the configuration change, causing a schema conflict. This was not properly tested prior to the rollout. Additionally, this component version does not support gradual exposure across regions--so many regions were impacted at once. Once we detected the issue and determined the configuration change was the cause, we worked to carefully roll back the large schema change. Due to the complexity of the rollback, the procedure took an extended period of time. Once the rollback was complete and metrics tracking new Codespaces creations were healthy, we changed the status of the service back to green at 02:58 UTC. After analyzing this incident, we determined we can eliminate our dependency on this older configuration type and have repair work in progress to eliminate this type of configuration from Codespaces entirely. We have also verified that all future changes to any component will follow safe deployment practices (one test region followed by individual region rollouts) to avoid global impact in the future. October 5 06:30 UTC (lasting 31 minutes) On October 5, 2022 at 06:30 UTC, webhooks experienced a significant backlog of events caused by a high volume of automated user activity causing a rapid series of create and delete operations. This activity triggered a large influx of webhook events. However, many of these events caused exceptions in our webhook delivery worker because data needed to generate their webhook payloads had been deleted from the database. Attempting to retry these failed jobs tied up our worker and it was unable to process new incoming events, resulting in a severe backlog in our queues. Downstream services that rely on webhooks to receive their events were unable to receive them, which resulted in service degradation. We updated GitHub Actions to status red because the webhooks delay caused new job execution to be severely delayed. Investigation into the source of the automated activity led us to find that there was automation creating and deleting many repositories in quick succession. As a mitigation, we disabled the automated accounts that were causing this activity in order to give us time to find a longer term solution for such activity. Once the automated accounts were disabled, it brought the webhook deliveries back to normal and the backlog got mitigated at 07:01 UTC. We also updated our webhook delivery workers to not retry any jobs for which it was determined that the data did not exist in the database. Once the fix was put in place, the accounts were re-enabled and no further problems were encountered with our worker. We recognize that our services must be resilient to spikes in load and will make improvements based on what we've learned in this incident. September 28 03:53 UTC (lasting 1 hour and 16 minutes) On September 27, 2022 at 23:14 UTC, we performed a routine secret rotation procedure on Codespaces. On September 28, 2022 at 03:21 UTC, we received an internal report stating that port forwarding was not working on the Codespaces web client and began investigating. At 03:53 UTC. we statused yellow due to the broad user impact we were observing. Upon investigation, we found that we missed a step in the secret rotation checklist a few hours earlier, which caused some downstream components to fail to pick up the new secret. This resulted in some traffic not reaching backend services as expected. At 04:29 UTC, we ran the missed rotation step, after which we quickly saw the port forwarding feature returning to a healthy state. At this moment, we considered the incident to be mitigated. We investigated why we did not receive automated alerts about this issue and found that our alerts were monitoring error rates but did not alert for lack of overall traffic to the port forwarding backend. We have since improved our monitoring to include anomalies in traffic levels that cover this failure mode. Several hours later, at 17:18 UTC, our monitors alerted us of an issue in a separate downstream component, which was similarly caused by the previous missed secret rotation step. We could see Codespaces creation and start failures increasing in all regions. The effect from the earlier secret rotation was not immediate because this secret is used in exchange for a token, which is cached for up to 24 hours. Our understanding was that the system would pick up the new secret without intervention, but in reality this secret was picked up only if the process was restarted. At 18:27 UTC, we restarted the service in all regions and could see that the VM pools, which were heavily drained before, started to slowly recover. To accelerate the draining of the backlog of queued jobs we increased the pool size at 18:45 UTC. This helped all but two pools in West Europe, which were still not recovering. At 19:44 UTC, we identified an instance of the service in West Europe that was not rotated along the rest. We rotated that instance and quickly saw a recovery in the affected pools. After the incident, we identified why multiple downstream components failed to pick up the rotated secret. We then added additional monitoring to identify which secret versions are in use across all components in the service to more easily track and verify secret rotations. To address this short term, we have updated our secret rotation checklist to include the missing steps and added additional verification steps to ensure the new secrets are picked up everywhere. Longer term, we are automating most of our secret rotation processes to avoid human error. In summary Please follow our status page for real-time updates on status changes. To learn more about what we're working on, check out the GitHub Engineering Blog. Tags: * GitHub Availability Report The GitHub Insider Newsletter Get the best of GitHub. Once a month. Directly to your inbox. Subscribe More on GitHub Availability Report GitHub Availability Report: September 2022 GitHub Availability Report: September 2022 In September, we experienced one incident that resulted in degraded performance across GitHub services. We also experienced one incident resulting in significant impact to Codespaces. We are still investigating that incident and will include it in next month's report. This report also sheds light into an incident that impacted Codespaces in August and an incident that impacted Actions in August. Jakub Oleksy GitHub Availability Report: August 2022 In August, we experienced one incident resulting in significant impact to Codespaces. We're still investigating that incident and will include it in next month's report. This report also sheds light into an incident that impacted Codespaces in July. Jakub Oleksy GitHub Availability Report: July 2022 In July, we experienced one incident that resulted in degraded performance for Codespaces. This report also acknowledges two incidents that impacted multiple GitHub.com services in June. Jakub Oleksy Interested in bringing GitHub Enterprise to your organization? Start your free trial for 30 days and increase your team's collaboration. $21 per user/month after trial expires. Curious about other plans? Related posts Choose your own GitHub Universe 2022 adventure Community Choose your own GitHub Universe 2022 adventure You can now build your agenda on GitHubUniverse.com! Whether you're just getting started or you're a seasoned industry professional, there's a session for you. Laura Lindeman 3 strategies for consolidating your toolkit and boosting productivity Enterprise 3 strategies for consolidating your toolkit and boosting productivity Explore how GitHub Enterprise can help you transform your software engineering organization and practices. Chris Reddington & Mark Paulsen Why and how GitHub encrypts sensitive database columns using ActiveRecord::Encryption Engineering Why and how GitHub encrypts sensitive database columns using ActiveRecord::Encryption You may know that GitHub encrypts your source code at rest, but you may not have known that we encrypt sensitive database columns as well. Read about our column encryption strategy and our decision to adopt the Rails column encryption standard. Kylie Stradley Explore more from GitHub Engineering Engineering Posts straight from the GitHub engineering team. Learn more GitHub Universe '22 Dev GitHub Universe '22 The global developer event for cloud, security, community, and AI Register now GitHub Actions GitHub Actions Native CI/CD alongside code hosted in GitHub. Learn more Work at GitHub! Work at GitHub! Check out our current job openings. Learn more Subscribe to The GitHub Insider A newsletter for developers covering techniques, technical guides, and the latest product innovations coming from GitHub. [ ] Subscribe Product * Features * Security * Enterprise * Customer Stories * Pricing * Resources Platform * Developer API * Partners * Atom * Electron * GitHub Desktop Support * Docs * Community Forum * Training * Status * Contact Company * About * Blog * Careers * Press * Shop * GitHub on Twitter * GitHub on Facebook * GitHub on YouTube * GitHub on Twitch * GitHub on TikTok * GitHub on LinkedIn * GitHub's organization on GitHub * (c) 2022 GitHub, Inc. * Terms * Privacy