https://status.cloud.google.com/incidents/WMmjrixdPfBGFKCohYGd#Z6wKuSfovJCG51wpbwiB [google-clo] Console * Google Cloud Service Health * Incidents * Global: Google Kubernetes Engine Nodepool Upgrade Failures Service Health This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/. * Available * Service information * Service disruption * Service outage Incident affecting Google Kubernetes Engine Global: Google Kubernetes Engine Nodepool Upgrade Failures Incident began at 2023-10-02 11:29 (all times are US/Pacific). Currently affected location(s) Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Delhi (asia-south2)Singapore (asia-southeast1)Jakarta (asia-southeast2) Sydney (australia-southeast1)Melbourne (australia-southeast2)Warsaw (europe-central2)Finland (europe-north1)Madrid (europe-southwest1) Belgium (europe-west1)Berlin (europe-west10)Turin (europe-west12) London (europe-west2)Frankfurt (europe-west3)Netherlands (europe-west4)Zurich (europe-west6)Milan (europe-west8)Paris (europe-west9)Doha (me-central1)Dammam (me-central2)Tel Aviv (me-west1)Montreal (northamerica-northeast1)Toronto (northamerica-northeast2)Sao Paulo (southamerica-east1)Santiago (southamerica-west1)Iowa (us-central1)South Carolina (us-east1) Northern Virginia (us-east4)Columbus (us-east5)Dallas (us-south1) Oregon (us-west1)Los Angeles (us-west2)Salt Lake City (us-west3)Las Vegas (us-west4) Date Time Description Summary: Global: Google Kubernetes Engine Nodepool Upgrade Failures Description: We believe the issue with Google Kubernetes Engine is partially resolved. We do not have an ETA for full resolution at this point. 10 13:27 We will provide an update by Wednesday, 2023-10-11 12:00 Oct PDT US/Pacific with current details. 2023 Diagnosis: A small number of customers are experiencing failed nodepool upgrades. Customers experiencing this, may see "Internal error" in Google Cloud Console. Retrying is suggested but may not always work. Workaround: Customers can re-create nodepools at the new version instead of upgrading in-place. Summary: Global: Google Kubernetes Engine Nodepool Upgrade Failures Description: A mitigation has been rolling out and we are assessing its effectiveness. We will provide an update by Tuesday, 2023-10-10 14:00 US/Pacific with current 10 details. Oct 11:51 2023 PDT Diagnosis: A small number of customers are experiencing failed nodepool upgrades. Customers experiencing this, may see "Internal error" in Google Cloud Console. Retrying is suggested but may not always work. Workaround: Customers can re-create nodepools at the new version instead of upgrading in-place. Summary: Global: Google Kubernetes Engine Nodepool Upgrade Failures Description: A mitigation has been rolling out and we are assessing its effectiveness. We will provide an update by Tuesday, 2023-10-10 12:00 US/Pacific with current 9 details. Oct 12:02 2023 PDT Diagnosis: A small number of customers are experiencing failed nodepool upgrades. Customers experiencing this, may see "Internal error" in Google Cloud Console. Retrying is suggested but may not always work. Workaround: Customers can re-create nodepools at the new version instead of upgrading in-place. Summary: Global: Google Kubernetes Engine Nodepool Upgrade Failures Description: A mitigation has been rolling out and we are assessing its effectiveness. We will provide an update by Monday, 2023-10-09 16:00 US/Pacific with current details. 9 10:57 Oct PDT Diagnosis: A small number of customers are experiencing 2023 failed nodepool upgrades. Customers experiencing this, may see "Internal error" in Google Cloud Console. Retrying is suggested but may not always work. Workaround: Customers can re-create nodepools at the new version instead of upgrading in-place. Summary: Global: Google Kubernetes Engine Nodepool Upgrade Failures Description: A mitigation has been rolling out and we are assessing its effectiveness. We will provide an update by Monday, 2023-10-09 12:00 US/Pacific with current details. 6 10:28 Oct PDT Diagnosis: A small number of customers are experiencing 2023 failed nodepool upgrades. Customers experiencing this, may see "Internal error" in Google Cloud Console. Retrying is suggested but may not always work. Workaround: Customers can re-create nodepools at the new version instead of upgrading in-place. Summary: Global: Google Kubernetes Engine Nodepool Upgrade Failures Description: Our engineering team is working on a mitigation which will rollout over the next few days. We will provide an update by Friday, 2023-10-06 12:00 US/ 3 Pacific with current details. Oct 11:07 2023 PDT Diagnosis: A small number of customers are experiencing failed nodepool upgrades. Customers experiencing this, may see "Internal error" in Google Cloud Console. Retrying is suggested but may not always work. Workaround: Customers can re-create nodepools at the new version instead of upgrading in-place. Summary: We are experiencing an issue with Google Kubernetes Engine Description: Our engineering team continues to investigate mitigation pathways. We will provide an update by Tuesday, 2023-10-03 12:00 US/Pacific with 2 current details. Oct 14:57 2023 PDT Diagnosis: A small number of customers are experiencing failed nodepool upgrades. Customers experiencing this, may see "Internal error" in Google Cloud Console. Retrying is suggested however may not work. Workaround: Customers can re-create nodepool at the new version instead of upgrading in-place. Summary: We are experiencing an issue with Google Kubernetes Engine Description: Our engineering team continues to investigate mitigation pathways. We will provide an update by Tuesday, 2023-10-03 12:00 US/Pacific with 2 current details. Oct 14:40 2023 PDT Diagnosis: A small number of customers are experiencing failed nodepool upgrades. Customers experiencing this will see "Internal error" in Pantheon. In this scenario, retrying may not work. Workaround: Customers can re-create nodepool at the new version instead of upgrading in-place. Summary: We are experiencing an issue with Google Kubernetes Engine Description: We are experiencing an issue with Google Kubernetes Engine. Our engineering team continues to investigate the issue. 2 14:30 Oct PDT We will provide an update by Monday, 2023-10-02 14:45 US/ 2023 Pacific with current details. Diagnosis: Customers with nodepools at version 1.24 cannot upgrade them to the next minor version, 1.25. Workaround: Customers can re-create nodepool at the new version instead of upgrading in-place. Summary: We are experiencing an issue with Google Kubernetes Engine Description: We are experiencing an issue with Google Kubernetes Engine. Our engineering team continues to investigate the issue. 2 12:14 Oct PDT We will provide an update by Monday, 2023-10-02 14:30 US/ 2023 Pacific with current details. Diagnosis: Customers with nodepools at version 1.24 cannot upgrade them to the next minor version, 1.25. Workaround: Customers can re-create nodepool at the new version instead of upgrading in-place. Summary: We are experiencing an issue with Google Kubernetes Engine Description: We are experiencing an issue with Google Kubernetes Engine. Our engineering team continues to investigate the issue. 2 12:11 Oct PDT We will provide an update by Monday, 2023-10-02 12:47 US/ 2023 Pacific with current details. Diagnosis: Customers with nodepools at version 1.24 cannot upgrade them to the next minor version, 1.25. Workaround: Customers can re-create nodepool at the new version instead of upgrading in-place. Summary: We are experiencing an issue with Google Kubernetes Engine Description: We are experiencing an issue with Google Kubernetes Engine. Our engineering team continues to investigate the issue. 2 11:47 Oct PDT We will provide an update by Monday, 2023-10-02 12:47 US/ 2023 Pacific with current details. Diagnosis: Customers with nodepools at version 1.24.14-gke.2700 cannot upgrade them to the next minor version, 1.25.12-gke.500. Workaround: Customers can re-create nodepool at the new version instead of upgrading in-place. * All times are US/Pacific * Documentation * RSS Feed * JSON History | Schema * JSON Product Catalog | Schema * Send feedback * Privacy policy