https://status.cloud.google.com/incidents/LnvJwfYu3TCyUrcrP7yf#RP1d9aZLNFZEJmTBk8e1 [google-clo] * Google Cloud Service Health * Incidents * Multiple services are being impacted globally within Google Cloud Service Health This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/. Incident affecting Google BigQuery, Apigee, Cloud Developer Tools, Cloud Firestore, Google Compute Engine, Google Cloud Bigtable, Google Cloud Storage, Google Cloud Networking, Identity and Access Management, Google Cloud Pub/Sub, Cloud Build, Google Cloud Functions, Container Registry Multiple services are being impacted globally within Google Cloud Incident began at 2023-02-27 04:58 and ended at 2023-02-27 05:11 (all times are US/Pacific). Previously affected location(s) Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Delhi (asia-south2)Singapore (asia-southeast1)Jakarta (asia-southeast2) Sydney (australia-southeast1)Melbourne (australia-southeast2)Warsaw (europe-central2)Finland (europe-north1)Madrid (europe-southwest1) Belgium (europe-west1)London (europe-west2)Frankfurt (europe-west3) Netherlands (europe-west4)Zurich (europe-west6)Milan (europe-west8) Paris (europe-west9)GlobalTel Aviv (me-west1)Montreal (northamerica-northeast1)Toronto (northamerica-northeast2)Sao Paulo (southamerica-east1)Santiago (southamerica-west1)Iowa (us-central1) South Carolina (us-east1)Northern Virginia (us-east4)Columbus (us-east5)Dallas (us-south1)Oregon (us-west1)Los Angeles (us-west2) Salt Lake City (us-west3)Las Vegas (us-west4) Date Time Description Mini Incident Report We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/ support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213. (All Times US/Pacific) Incident Start: 27 Feb 2023 04:58 Incident End: 27 Feb 2023 05:11 Duration: 13 minutes Affected Services and Features: Multiple Google Cloud Platform and Google Workspace services Regions/Zones: Multiple regions Description: Multiple Google Cloud Platform and Google Workspace services experienced elevated error rates for 13 minutes due to packet loss on Google's production backbone network. Google will be completing a full Incident Report in the following days that will include additional details and provide a full root cause. Customer Impact: Google Cloud Platform Services: * Apigee - Affected users experienced client connect timeouts and elevated 5XX errors. * Virtual Private Cloud (VPC) - Affected users experienced increased packet loss for cross region traffic. * Cloud Interconnect - Affected users in regions us-east4, asia-southeast1, southamerica-east1, us-east7, us-west4 experienced increased packet loss on their interconnects. * BigQuery - Affected users experienced availability issues with BigQuery. * Cloud Dataflow - Affected users experienced elevated latency and error rates for batch and streaming jobs. Some users may have experienced increased latency in Create Workflow operations. * Cloud Bigtable - Affected users experienced deadline exceeded or unavailable errors with Data and Admin APIs in us-east4, us-central1, and us-west1 regions * Cloud Key Management Service (KMS) - Affected users experienced "unavailable" errors when attempting to use the service. * Cloud Monitoring - Affected users experienced elevated latency on Cloud Monitoring dashboard when accessing resources. * Persistent Disk - Affected users experienced elevated read/write latency. * Cloud SQL - Customers may have seen intermittent connectivity issues for ~10 minutes from 05:00-05:10 (us-central1, us-east4, us-west4, southamerica-east1). Retrying should have succeeded, 27 10:54 as the success rate was ~80-95% depending on region Feb PST and method. 2023 * Cloud Workflows - Affected users may have seen increased error rates. * Cloud Console - Affected users may have seen increased GUI failures for the impact duration. * Google Compute Engine - Affected users experienced elevated 500 error rates when sending HTTP requests to Google Compute Engine APIs. * Cloud Load Balancing - Affected users experienced elevated 500 error rates from load balancers for traffic passing through asia-southeast1, us-central1, us-west4. * Cloud Run - Affected users experienced elevated error rates (400 or 500 errors), request time outs, and control plane request failures. Retrying the requests may have succeeded in some cases. * Cloud App Engine - Affected users experienced elevated error rates (400 or 500 errors), request time outs, and control plane request failures. Retrying the requests may have succeeded in some cases. * Cloud Functions - Affected users experienced elevated error rates (400 or 500 errors), request time outs, and control plane request failures. Retrying the requests may have succeeded in some cases. * Cloud VPN - Affected users experienced elevated packet loss in us-east4, asia-southeast1, southamerica-east1, us-east7, us-west4. * Identity and Access Management - Affected users experienced NOT_FOUND, PERMISSION_DENIED, DEADLINE_EXCEEDED and UNAVAILABLE errors. * Cloud Pub/sub - Affected users experienced availability issues. * Google Cloud Storage- Affected users experienced "unavailable" errors * Cloud Firestore-- Affected users experienced issues with Firestore service availability. * Cloud Data Loss Prevention - Affected users experienced elevated RPC error rates. * Cloud Memorystore - Affected users experienced increased latency and deadline exceeded errors for some requests like GetInstance, ListInstance, ExportInstance. Some users experienced issues with creating new Memorystore for Redis instances. Workspace services: * Gmail - Affected users experienced elevated error rates * Google Calendar - Affected users experienced availability issues with the calendar. * Google Chat - Affected users experienced issues with chat interactions on web and mobile * Google Meet - Affected users experienced failures to join a meeting. Retry attempt to join the meeting succeeded for most of the affected users. * Google Docs - Affected users experienced errors when loading or accessing a document. * Google Drive - Affected users experienced elevated errors. * Google Tasks - Affected users experienced availability issues with tasks. * Google Voice - Affected users experienced elevated error rate when interacting with Voice API. In most of the cases for Workspace services, retrying or refreshing the request had a chance of succeeding. A subset of users experienced some persistent (more than ~3-10 servers errors in a 10 minute timespan) unavailability. The issue with Apigee, Cloud Build, Cloud Firestore, Container Registry, Google BigQuery, Google Cloud Bigtable, Google Cloud Networking, Google Cloud Pub/Sub, 27 Google Cloud Storage, Google Compute Engine, Identity and Feb 08:04 Access Management has been resolved for all affected 2023 PST users as of Monday, 2023-02-27 05:11 US/Pacific. We thank you for your patience while we worked on resolving the issue. The issue with Apigee, Cloud Build, Cloud Firestore, Container Registry, Google BigQuery, Google Cloud Bigtable, Google Cloud Networking, Google Cloud Pub/Sub, 27 Google Cloud Storage, Google Compute Engine, Identity and Feb 06:26 Access Management has been resolved for all affected 2023 PST users as of Monday, 2023-02-27 05:11 US/Pacific. We thank you for your patience while we worked on resolving the issue. Summary: Multiple services are being impacted globally within Google Cloud Description: We are experiencing an issue with Cloud Build, Cloud Firestore, Google BigQuery, Google Cloud Networking, Google Cloud Pub/Sub, Google Compute Engine. 27 Our engineering team continues to investigate the issue Feb 06:02 while some services are starting to be restored. 2023 PST We will provide an update by Monday, 2023-02-27 06:40 US/ Pacific with current details. Diagnosis: Multiple services can be affected globally due to an ongoing outage Workaround: None at this time Summary: Multiple services are being impacted globally within Google Cloud Description: We are experiencing an issue with Cloud Build, Cloud Firestore, Google BigQuery, Google Cloud Networking, Google Cloud Pub/Sub, Google Compute Engine. 27 Our engineering team continues to investigate the issue Feb 05:58 while some services are starting to be restored. 2023 PST We will provide an update by Monday, 2023-02-27 06:30 US/ Pacific with current details. Diagnosis: Multiple services can be affected globally due to an ongoing outage Workaround: None at this time Summary: Multiple services are being impacted globally within Google Cloud Description: We are experiencing an issue with Cloud Build, Cloud Firestore, Google BigQuery, Google Cloud Networking, Google Cloud Pub/Sub, Google Compute Engine. 27 05:55 Our engineering team continues to investigate the issue. Feb PST We will provide an update by Monday, 2023-02-27 06:30 US/ 2023 Pacific with current details. Diagnosis: Multiple services can be affected globally due to an ongoing outage Workaround: None at this time Summary: Multiple services are being impacted globally within Google Cloud Description: We are experiencing an issue with Cloud Build, Cloud Firestore, Google BigQuery, Google Cloud Networking, Google Cloud Pub/Sub, Google Compute Engine. 27 Feb 05:41 Our engineering team continues to investigate the issue. 2023 PST We will provide an update by Monday, 2023-02-27 06:10 US/ Pacific with current details. Diagnosis: Multiple services can be affected globally due to an ongoing outage Workaround: None at this time * All times are US/Pacific * Documentation * RSS Feed * JSON History | Schema * Send feedback * Privacy policy * Available * Service information * Service disruption * Service outage