https://status.cloud.google.com/incidents/YrjzRWPFBUZU5HJZ4mN7#qiycz4eo8qffHEFsX7Kp [google-clo] * Google Cloud Service Health * Incidents * Google Cloud Networking experienced increased packet loss for egress traffic from Google to the Middle East, and elevated latency between our Europe and Asia Regions. Service Health This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/. Incident affecting Hybrid Connectivity, Virtual Private Cloud (VPC), Google Cloud Networking, Cloud NAT Google Cloud Networking experienced increased packet loss for egress traffic from Google to the Middle East, and elevated latency between our Europe and Asia Regions. Incident began at 2022-06-07 05:50 and ended at 2022-06-07 09:02 (all times are US/Pacific). Previously affected location(s) Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Delhi (asia-south2)Singapore (asia-southeast1)Jakarta (asia-southeast2) Sydney (australia-southeast1)Melbourne (australia-southeast2)Warsaw (europe-central2)Finland (europe-north1)Belgium (europe-west1)London (europe-west2)Frankfurt (europe-west3)Netherlands (europe-west4) Zurich (europe-west6)GlobalMontreal (northamerica-northeast1)Toronto (northamerica-northeast2)Sao Paulo (southamerica-east1)Santiago (southamerica-west1)Iowa (us-central1)South Carolina (us-east1) Northern Virginia (us-east4)Oregon (us-west1)Los Angeles (us-west2) Salt Lake City (us-west3)Las Vegas (us-west4) Date Time Description Mini Incident Report We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https:// cloud.google.com/support or help article https:// support.google.com/a/answer/1047213. We will provide a full Incident Report with additional details. (All Times US/Pacific) Incident Start: 07 June 2022 05:50 Incident End: 07 June 2022 09:02 Duration: 3 hours, 12 minutes Affected Services and Features: Google Cloud Networking Cloud NAT Hybrid Connectivity Virtual Private Cloud (VPC) Cloud Interconnect Google 7 Kubernetes Engine (GKE) Jun 13:09 2022 PDT Regions/Zones: Packet Loss (Global to Middle East Internet Users), Latency (Between Asia and Europe regions) Description: Google Cloud Networking experienced increased packet loss for egress traffic from Google to the Middle East, and elevated latency between our Europe and Asia Regions as a result, for 3 hours and 12 minutes, affecting several related products including Cloud NAT, Hybrid Connectivity and Virtual Private Cloud (VPC). From preliminary analysis, the root cause of the issue was a capacity shortage following two simultaneous fiber-cuts. Customer Impact: Affected customers would have experienced increased packet loss and increased latency for the affected metros which would have caused delays and timeouts for the following services : * Cloud Load Balancing * Cloud Interconnect * Virtual Private Cloud (VPC) * Cloud NAT * Google Kubernetes Engine (GKE) The issue with Cloud NAT, Google Cloud Networking, Hybrid Connectivity, Virtual Private Cloud (VPC) has been resolved for all affected users as of Tuesday, 2022-06-07 09:02 US/Pacific. 7 09:12 Jun PDT We will publish an analysis of this incident once we have 2022 completed our internal investigation. We thank you for your patience while we worked on resolving the issue. Summary: Packet loss observed from Internet in Middle East to Google Description: We believe the issue with Cloud NAT, Google Cloud Networking, Hybrid Connectivity, Virtual Private Cloud (VPC) is partially resolved. Packet loss should be mitigated, but some traffic continues to have elevated latency. 7 Jun 08:51 Full resolution is expected to complete by Tuesday, 2022 PDT 2022-06-07 09:30 US/Pacific. We will provide an update by Tuesday, 2022-06-07 09:30 US /Pacific with current details. Diagnosis: Increased packet loss for traffic traversing the affected metros. Workaround: None at this time. Summary: Packet loss observed from Internet in Middle East to Google Description: Mitigation work is currently underway by our engineering team to redirect traffic and increase capacity. 7 The mitigation is expected to complete around Tuesday, Jun 08:40 2022-06-07 09:00 US/Pacific. 2022 PDT We will provide more information by Tuesday, 2022-06-07 09:30 US/Pacific. Diagnosis: Increased packet loss for traffic traversing the affected metros. Workaround: None at this time. Summary: Packet loss observed from Internet in Middle East to Google Description: We are experiencing an issue with Cloud NAT, Google Cloud Networking, Hybrid Connectivity, Virtual Private Cloud (VPC) beginning at Tuesday, 2022-06-07 06:00 US/Pacific. There is a latency in connectivity between Europe and 7 Asia. Jun 07:44 2022 PDT Mitigation work is still underway by our engineering team. We will provide more information by Tuesday, 2022-06-07 09:15 US/Pacific. Diagnosis: Increased packet loss for traffic traversing the affected metros. Workaround: None. Summary: Major packet loss observed from Internet in Middle East to Google Description: We are experiencing an issue with Cloud NAT, Google Cloud Networking, Hybrid Connectivity, Virtual Private Cloud (VPC) beginning at Tuesday, 2022-06-07 06:00 US/Pacific. 7 Jun 07:21 Our engineering team continues to investigate the issue. 2022 PDT We will provide an update by Tuesday, 2022-06-07 08:00 US /Pacific with current details. Diagnosis: Increased packet loss for traffic traversing the affected metros. Workaround: None. Summary: Major packet loss observed between multiple regions - Europe and Asia Description: We are experiencing an issue with Cloud NAT, Google Cloud Networking, Hybrid Connectivity, Virtual Private Cloud (VPC) beginning at Tuesday, 2022-06-07 06:00 US/Pacific. 7 07:01 Our engineering team continues to investigate the issue. Jun PDT 2022 We will provide an update by Tuesday, 2022-06-07 08:00 US /Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Increased packet loss for traffic traversing the affected metros. Workaround: None. * All times are US/Pacific * Documentation * RSS Feed * JSON History | Schema * Send feedback * Privacy policy * Available * Service information * Service disruption * Service outage