https://www.ory.dev/global-identity-and-access-management-multi-region/ This app works best with JavaScript enabled. Open main menu Products Solutions Resources Company Pricing ContactLog InRegister Ory Network How to Build a Globally Distributed, Multi-Region Identity and Access Platform with Go [svg] Photo of Aeneas Rekkas Aeneas RekkasFounder, CTO August 07, 2023 As the world becomes increasingly interconnected, businesses are no longer serving customers in a single geographic region. This shift towards a global marketplace brings with it new challenges when it comes to identity and access management (IAM). At Ory, we recognized that our customers needed a solution that could meet where our customers' users are: everywhere in the world. These users expect the service to be fast, reliable, and compliant with local laws. Ory Network is a global identity and access platform for aspiring and established web-scale businesses. In this blog post, you will learn how Ory built a globally distributed system for identity and access management. This blog post is for you if you want to learn * what multi-region means. * how Ory implemented a scalable multi-region architecture. * how Ory migrated a live production system from a single region to a multi-region architecture. What is Ory? Ory runs a global identity and access (IAM) platform (Ory Network), based on open source, that companies use to solve login, registration, permissions, user management, and advanced topics like OAuth2 and OpenID Connect. Ory Network is the only multi-region IAM in the world. Companies using Ory Network reduce risk, lower their TCO, reduce time-to-market, and win over users with ultra-fast websites and apps. Every day, Ory is handling around 3 billion API requests in more than 11.000 production environments worldwide. Why multi-region We will explore the fundamental benefits of a multi-region architecture throughout the article. The most compelling reasons to build a multi-region architecture is that modern software applications need to be fast everywhere in the world, always on and reliable like google.com, and compliant with laws and regulations like GDPR.' Multi-region architectures help solve these issues efficiently and effectively. At the same time, multi-region architectures are not easy to build, and this article explores how Ory achieved this milestone for its IAM product: Ory Network. Single- and multi-regions In this section, we will explore the fundamental differences between single region and multi-region architecture, two contrasting approaches in designing and deploying cloud-based systems. Availability zones are not regions Cloud platform regions and availability zones are two distinct concepts that people often confuse. Regions refer to geographic locations where cloud providers have their data centers, and each region operates independently. Data stored in one region is not automatically replicated in others, meaning that an outage in a region only affects applications and data within that specific region. Availability zones, on the other hand, are isolated areas within a region, each with its own data center. Deploying across multiple availability zones enhances redundancy and availability, allowing applications and data to fail over to another zone in case of an outage. However, some cloud providers may have separate availability zones located in the same physical data center, which can pose risks during catastrophic events. For example, the failure of all three availability zones in Google Cloud Platform's europe-west9 region was caused by a fire in the data center where all three zones were located, despite each zone having independent resources. Single region architecture Single region architecture Single region architecture can be problematic for several reasons, primarily because it lacks the necessary redundancy and fault tolerance to handle various types of failures. 1. Increased risk of downtime: In a single region architecture, if the entire region experiences an outage due to natural disasters, infrastructure failures, or other unforeseen events, the application and services hosted in that region will become inaccessible. This can lead to prolonged downtime and significant disruption to business operations. 2. Data loss and corruption: Without data replication across regions, a catastrophic failure in the single region could result in data loss or corruption. If backups and recovery processes are not robust, recovering lost data may be difficult or impossible. 3. Limited disaster recovery: In a single region setup, disaster recovery options are constrained. If the primary region faces a disaster, the lack of geographically distributed resources makes it challenging to switch to an alternative location and recover quickly. 4. Performance bottlenecks: Having a single region could lead to performance bottlenecks, especially if the user base is spread across different geographical locations. Users farther away from the data center may experience increased latency and slower response times. 5. Vulnerability to network failures: A single region is more susceptible to network connectivity issues. If there is a network outage that affects the region, all services hosted within that region could be impacted. Single region architecture in multiple regions Single-region architecture in multiple regions A single region architecture involves deploying a copy of the entire software stack in each location. Deploying a single region architecture in multiple regions to reduce latency and address compliance concerns might seem like an attractive approach at first glance. However, it still shares many of the drawbacks and limitations inherent in a traditional single region architecture. 1. Data fragmentation and inconsistency: While deploying the architecture in multiple regions may improve latency for users in those regions, it introduces data fragmentation and inconsistency across regions. Each region operates independently, resulting in potential disparities in data sets and configurations. This can lead to challenges in data synchronization, data consistency, and maintaining a unified view of data across regions. 2. Increased management overhead: Adopting the same architecture in multiple regions means maintaining multiple production deployments, each with its own set of codebases (GitOps), configurations, and updates. This can lead to increased complexity in software management, making it challenging to roll out updates, bug fixes, and new features consistently across all regions. 3. Data merge and movement complexity: Moving and merging data between multiple regions in a replicated architecture can be complex and error-prone. Transferring data between regions for load balancing or compliance purposes requires careful planning and execution to avoid data inconsistencies or potential data loss. 4. Increased operational overhead: Running multiple instances of the same architecture in various regions requires additional management overhead. Each region needs separate monitoring, maintenance, and support, adding complexity to the overall system management. Multi-region architecture A multi-region architecture is challenging to build and run, but can address the limitations of the single-region architectures. Here, the data plane is replicated and shared and available in all regions. Stateless services are replicated the same regions as well. This approach has several advantages. 1. Improved high availability: Multi-region architecture offers higher availability by distributing application instances across multiple regions. If one region goes down, traffic can be seamlessly routed to other functioning regions, ensuring continuous service availability. 2. Disaster recovery and business continuity: With multiple regions, companies can implement robust disaster recovery plans. In case of a disaster or outage in one region, they can failover to other regions, minimising downtime and ensuring business continuity. 3. Reduced latency: By hosting resources closer to end-users in different regions, companies can reduce latency and provide better user experiences. 4. Enhanced data redundancy and security: Replicating data across regions ensures data redundancy, improving data resilience and mitigating the risk of data loss. It also enhances data security, as sensitive data can be stored in specific regions to comply with local regulations. 5. Compliance and risk management: For companies with global operations, hosting data in multiple regions helps them comply with data sovereignty regulations and reduces risks associated with region-specific legal and compliance issues. Multi-region Identity and Access Management Multi-region identity and access management Ensuring robust identity and access management across multiple regions is a critical aspect of an organization's infrastructure. Multi-region IAM addresses several key challenges and requirements to provide a seamless and secure user experience while adhering to data regulations and ensuring high availability. Low-latency access: Since identity and access management are integral to almost every user request made to a system, achieving low-latency response times is vital. With IAM services distributed across multiple regions, users experience reduced latency during identity verification and access authorization, enhancing overall system responsiveness and user satisfaction. Enhanced availability and reliability: Multi-region IAM provides increased availability and reliability by offering redundancy across regions. In the event of a single-region failure or outage, users can seamlessly access applications and services from alternative regions, minimizing downtime and ensuring continuous availability. This level of fault tolerance is particularly crucial for critical services that need to be accessible at all times, irrespective of localized failures. Compliance with data homing regulations: While having a unified IAM system is crucial, replicating user data across all regions can conflict with data homing regulations (GDPR, LGPD, PIPA, CCPA, ...) that require storing data within a user's country. To address this challenge, organizations must strike a balance between centralizing IAM functions and adhering to regional data storage requirements. Implementing data sharding by geography allows organizations to scale their infrastructure based on regional demands while maintaining compliance with data regulations. Unified user identity across regions: One of the primary concerns in multi-region architectures is maintaining a coherent and unified view of user identities and their permissions across all regions. Regardless of the user's home region, it is essential to have a centralized IAM system that can efficiently handle identity verification and access control for each user. This enables a consistent user experience, reduces complexity, and streamlines administrative tasks related to user management. Dynamic scalability: The geographic sharding of data and IAM services enables organizations to scale their infrastructure dynamically based on regional demand. For instance, during peak times like Black Friday in the US, organizations can scale up the IAM infrastructure in that region to handle increased user traffic. Simultaneously, regions with lower demand can remain efficiently scaled, optimizing resource usage and cost efficiency. Regulatory compliance and the need for data homing and encryption The handling of personal data has become a complex legal challenge, with no clear global consensus on how such data can be moved between countries. Despite being close allies, even the European Union (EU) and the United States have struggled to establish stable legal frameworks for cross-border data transfers: 1. Safe Harbor Agreement: The Safe Harbor agreement, which aimed to facilitate data transfers between the EU and the US, was invalidated after the revelations of Edward Snowden regarding US government surveillance practices. This decision highlighted the difficulties in ensuring data privacy in international data transfers. 2. EU-US Privacy Shield: Subsequently, the EU and the US attempted to establish a successor to the Safe Harbor agreement through the Privacy Shield framework. However, the Court of Justice of the EU declared the Privacy Shield invalid due to concerns over US surveillance practices and the lack of adequate protection for EU citizens' personal data. 3. EU-US Data Privacy Framework (Privacy Shield II): Currently, efforts are underway to establish a new EU-US data privacy framework. However, uncertainties persist regarding its approval, and the possibility of it facing legal challenges is very likely. Given the evolving and diverse regulatory landscape worldwide, businesses face significant uncertainties in handling personal data across borders. Data homing can mitigate the risks associated with regulatory uncertainty and changes. Data homing Data homing Data homing involves storing personal data within the user's country of origin and complying with data localization requirements that various countries may impose. This strategy minimizes the risk of non-compliance with regional data protection laws and ensures that sensitive data remains within the jurisdictional boundaries where it is subject to specific legal protections. Encryption While data homing provides a promising solution, it is not the only approach to safeguarding data privacy. Encryption, particularly advancements like homomorphic encryption, has gained attention as an alternative method. We considered this method, but it had to be ruled out in its current state. Homomorphic encryption Homomorphic encryption allows computations on encrypted data without the need for decryption, preserving data privacy throughout processing. This can be a powerful technique for protecting sensitive data during cross-border transfers and while stored in databases. Fully Homomorphic Encryption is not yet ready for market though: Today, conventional wisdom suggests that an additional performance acceleration of at least another 1 million times would be required to make FHE operate at commercially viable speeds. - Source Column-Level Encryption in SQL databases Implementing encryption at the column level in SQL databases offers another layer of data protection. By encrypting specific columns containing sensitive information, businesses can maintain data confidentiality while still supporting necessary operations like unique constraints. Different approaches to column-level encryption, such as deterministic and randomized encryption, provide varying trade-offs between security and query performance. Popular open-source databases (MariaDB, PostgreSQL, MySQL, CockroachDB) do not offer advanced column-level encryption that can deal with constraints (foreign keys, unique) and efficient range queries (SELECT * FROM table WHERE encrypted_column > 'some_value'). We would have preferred column-level encryption for managing personal data, but there is no good solution available for this yet. Like what you're reading? Get more high quality content delivered to your inbox and sign up to Ory's newsletter for free! Let's design a multi-region application architecture So far we explored the basics of multi-region architecture, regulatory considerations surrounding data homing and encryption. We also talked about the challenges that are posed by the lack of global data regulation and the potential complexities of cross-border data transfers. In this second part we show a practical example of how the single region Ory Network is transformed into a multi-region architecture, an essential step towards achieving high availability, fault tolerance, and scalability. Multi-region architecture Understand your data To optimize data management and compliance efforts, a thorough understanding of the data types within the system is essential. We distinguish between three main categories of data at Ory: 1. Metadata encompasses non-personal information such as session IDs, session tokens, or other data that does not contain identifiable details. These data elements are typically short-lived and used for session management and tracking user interactions. 2. Transient Data may contain personal information but is short-lived in nature. Examples include cached data that is removed after a few hours or temporary storage of certain user attributes during a session. 3. Persistent Personal Data: This category covers data that can directly identify a user, including emails, IP addresses, names, and addresses. Persistent personal data is stored for more extended periods, often within databases, and requires adherence to stringent data privacy regulations. Choose the right distributed data store for a multi-region architecture Building a robust multi-region data architecture requires careful consideration when selecting the appropriate distributed data store. Traditional SQL databases, such as MSSQL, MySQL, MariaDB, and PostgreSQL, are not suitable as the backbone of a multi-region setup due to their limitations in spanning multiple regions. Instead, organizations need to explore databases specifically designed for distributed environments. Some popular options include AWS Aurora, Google Cloud Spanner, CockroachDB, YugaByte, and other distributed databases. Each distributed database comes with its unique qualities and trade-offs. However, at the core, they all grapple with the complexities of distributed systems. These databases are relatively young compared to traditional SQL databases, meaning that users may encounter unforeseen issues during implementation and operations. Being aware of the trade-offs is crucial in making informed decisions for the specific requirements of the architecture. Aligning database choice with data model and regulatory needs Selecting the right database also heavily depends on the data model and compliance requirements. For organizations dealing with sensitive information and data homing obligations, databases that can set regions per row or leverage encryption for compliance become essential. Adhering to data localization laws may influence the choice of database to ensure data is stored within the appropriate regions to maintain regulatory compliance. Adjust latency expectations In a multi-region architecture, data distribution across numerous nodes results in increased latency compared to single-region setups. As data needs to be retrieved from multiple nodes, the roundtrip latency between these nodes can significantly impact query performance. And some writes need to travel around the globe to be finalized. Be prepared to manage expectations regarding latency and consider potential optimizations to reduce query response times. Multi-region architectures are often fast at reading and slow at writing. The Ory Network Stack The Ory Network Stack is built on a carefully curated set of technologies that enable the platform to achieve its multi-cloud capabilities, seamless deployment, and efficient management. Each technology choice was made with specific considerations in mind to ensure scalability, performance, and robustness. The Ory Network multi-region software stack Golang Golang is known for its speed, efficiency, and robustness, making it an excellent choice for building scalable and high-performance applications. Its concurrency features enable efficient handling of multiple tasks simultaneously, making it ideal for multi-region architectures where parallel processing is crucial. Additionally, Golang's strong standard library and well-defined syntax contribute to faster development cycles and easier maintenance. Its static typing and built-in garbage collection provide a level of safety and reliability, reducing the likelihood of runtime errors. Golang's simplicity and ease of deployment also contribute to smoother application development and deployment processes. All of these properties made Golang the obvious choice for developing all of Ory's API services. Kubernetes Ory Network relies on Kubernetes as the underlying container orchestration platform. Kubernetes offers the flexibility to run applications across various cloud providers, making it an ideal choice for achieving multi-cloud capabilities. This is particularly beneficial when serving private instances for customers who might have specific cloud preferences or requirements such as running on Azure. Kubernetes' ability to abstract away the infrastructure details also simplifies the deployment and management of Ory Network's services. ArgoCD Automated deployment and lifecycle management are critical components of Ory Network's architecture. ArgoCD, as a GitOps tool, streamlines the deployment process by automatically reconciling the desired state specified in Git repositories with the actual state of Kubernetes clusters. This approach ensures consistent and reliable deployments, making it easier for the Ory team to maintain synchronization across multiple regions. Crossplane Crossplane plays a crucial role in Ory Network's architecture, enabling seamless multi-cloud resource provisioning. By using Crossplane's Kubernetes-style APIs, Ory simplifies infrastructure management and ensures uniformity across diverse cloud providers. This integration enhances Ory's agility and resilience, allowing consistent resource scaling and deployment across different clouds. CockroachDB CockroachDB was chosen as the primary database for Ory Network due to its unique features and capabilities. Built on the Raft consensus algorithm and based on RocksDB, CockroachDB offers a distributed and highly available database solution. These features are crucial for Ory Network's multi-region architecture, enabling data to be stored across regions and sharded efficiently for better scalability. Cloudflare The anti-bot and anti-fraud capabilities ensure enhanced security and protection against malicious attacks. Additionally, global load balancing allows Ory to steer traffic to the correct geolocation based on the user's origin. Ory's global edge sessions use Cloudflare workers to perform session validation in milliseconds everywhere in the world. Grafana, Loki, Tempo, Thanos Ory employs a comprehensive logging and monitoring stack, comprising Grafana, Loki, Tempo, and Thanos. Grafana provides a user-friendly interface for visualization and analysis of performance metrics. Loki, a horizontally-scalable log aggregation system, efficiently handles log data. Tempo serves as a high-scale distributed tracing system, facilitating application performance monitoring. Thanos ties these components together, offering a seamless experience for monitoring and debugging Ory's applications. The journey with CockroachDB Implementing a global multi-region system with CockroachDB brings both excitement and challenges. While CockroachDB enables Ory to achieve its ambitious multi-region architecture, the journey has been marked by valuable learnings and trade-offs despite following best practices from the outset. Let's explore working with CockroachDB in the context of a global multi-region production setup. Multi-region database CockroachDB Sharding and latency trade-offs Sharding and replicating a database across regions introduces challenges primarily related to latency between nodes. Designing primary keys, unique and foreign key constraints, and secondary indices requires careful consideration. To optimize performance, avoid operations that scan rows across several nodes or regions. Range queries and ordering operations that span rows across multiple nodes or regions can be slow. Three features for regionality CockroachDB offers three features for handling regionality: 1. Regional tables: Tables stored and accessed in a single region provide fast access within the region but experience severe latency penalties when accessed from other regions. This type of table is less common in multi-region architectures. 2. Regional by row tables: These tables are sharded across regions, and each row specifies the region in which it is stored. Read and write queries on such tables come with high latency penalties, necessitating careful optimization to avoid performance issues. Query optimization involves avoiding ordering, limiting result sets, and using primary or secondary indices for lookups. 3. Global tables: These tables deliver fast consistent reads across all regions but incur latency penalties on writes. Transactions experience a non-removable sync time (~300-600ms) during the COMMIT phase, and all writes occur in the table's primary region, adding additional latency for INSERT and UPDATE operations. Eventually consistent reads CockroachDB supports eventual consistency, where stale reads may occur, providing data that might not be up-to-date. In such systems, when a user performs a write operation (e.g., signing in and updating session data), it may take a certain amount of time for the changes to propagate and become available for subsequent read operations. This duration is referred to as the replication lag and can range from 1 to 30 seconds, with CockroachDB's replication lag typically around 4 seconds. To address eventual consistency and provide more control over data freshness, CockroachDB offers a mechanism for specifying a timestamp for read operations. By using the AS OF SYSTEM TIME clause in SQL queries, clients can request data as of a specific timestamp, ensuring they receive data up to that point in time. For instance: SELECT * FROM table AS OF SYSTEM TIME "2023-07-20-12-55-01"; However, employing this approach poses challenges for application developers. The responsibility of managing the timestamp now shifts from the SQL server to the client. Clients must maintain and provide the appropriate timestamp for their read operations. This can be non-trivial, particularly in scenarios where the system requires strict consistency for certain operations. One potential solution is to use immutable keys. When performing an INSERT or UPDATE, a new key is returned, encoding both the primary key and the timestamp. This way, the client can use the immutable key to ensure consistent reads. However, this adds complexity as the client must manage and maintain the consistency of immutable keys for every update operation. It is essential for developers to carefully consider the implications of eventual consistency when designing their REST or gRPC APIs. By setting clear expectations regarding data freshness and potential staleness in the API documentation, clients can be informed about the trade-offs between fast but eventually consistent results and potentially slower but strongly consistent data. Solving compliance with data domiciling CockroachDB has been instrumental in addressing compliance concerns by leveraging regional by row tables. Persistent personal data is stored and replicated within super regions, ensuring compliance with data localization requirements. Additionally, the data is encrypted using regional encryption keys, strengthening data security and privacy. Data privacy (GDPR) compliance API Multi-region data architectures take time, monitoring, and effort Developing multi-region data models with CockroachDB demands meticulous planning and substantial time investment. Effective application monitoring and query tracing are essential to identify performance bottlenecks in live production traffic, making it a slow and iterative process. While CockroachDB empowers global multi-region systems, its unique architectural considerations necessitate a thorough understanding and optimization of queries to achieve optimal performance. Despite the challenges, the ability to create a robust and scalable multi-region architecture is within reach, provided teams embrace the learning curve and are committed to continuous improvement. We are quite happy with the results even if the journey to getting here has been rocky. We think that Cockroach Labs could do more in terms of documenting the trade-offs and downsides of their product, which may have saved engineering time. Having said that, engineers and customer success at Cockroach Labs have been very helpful in helping us to go live! Gain a competitive edge and embrace open source Ory's journey began as an open source project, and over time, it has evolved into an open source company. Within the competitive landscape, Ory Network stands out with a unique competitive advantage over entities that self-host open source solutions or attempt to offer Ory's open source software as a service, as happened with AWS and ElasticSearch. To meet the complex demands of a multi-region environment and offer enhanced services to its customers, Ory recognized the need for a specialized multi-region, multi-tenant database abstraction layer. While the core components of Ory remain open source, this proprietary layer serves as a strategic differentiator, allowing Ory Network to deliver efficient and scalable database management tailored to the specific requirements of multi-region deployments. At the core of Ory's success lies its innovative approach to multi-region deployments. Keeping the complexities and challenges of managing data across regions in mind, we took a decisive step by implementing a proprietary multi-tenant database abstraction layer. This strategic move allows Ory to deliver an unparalleled level of efficiency and scalability, tailored to the unique demands of multi-region architectures. While the core components of Ory remain open source, this proprietary layer proves Ory's commitment to delivering cutting-edge solutions. Developing a robust multi-region architecture and surrounding tooling and monitoring has been a monumental effort for the team. Synchronizing application and deployment states across multiple regions demanded meticulous orchestration, ensuring smooth operations across the distributed landscape. Furthermore, Ory's focus on cost-effective monitoring has ensured resource optimization while delivering exceptional performance in a multi-region environment. Despite its growth and evolution, Ory remains firmly grounded in the principles of open source. The vibrant community surrounding Ory continues to drive innovation, fostering a collaborative environment where ideas are exchanged and improvements are continuously made. This commitment to open source principles is a cornerstone of Ory's success, enabling the company to stay adaptive and forward-thinking. Migrating live production traffic from a single region to multiple regions So far we discussed the motivation behind migrating from a single-region to a multi-region architecture within Ory Network. Understanding the importance of IAM, data regulation, compliance, and GDPR, we set the stage for the challenges and opportunities that lay ahead. From single-region IAM to multi-region Ory Network's journey started with a single-region setup in Europe. Recognizing the potential benefits of a multi-region architecture, we embarked on a phased approach to roll out this advanced infrastructure. Experimenting with multi-region latencies One of the pivotal challenges we encountered was the need for real-world experience to comprehend how queries behave in a multi-region system. Unlike laboratory environments, the complexities of real-world scenarios demanded extensive experimentation. In this phase, we faced many challenges with multi-region capabilities in CockroachDB. Most of our smoke tests encountered timeouts shedding light on the substantial work that awaited us. The road to a successful multi-region deployment necessitated addressing the myriad of challenges, ranging from optimizing performance to ensuring data compliance across regions. Our experiments and insights paved the way for a more comprehensive understanding of the intricacies involved. The challenge of real-world experience Transitioning from a single-region setup to a multi-region powerhouse was no easy feat. We navigated through several phases, each bringing its unique set of obstacles. One of the most profound challenges was understanding how latencies behave in a real multi-region system. No laboratory environment could replicate the intricacies of a live, dynamic ecosystem. It took significant time, effort, and real-world experience to grasp the nuances and intricacies, shaping our system's architecture and performance optimization. Exploring CockroachDB multi region features In the second phase of our journey towards a multi-region architecture, we explored different multi-region features, with a particular focus on the newly introduced capabilities in CockroachDB, such as Global Tables and Super Regions. This phase served as a crucial testing ground to gain insights into the performance and behavior of various table types. Testing global tables, regional tables, and regional by row tables We tested and evaluated three distinct table types offered by CockroachDB to accommodate multi-region setups: 1. Global tables: These tables offered the advantage of fast reads across all regions but introduced latency penalties on writes. Because of global replication, they can also not be used for PII. 2. Regional tables: Regional tables are stored and accessed within a single region, ensuring fast performance for local operations. Regional tables are rarely useful for global state unless follower reads can be used. 3. Regional by row tables: The most complex table type we experimented with, regional by row tables are sharded across regions, with each row specifying the region it is stored in. While this approach provided the flexibility of data distribution, it also presented significant challenges related to query performance and latency. As we conducted extensive tests with these multi-region features, we consistently encountered long-running queries, leading to issues of system unreliability and, at times, unavailability. The performance bottlenecks revealed the intricacies involved in optimizing a multi-region database abstraction layer written in Golang. In light of the challenges and performance issues, we committed ourselves to gaining a deeper understanding of the underlying factors contributing to long query times. Optimizing critical APIs for enhanced performance In the third phase of our migration journey, we set our sights on optimizing the performance of critical APIs that accounted for approximately 95% of the overall traffic in Ory Network. These APIs were responsible for key functionalities, such as validating sessions, checking user permissions, and verifying access tokens which are typically called in the serving path of critical systems and thus are the most sensitive to increased latency. Our goal was to significantly improve the response times of these APIs, thereby enhancing the overall speed and latency of the system. Addressing Performance Constraints To accomplish this, we dedicated our efforts to addressing performance constraints in three fundamental APIs: 1. Session Validation: Ensuring the validity of user sessions is crucial for maintaining a secure and seamless user experience. We diligently worked to optimize this process, minimizing response times and enhancing overall session management. 2. Authorization Checks: Verifying user permissions and determining whether an action is allowed is a fundamental part of Ory Network's functionality. Our optimization efforts aimed to streamline authorization checks, enabling faster decision-making without compromising on security. 3. Access Token Validation: Validating access tokens is critical for safeguarding resources and ensuring that only authorized users gain access to protected endpoints. We focused on optimizing this process to reduce access token validation times. Tracing Instrumentation and Analysis To achieve these performance improvements we instrumented critical APIs, closely monitoring request and query patterns to identify areas that required optimization. As you can see on the right, we gained insight into how the system behaves with on average ~40 spans per API call. This thorough analysis provided insights into the underlying bottlenecks and inefficiencies that needed to be addressed. By taking these insights for Session Validation and improving the business logic, we reduced Session Validation end-to-end latency from ~140ms to ~29ms on the median. This includes fetching the full identity profile including personal data. Screenshot 2023-08-01 at 14.27.50.png Data homing practices for enhanced compliance As part of our ongoing commitment to data regulation and compliance, we also ensure personal data follows data homing practices. By adhering to regional data storage regulations, we fortified our infrastructure's compliance with applicable data protection laws and reinforced user trust in the security of their data. As a result of our focused efforts during this phase, Ory Network experienced significant performance enhancements and a reduction in latency for critical APIs. These optimizations paved the way for an efficient and responsive multi-region architecture, capable of delivering high-quality services to our users across the globe. Query, data model, and API optimization Once we had a comprehensive understanding of the issues affecting performance, we began the process of optimizing queries, data models, and APIs. With a data-driven and methodical approach, we fine-tuned the database queries, redesigned data models, and streamlined the API interactions to eliminate performance bottlenecks. Satisfactory results and road to roll out Our optimization efforts yielded promising results, significantly reducing the issues we had previously encountered. As the performance of critical APIs improved, the overall system's speed and responsiveness witnessed a substantial enhancement. With the critical APIs now optimized, we set the stage for the final phase - the rollout of the multi-region architecture. The groundwork laid during this phase would prove invaluable in ensuring a smooth and efficient transition to the enhanced architecture. Sequencing the roll-out As we neared the final stages of transitioning to a robust multi-region architecture, Phase 4 involved sequencing the roll-out to ensure a seamless and non-disruptive migration. With our multi-region test instances successfully passing end-to-end and smoke tests, we initiated the process of drafting a comprehensive migration plan. This plan was designed to facilitate the migration of CockroachDB while minimizing any potential impact on our customers. Roll-out phases The roll-out plan was thoughtfully structured into several distinct phases: 1. Provisioning Kubernetes clusters and CockroachDB nodes The initial step involved the establishment of Kubernetes clusters within the designated regions, specifically US East and US West. Correspondingly, we deployed CockroachDB nodes within these regions, setting the foundation for the subsequent migration phases. 2. Gradual migration of services The migration was strategically phased to minimize disruption. We began by migrating our permission service, a pivotal component with relatively low impact on the database. This approach ensured a smooth transition while maintaining service availability. 3. Transitioning the service control plane Following the successful migration of the permission service, we proceeded with transitioning our service control plane. This encompassed all elements responsible for managing Ory Network projects and entailed additional business logic associated with multi-tenancy and domain routing. 4. Migrating Ory Identity service A critical milestone involved migrating the Ory Identity service, the largest service within Ory Network. This step required meticulous planning and execution to maintain data integrity and ensure seamless user experiences. 5. Relocating Ory OAuth2 service The final phase focused on migrating the Ory OAuth2 service, catering to our significant customer base that relies on this service. The culmination of this phase marked a significant achievement in our multi-region migration journey. 6. Enabling global traffic and customer onboarding Once the intricate database migrations were successfully executed, we selectively enabled global traffic for Ory Network itself. This was extended to our customers who expressed interest in trying out the new features serving traffic within the US regions. The roll-out This section provides an overview of the challenges encountered and the successful outcomes achieved during this complex and critical phase of Ory Network's development. Ory Network is now multi-region! In the broader context, the multi-region migration was a success! Presently, users located in the United States experience comparable latency to those in the European Union, because of the new multi-region capabilities. This achievement marked a significant improvement in service performance, reliability, and user experience, aligning with Ory Network's vision for global accessibility and responsiveness. Navigating the challenges The migration journey was not without its share of challenges. While the outcome proved fruitful, the process demanded meticulous planning, diligent execution, and quick adaptation to unforeseen issues. Some of the prominent challenges encountered during the migration are highlighted below: CPU usage and P95 latency increase: During the migration phase, the system experienced spikes in CPU usage and a rise in P95 latency. These observations raised concerns about potential instability if subjected to a large surge in traffic. Fortunately, the system remained resilient, and P95 latency returned to acceptable levels after the completion of the database migration. P95 latency during migration Migrating large tables to regional-by-row tables: The live migration of large tables to regional-by-row tables presented a significant time-consuming task due to the sheer size of the database. This process required several hours to complete successfully. Service degradation and outages Ory Network status page The migration period, to our regret, was also marked by service degradation and outages. Several contributing factors led to these challenges: * Deployment rollouts across multiple regions: Deploying changes across various regions resulted in varying upgrade speeds, leading to compatibility issues with previous versions. These incompatibilities caused partial system downtime and required immediate resolution. You can find a post-mortem about this issue on our website. * Zombie process impacting email sending: A critical outage arose from a persistent zombie process attempting an UPDATE ... statement on a large dataset to change row regions. This operation locked a table for approximately two hours, impacting email sending functionality. Updating more than 100,000 rows simultaneously led to TRANSACTION_RETRY errors. To overcome this obstacle, dedicated tooling was developed to update large tables in manageable chunks, a process that spanned approximately two days. The multi-region migration journey presented Ory with a series of challenges, from performance bottlenecks to service degradation. These obstacles, while demanding, have provided invaluable insights that are driving continuous improvement and shaping the future resilience of the system and we are applying these learnings to reduce the likelihood of such events. Make a multi-region request yourself! Let's make a multi-region AJAX request together. Depending on your location in the world, you will see that your API call will be served by either EU-West (euw), US East(use) or US West (usw). Let's make such a request together using a shell hosted in the US: https://replit.com/@aeneasr/ Ory-Network-multi-region-session-validation-from-the-US#main.sh To test this out from your own network, run this curl command in a shell: curl -sIXGET https://playground.projects.oryapis.com/sessions/whoami 2>&1 | \ grep -i ory-network-region What is next for Ory Network? As Ory Network continues to advance its multi-region architecture and provide an exceptional user experience, a road map for future enhancements has been formulated. The team is committed to addressing existing challenges and delivering a more efficient, flexible, and user-centric system. Here are some of the key initiatives planned for the next phases: Pursuing enhanced latency and performance While significant progress has been achieved in migrating to a multi-region architecture, Ory Network recognizes that there is more to be done to further optimize latency and performance. The road map ahead includes a concerted effort to bring down P90 and P95 latency metrics, with the expectation that P99 latency will likely remain in the 600-900ms range for expensive operations. Certain complex queries may pose challenges for optimization, such as retrieving a comprehensive list of users spanning all regions. Empowering data compliance and mobility Ory Network is committed to strengthening data compliance and user data mobility through innovative solutions. A pivotal advancement on the horizon is the development of a new data compliance API. This forthcoming API will empower developers with the capability to seamlessly transfer users and their associated data from one region to another. This initiative not only aligns with regulatory requirements but also provides users with greater control over their data placement. User-centric self-service capabilities Recognizing the importance of user autonomy, Ory Network is dedicated to introducing self-service tooling and user interfaces. These user-centric features will enable individuals to independently change their designated region, facilitating a seamless and personalized experience. By placing the power of data location in the hands of users, Ory Network aims to enhance flexibility and user satisfaction. Project-level data placement control In its continuous pursuit of comprehensive data management, Ory Network is actively working on introducing data placement restrictions at the project level. This strategic enhancement will provide project administrators with granular control over the geographic distribution of data. By tailoring data placement to specific project requirements, Ory Network enhances compliance adherence while offering flexibility to accommodate diverse regulatory landscapes. An exciting journey ahead As Ory Network embarks on the next phase of its evolution, the organization remains steadfast in its commitment to delivering a resilient, compliant, and high-performance platform. With a clear vision for the future and a robust road map in place, Ory Network is poised to continue its mission of empowering developers and users alike, while embracing technological advancements and best practices in the ever-evolving realm of multi-region architecture. Searching personal data across regions Implementing a robust search functionality poses a unique challenge in a multi-region setup. The presence of Personally Identifiable Information (PII) in multiple regions necessitates a search engine that respects data regulations while delivering fast and accurate results. Currently, no multi-region search engine exists, and we are committed to exploring solutions that strike the perfect balance between efficiency, compliance, and user privacy. Scaling fast At current growth projection, Ory will be handling 100 million active users by the end of 2023, scaling up to 1 billion active users by the end of 2024. This expansion brings the biggest challenge yet: scaling a multi-region, distributed system at a rapid pace. To meet this challenge head-on, we are dedicated to find effective solutions in record time. Conclusion You made it to the end, congratulations! I hope this article helped you understand that multi-region architectures are hard but possible. If you like what you read, please try out Ory Network! It's free to play around, and we at Ory hope you enjoy the product as much as we enjoy building it... :) Never miss an article - Subscribe to our newsletter! [ ]Subscribe Back to blog Footer GithubYouTubeTwitterLinkedIn Platform * Product Overview * Login & Authentification * Identity Management * Permissions & Access Control * API Access & OAuth2 * Machine-To-Machine * Pricing Popular Features * Passkeys & Passwordless Login * Single Sign-On across Apps * Multi-Factor Authentification * Social Sign-In * Third Party API Access * Fine-Grained Permissions * Profile & Identity Management Solutions * Zero Trust * Privacy & GDPR Compliance * Security * Scalability * Migrate to Ory * Network vs. Self Hosted * Adopters * Case Study: Fandom * Case Study: zezam * Case Study: Maxroll Company & Resources * About us * Compare Ory * Support Plans * Jobs * Ory Summit Talks * Documentation * Ory CLI * Ory Open Source * Blog * Community Status Privacy Policy Terms of Use Imprint (c) 2023 Ory. All rights reserved. tracker