https://building.nubank.com.br/why-we-killed-our-end-to-end-test-suite/ * Working at Nubank View category Working at Nubank + Recruiting & Onboarding + Career & Life Stories * Culture & Values * Engineering & Tech View category Engineering & Tech + Software engineering + Data science + Product * Design * Press * Nu Impact About Nubank Go back [ ] {{#posts}} * {{{title}}} {{/posts}} {{#tags}} * {{tag_name}} {{/tags}} Menu * Working at NubankView category Working at Nubank + Recruiting & OnboardingView category Recruiting & Onboarding + Career & Life StoriesView category Career & Life Stories * Culture & ValuesView category Culture & Values * Engineering & TechView category Engineering & Tech + Software engineeringView category Software engineering + Data scienceView category Data science + ProductView category Product * DesignView category Design * PressView category Press * Nu ImpactView category Nu Impact |Search post Why We Killed Our End-to-End Test Suite How Nubank switched to a Contract and Acceptance testing strategy to scale to over 1k engineers 24 Sep 21 Engineering & Tech by Alexandre Freire - Director of Engineering Navigate [End-to-end] In the past at Nubank we used an End-to-End test suite to find issues across the boundaries of our microservices architecture in a staging environment. That implies that the interactions between services very often are backed by actual databases, messaging systems, HTTP requests, etc. In general, End-to-End are black-box tests in the sense that we stimulate one of the inputs of the system (e.g. by making an HTTP request to an endpoint or producing an asynchronous message to a topic). The system then produces several interactions between its parts; for example: possibly many other HTTP requests and/or asynchronous messages are produced. And we check the validity of these interactions by checking specific outputs; for example by calling another HTTP endpoint to check if the desired effect has been produced). End-to-end in a fintech As a fintech, quality is of the utmost importance for us. We need our customers to trust us with their money. Our End-to-End test suite complemented our testing strategy to assure our systems were of very high quality and integrity. If this is such a well known best practice among tech companies and other banks, why would we consider changing strategies? It turns out that this practice has a lot of downsides. In my early days working here I conducted an assessment of engineering pain points. Speaking to many different people and teams, the struggle with this type of test became a common theme. A diagnosis of our end-to-end test suite Most companies still believe that End-to-End Integration Tests are the best way to catch bugs. But they also experience a progressively slow down of value delivery due to these pain points that we uncovered during the assessment: 1. Waiting. Engineers had to wait more and more to get feedback from this long-running suite; 2. Lack of confidence. Flaky tests meant that we had to re-run the suite frequently to see if something was really wrong or just a false negative; 3. Expensive to maintain. Manual changes in our staging environment corrupted test data fixtures and maintaining the environment "clean" was a challenge; 4. Failures don't point to obvious issues. Test failures were very hard to debug, specially due to our reliance on asynchronous communication that make it hard to connect the cause of failure (a message not published to a queue) with its effect (changes not made in another system); 5. Slower value delivery. Queueing of commits in the End-to-End suite resulted in less frequent deployments; 6. Not efficient. Few bugs caught in this stage. One experiment suggested that, for every 1000 runs, we had 42 failures, only 1 bug; 7. Not effective. Bugs were still being found in production. Presentation to engineering leadership When showing the results of my assessment to the CTO and engineering leadership, I presented the situation: "If these trends continue, we'll take more and more time in our continuous integration and deployment pipelines, and eventually find ourselves stuck in a corner. Folks will commit bigger and bigger batches of changes that will take longer to be deployed, while still being open to the risk of bugs in production". A new hope: Contract Testing This diagnosis was not something novel or specific to Nubank. In my previous experiences I had seen the same situation in many Fortune 500 companies that struggled with their End-to-End test automation. But at Nubank we were determined to change this. One of our Sr Staff Engineers, Rafael Ferreira, ran some numbers and applied queueing theory. He determined that by 2021 Nubank's End-to-End test suite would take... an infinite time to run! So we decided to explore Consumer Driven Contract (CDC) testing as an option. Contract tests allow us to describe the interactions between our services through expectations to be met by their inputs and outputs. For instance, a billing service needs the customer's first and last name to generate a bill. So, we may describe the interaction between billing and customer services according to a contract: the customer service exposes the endpoint GET / customers/{id} and expects to receive a valid UUID. The billing service in turn, as a consumer of this endpoint, expects to receive two attributes: first_name and last_name (both non-empty strings). Any unexpected changes on these assumptions (e.g. the customer service changes the last_name attribute to be optional meaning that it may be empty) consist in a contract breakage that may affect the behavior of business flows in runtime and must be caught by contract testing tools. So in summary: * End-to-End tests are helpful at the beginning of a project when things are still simple and confined to a single team. However, as both project and team grow, they start to become a bottleneck because they demand coordination between teams and infrastructure work to keep them running reliably. * Contract Tests, on the other hand, require less coordination and infrastructure. It means that teams can evolve independently of each other. This is especially useful in a microservices architecture because the number of integration points tends to grow exponentially. Read also: "Nubank Data Analysts are now called Analytics Engineers" Why we chose to build our own framework How these contracts are declared and validated depends on the contract testing framework in question. However, a common characteristic is that inputs and outputs are collected and validated without executing black-box tests against actual instances running on production-like environments. Implementations of such concept may vary: 1. one can verify declarative schemas through a sort of generative test; or 2. test the calls made by a client application (the billing service in our example) against a mock server that responds with the same data returned by the actual server (the customer service). The second approach is the one implemented by Pact. We started it, since it's a quite consolidated framework for writing Contract Tests. However, two aspects caught our attention in our first experiments: * The support for messaging tests was immature in the JVM implementation: most of the critical interactions between our microservices occurs through Kafka messages (we favor mutations in asynchronous flows while HTTP calls are mostly reserved for read-only operations). Therefore, a solid support for asynchronous interactions was a crucial factor for us; * The JVM version lacked satisfactory support for Clojure, the programming language we use for the absolute majority of our microservices. Our decision Rafael and our team, including Lead Engineers Rui Hayashi and Alan Ghelardi, considering the above aspects, decided we should develop our own implementation of a Contract Tests tool. In fact, we picked many aspects from the Pact framework as well as from the consumer-driven contracts pattern as a source of inspiration. However, throughout this journey, we realized that even the traditional model of Contract Tests had some downsides that could make their adoption difficult for our circumstances. In general, Contract Tests are strongly dependent on the correct (and often complex) initial state in the microservices being tested to exercise relevant interactions among them. For very simple interactions (like those that frequently appear in examples of CDC tests) this might not be a problem, but in the context of a financial company with complex business rules spread across a wide variety of services, certainly, engineers would struggle very often to put their microservices into valid states to write useful tests. Read also: "Scaling with Flutter Building Sachem And so we decided to create Sachem, our very own contract testing framework, to deprecate End-to-End testing in staging as a practice. One interesting thing about this project is that we realized we already had the contracts in our microservices. Clojure has this library Schema that allows you to richly describe data structures. It was already common practice to write schemas for every HTTP endpoint and Kafka topic. What we did was to build a tool that collects those schemas and checks if they are compatible. Therefore, nothing really changed in terms of communication between teams, but it shows the value of having those contracts in place. Factors considered We have learned in a few experimental sessions with our engineers that, to be successful, our solution should be less intrusive and organically adhere to our codebase by leveraging existing aspects of our architecture, preferably, without forcing engineers to write complex tests. To proceed, we have considered the following factors: * Homogeneity. Our microservices are pretty homogeneous - they are mostly written in Clojure, generated by the same template, and follow very similar standards; * Schemas. As mentioned, all ports of our services (HTTP clients and servers as well as Kafka consumers and producers, following the Ports and Adapters pattern) declare schemas for their inputs and outputs; * Validation. Those schemas are plain Clojure data structures used to parse and validate inputs and outputs in runtime; * Violations. In our analysis, we figured out that the most frequent category of bugs caught by End-to-End tests was schema violations. With those aspects in mind, we gave up on the idea of providing a replacement for testing distributed business logic among services through contract tests and concentrated on validating uniquely their schemas with Sachem. Read also: "The value of canonictiy". One strategy to rule them all? Getting rid of E2E tests really helped to eliminate the coordination problems. Before Sachem we experienced issues frequently, since each team had a turn in the E2E pipeline, and they would barter to skip ahead of the queue to get their change in production faster, causing friction with from other teams. The engineering cost of keeping the tests working, waiting long turns to get code into production, and so on, were the main drivers of this change. It used to take at least two hours, in the best scenario, to get some code into production. It could take more than a day. Just think of the cost of context switching between tasks while you wait for something else to go to production. At first, the goal was to completely remove the End-to-End tests, but in the long-term, we found that we missed some of its benefits. Mainly, Contract Tests can catch structural incompatibilities, but they are not good in testing behavior. The guarantee But to keep our high quality standard we also needed to guarantee critical behaviour of our applications. So we found another way to complement our suites, by building what we call acceptance tests (not to be confused with Gherkin style acceptance tests). The main difference to the old E2E is that they encompass only a subset of services and don't require spinning a production-like environment (the services run in memory on a single JVM and HTTP/ Kafka communication is replaced by in-process communication). They are used in specific flows that we find too critical to only rely on Contract Tests. We also have an experimentation platform whereby people can run experiments against a subset of our customer base. It's a common practice to use techniques like feature flags, percentage rollouts, a /b testing, and so on. This has been working well both as a pure technical testing mechanism, but most importantly to gather business insights on new features. We have tried to implement distributed tracing instrumentation but turned it off because of a mix of not having much uptake from our internal users, and for the difficulty of maintaining the infrastructure. We intend to revisit that in the future. The results The results were quite remarkable, especially for two important metrics: * Cycle time: the time from merging something into the main branch until deploying in production went from unpredictable (hours or even days) to about 20 mins. * The number of deploys per week: we were doing at most a hundred deploys a week with E2E. After Contract Testing, that started to grow exponentially. Today, we are in the order of a thousand deploys per week. You may say that was just the growth of the company, but since E2E requires a queue, we would always be bound to a maximum number of deploys per week, which also means the cycle time would get worse and worse. It is important to mention that these are two of the four "Accelerate" metrics. The Accelerate book shows that companies that excel in those metrics are among the high performers in the Industry. Now if you, dear reader, want to join us in a high performing company that's liberating customers from complexity and giving them ownership of their financial lives, and also liberating software engineers from slow and brittle automated end-to-end test suites, just submit your information here. Co-authors: Rui Hayashi, Alan Ghelardi, Cesar Vortmann, Paulo Victor, Rafael Ferreira I would like to thank Cesar Vortmann for the inspiration for this post and questions that drove it we mentioned on CDC in this podcast and on this video on how we do end-to-end tests for our Microservices architecture. It became clear people wanted to hear more and we hope this post will help folks out there considering these different testing strategies. Also thanks to Rui Hayashi and Alan Ghelardi for being co-writers on getting answers for this article and Rafael Ferreira and Ezequiel Siddig for your review. Contract testing End-to-end Software engineering Hide comments [ ] Enter your name [ ] Enter your e-mail Your email address is invalid. Enter a valid address and try again. [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [Send comment] {{{message}}} {{{error}}} Submitting... Related Posts * Related Posts * Most Discussed * Most Popular 09 Sep 21 Press Nubank launches reward program for ethical hackers in partnership with HackerOne The Bug Bounty program rewards cybersecurity researchers who find potential vulnerabilities in our systems. 30 Aug 21 Engineering & Tech Yes, She Codes! Nubank's software engineering women's hiring program "Yes, She Codes" 3rd edition seeks to recruit new talents, in addition to promoting the exchange of experiences among candidates 24 Aug 21 Engineering & Tech How we deal with technical incidents How Nubank deals with technical incidents in a healthy and simple way 30 Sep 20 About Us Nubank takes next step in international expansion and launches in Colombia With more than 30 million customers across Brazil and Mexico, Nubank reinforces its positioning as leader of digital banking in Latin America. 22 Jul 20 Analytics Nubank Data Analysts are now called Analytics Engineers Aligning expectations and positioning ourselves towards better productivity and data governance 15 Jul 21 Analytics Scaling data analytics with software engineering best practices Analytical work can greatly benefit from software engineering standards in terms of quality and efficiency, here's how. 16 Jun 21 Data science Beyond prediction machines A huge part of Nubank's Data Science team has been recently focusing on leveraging causal inference along with traditional machine learning to solve some of our toughest problems. This post aims at explaining why causal inference is such a powerful tool to solve business problems with data. 02 Jun 21 About Us Nubank appoints Arturo Nunez as Chief Marketing Officer Former Apple, Nike, and NBA Exec. will lead global marketing efforts as NuBank expands, following the brand's recent repositioning 23 Apr 21 Engineering & Tech Technical Writing and the value of documentation Technical Writing team is responsible for supporting, teaching and helping engineers to create, maintain and publish useful, assertive and clear documentation. 09 Apr 21 Analytics Distributing the data team to boost innovation reliably Integrating Analytics Engineers into all cross-functional teams: a key success factor to scale a data platform efficiently and reliably 07 Apr 21 About Us Matt Swann is Nubank's new CTO With 25 years of experience leading technology companies, Matt Swann joins Nubank to continue accelerating our innovation pace. 11 Mar 21 Engineering & Tech A look into Nubank's tech hub in Berlin Our data infrastructure teams in Europe are instrumental in creating increasingly better products for our customers 08 Mar 21 Culture & Values Nubank agrees to have 50% of the leadership made up of women By 2025, the commitment is to hire 3,300 women and become the most egalitarian technology company in Latin America. 10 Feb 21 Engineering & Tech Scaling with Flutter A bit over a year ago, Alexandre Freire wrote an article about how we decided to standardize our mobile app development with Flutter, our early experiences with the platform, and why we decided to use it as our primary technology for cross-platform apps. In a recent interview with Nora Bezi from Bitrise, Alexandre, along with [...] 17 Nov 20 Culture & Values Coffee Break and MegaDemo: what the meetings with the entire Nubank team are like Waiting for the Covid-19 times to pass, we'd like to share these two Nubank events, designed to give employees visibility (in person!) of what is going on in the company. 30 Oct 20 Engineering & Tech The value of canonicity What happens when we run an engineering organization by constraining the number of tools in our toolbox "for the greater good"? 27 Oct 20 Our offices Behind the scenes: get to know Nubank headquarters in Sao Paulo and beyond Our workspaces reflect the values of the company: being efficient and thinking and acting like owners. (These are the environments we miss so much during our home office phase in 2020.) 23 Sep 20 About Us "We want to be remembered not only for the revolution we have created but for the way we got there" David Velez, founder and CEO of Nubank, writes about our commitment to having the best practices over social, environmental, and governance matters - including net-zero emissions. 23 Sep 20 Culture & Values Nubank neutralizes its carbon emissions and achieves net-zero target All the carbon we emit is offset through projects that neutralize CO2 in the environment. 12 Sep 20 About Us Nubank's financial statement: twice as many customers and BRL 19 billion cash Marcelo Kopel, Nubank's CFO, comments on the financial statement for the first half of 2020 - a period when we focused on supporting our more than 26 million customers during the pandemic and ensuring the well-being of our employees. 08 Sep 20 Culture & Values The Four Pillars of Customer Service at Nubank Yuri Dantas, Head of Customer Excellence, talks about how we developed the standards that help our team in their routine work - Anticipate, Solve, Care, Empower. 08 Sep 20 About Us What is Nubank? A company that is always on the side of their customers Get to know the essence, values, and history of the largest independent digital bank in the world. 04 Sep 20 Data science Now Nubank's Data Scientists have their own values It is not just a matter of collecting, analysing and interpreting data. Check out the six principles that build our data scientists' code. 04 Sep 20 Press Nubank, recognized by CB Insights as one of the fast-growing fintech startups The Latin American digital bank was selected from a pool of 16,000 companies from around the world, in 2020. 02 Sep 20 About Us Nubank's co-founder is recognized by Fortune as one of the world's most influential leaders under 40 years old Cristina Junqueira, co-founder of the largest independent digital bank in the world, is the only Brazilian featured in this year's Fortune 40 under 40 list. 21 Aug 20 Career & Life Stories What exactly does it mean to be a Business Architect? A quarter Product Manager, a quarter Business Analyst, a quarter Management Advisor and a quarter... something else? 21 Aug 20 Analytics More productive data analytics with Data Quality Governance Implementing a centralized data quality governance framework in a fast-growing, agile and data-hungry organization 14 Aug 20 Design How I redesigned my team's calendar to be more productive We decided to treat our rituals and calendar as a product. Testing, learning, and iterating with time. So every once in a while we move, delete, create things that suit our moment as a team, and make us more productive and focused. 14 Aug 20 Engineering & Tech Going agile: do less to deliver more How we decrease time needed to deliver our demands and increase the number of items completed per week 07 Aug 20 Culture & Values Nubank's real foundation: our culture and values Learn more about the pillars that guide our work and ensure we all have the same focus: doing what is best for our customers. 23 Jul 20 Working at Nubank Cognitect, creator of Clojure and Datomic, is now part of Nubank We are happy to welcome such talented engineers to our teams as we build together the next generation of financial products and continue supporting the open-source Clojure community. 09 Apr 20 Culture & Values How we are keeping Nubank's culture alive while working from home 7 steps that helped us keep our teams connected and healthy in a home-office regime. 23 Oct 19 Culture & Values What does Nubank culture have to do with customer focus? The values of a company are essential to ensure excellent customer service. 25 May 21 Engineering & Tech Nubank Credit Card from Product to Platform mindset Platformization has been a trend for a while, but how do we do this? How do you go from a product perspective to abstractions on a platform? We want to share with you a real case that we consider one of the greatest challenges taken by Nubank so far! 22 Jul 20 Analytics Nubank Data Analysts are now called Analytics Engineers Aligning expectations and positioning ourselves towards better productivity and data governance 30 Oct 20 Engineering & Tech The value of canonicity What happens when we run an engineering organization by constraining the number of tools in our toolbox "for the greater good"? 15 Jul 21 Analytics Scaling data analytics with software engineering best practices Analytical work can greatly benefit from software engineering standards in terms of quality and efficiency, here's how. 16 Jun 21 Data science Beyond prediction machines A huge part of Nubank's Data Science team has been recently focusing on leveraging causal inference along with traditional machine learning to solve some of our toughest problems. This post aims at explaining why causal inference is such a powerful tool to solve business problems with data. 02 Jun 21 About Us Nubank appoints Arturo Nunez as Chief Marketing Officer Former Apple, Nike, and NBA Exec. will lead global marketing efforts as NuBank expands, following the brand's recent repositioning 23 Apr 21 Engineering & Tech Technical Writing and the value of documentation Technical Writing team is responsible for supporting, teaching and helping engineers to create, maintain and publish useful, assertive and clear documentation. 09 Apr 21 Analytics Distributing the data team to boost innovation reliably Integrating Analytics Engineers into all cross-functional teams: a key success factor to scale a data platform efficiently and reliably 07 Apr 21 About Us Matt Swann is Nubank's new CTO With 25 years of experience leading technology companies, Matt Swann joins Nubank to continue accelerating our innovation pace. 11 Mar 21 Engineering & Tech A look into Nubank's tech hub in Berlin Our data infrastructure teams in Europe are instrumental in creating increasingly better products for our customers 08 Mar 21 Culture & Values Nubank agrees to have 50% of the leadership made up of women By 2025, the commitment is to hire 3,300 women and become the most egalitarian technology company in Latin America. 10 Feb 21 Engineering & Tech Scaling with Flutter A bit over a year ago, Alexandre Freire wrote an article about how we decided to standardize our mobile app development with Flutter, our early experiences with the platform, and why we decided to use it as our primary technology for cross-platform apps. In a recent interview with Nora Bezi from Bitrise, Alexandre, along with [...] 17 Nov 20 Culture & Values Coffee Break and MegaDemo: what the meetings with the entire Nubank team are like Waiting for the Covid-19 times to pass, we'd like to share these two Nubank events, designed to give employees visibility (in person!) of what is going on in the company. 27 Oct 20 Our offices Behind the scenes: get to know Nubank headquarters in Sao Paulo and beyond Our workspaces reflect the values of the company: being efficient and thinking and acting like owners. (These are the environments we miss so much during our home office phase in 2020.) 30 Sep 20 About Us Nubank takes next step in international expansion and launches in Colombia With more than 30 million customers across Brazil and Mexico, Nubank reinforces its positioning as leader of digital banking in Latin America. 23 Sep 20 About Us "We want to be remembered not only for the revolution we have created but for the way we got there" David Velez, founder and CEO of Nubank, writes about our commitment to having the best practices over social, environmental, and governance matters - including net-zero emissions. 23 Sep 20 Culture & Values Nubank neutralizes its carbon emissions and achieves net-zero target All the carbon we emit is offset through projects that neutralize CO2 in the environment. 12 Sep 20 About Us Nubank's financial statement: twice as many customers and BRL 19 billion cash Marcelo Kopel, Nubank's CFO, comments on the financial statement for the first half of 2020 - a period when we focused on supporting our more than 26 million customers during the pandemic and ensuring the well-being of our employees. 08 Sep 20 Culture & Values The Four Pillars of Customer Service at Nubank Yuri Dantas, Head of Customer Excellence, talks about how we developed the standards that help our team in their routine work - Anticipate, Solve, Care, Empower. 08 Sep 20 About Us What is Nubank? A company that is always on the side of their customers Get to know the essence, values, and history of the largest independent digital bank in the world. 04 Sep 20 Data science Now Nubank's Data Scientists have their own values It is not just a matter of collecting, analysing and interpreting data. Check out the six principles that build our data scientists' code. 04 Sep 20 Press Nubank, recognized by CB Insights as one of the fast-growing fintech startups The Latin American digital bank was selected from a pool of 16,000 companies from around the world, in 2020. 02 Sep 20 About Us Nubank's co-founder is recognized by Fortune as one of the world's most influential leaders under 40 years old Cristina Junqueira, co-founder of the largest independent digital bank in the world, is the only Brazilian featured in this year's Fortune 40 under 40 list. 21 Aug 20 Career & Life Stories What exactly does it mean to be a Business Architect? A quarter Product Manager, a quarter Business Analyst, a quarter Management Advisor and a quarter... something else? 21 Aug 20 Analytics More productive data analytics with Data Quality Governance Implementing a centralized data quality governance framework in a fast-growing, agile and data-hungry organization 14 Aug 20 Design How I redesigned my team's calendar to be more productive We decided to treat our rituals and calendar as a product. Testing, learning, and iterating with time. So every once in a while we move, delete, create things that suit our moment as a team, and make us more productive and focused. 14 Aug 20 Engineering & Tech Going agile: do less to deliver more How we decrease time needed to deliver our demands and increase the number of items completed per week 07 Aug 20 Culture & Values Nubank's real foundation: our culture and values Learn more about the pillars that guide our work and ensure we all have the same focus: doing what is best for our customers. 23 Jul 20 Working at Nubank Cognitect, creator of Clojure and Datomic, is now part of Nubank We are happy to welcome such talented engineers to our teams as we build together the next generation of financial products and continue supporting the open-source Clojure community. 09 Apr 20 Culture & Values How we are keeping Nubank's culture alive while working from home 7 steps that helped us keep our teams connected and healthy in a home-office regime. 23 Oct 19 Culture & Values What does Nubank culture have to do with customer focus? The values of a company are essential to ensure excellent customer service. 23 Jul 20 Engineering & Tech Nubank and Cognitect: a personal perspective Edward Wible, Nubank's CTO, shares his perspective on the future of Clojure and Datomic, and how the powerful ideas that guide these technologies helped shape Nubank's culture and business. * Explore + Nu Website + Blog Brazil + Blog Colombia + Blog Mexico + Privacy Policy Nu Pagamentos S.A. 18.236.120/0001-58 Rua Capote Valente, 39 - 05409-000 - Sao Paulo, SP.