https://martinfowler.com/articles/cant-buy-integration.html [mf-name-wh] [ ] * Refactoring * Agile * Architecture * About * Thoughtworks * * [ ] Topics Architecture Refactoring Agile Delivery Microservices Data Testing DSL about me About Books FAQ content Videos Content Index Board Games Photography Thoughtworks Insights Careers Products follow Twitter RSS Contents You can't buy integrationPut most of your energy into building clean interfacesUse a general purpose language to manage the interface evolution * General purpose languages excel at programming over time * Integration tools "think" in terms of implementation You Can't Buy Integration Commercial integration tools are a couple decades old now, but there has been little in the way of overarching architectural principles describing when and how to use them. In this article, I argue that "buy" decision mechanics have caused us to exaggerate the value proposition of such tools, often leading to mandates to use a certain integration tool over a general purpose language. I claim that such tools thrive in a world that views integration as primarily about connecting systems, but that digital organizations have reimagined integration to be primarily about putting clean interfaces in front of digital capabilities, emphasizing capabilities over systems. Finally, I list some of the key principles behind a modern view of integration and claim they are best managed with a general purpose language, reorienting the primary value proposition of commercial integration tools towards simplifying tactical implementation concerns. 07 December 2021 --------------------------------------------------------------------- Photo of Brandon Byars Brandon Byars Brandon Byars is Head of Technology for Thoughtworks North America, where he helps organizations navigate their digital transformation journeys. On the side, he enjoys live Dave Matthews Band jams, long Neal Stephenson novels, and other artistic creations that show that, sometimes, more is more. application integration enterprise architecture Contents * You can't buy integration * Put most of your energy into building clean interfaces * Use a general purpose language to manage the interface evolution + General purpose languages excel at programming over time + Integration tools "think" in terms of implementation --------------------------------------------------------------------- In the early days of computing, vendors sold software, including compilers and operating systems, as part of the hardware they ran on. That changed in 1974, when the US Commission on New Technological Uses of Copyrighted Works (CONTU) decided that computer programs were subject to copyright, creating a market for what were initially called "program products." Despite the resistance movement of the Free Software Foundation and open source, there was, and is, a clear market for commercial software products. "Build versus buy" decisions are everywhere today, and rightly so. Building software is risky and expensive, and software product companies can spread that risk and expense across multiple customers. However, as you may have guessed by the title of this article, such decisions don't apply to all contexts. You can't buy integration Despite a wide range of tools that aim to simplify wiring systems together, you can't buy integration. You can buy programming languages. After the 1974 CONTU ruling, it became common to pay for the compiler. Bill Gates' famous Open Letter To Hobbyists was a clarion call for the community to pay for Micro-Soft's Altair BASIC interpreter (they dropped the dash in later years). The Free Software Foundation's GCC compiler opened the door to the commoditization of programming languages but left open a commercial market for developer tooling. I'm happy to program in Java for example -- now freely available -- but I would not be excited to do so in vi or Notepad. Integration software products -- ESBs, ETL tools, API platforms, and cloud integration services -- are not products that directly solve a business problem. They are not in the same category, for example, as fraud detection products or analytics products or CRMs. They are programming languages, bundled with a toolchain and a runtime to support the compilation process. When you buy an integration product, you are agreeing to build the integration itself in a commercial programming language. Integration tools are almost always low-code platforms, which means they aim to simplify the development effort by providing a graphical design palette you can drag and drop integration workflow on top of. The source code is typically saved in a markup language that can be interpreted by the runtime. You might drag and drop some workflow onto a palette, but underneath the hood, the platform saves the source code as JSON or XML, and embeds a runtime that knows how to interpret the markup into actual machine code, no different than Micro-Soft's early compiler knew how to convert BASIC code into machine code on the Altair platform. For example, here is the "Hello, World" source code for Step Functions, an AWS orchestration engine: [step-funct] Figure 1: Step Functions represents a workflow with both JSON and graphical design palette Many integration tools, including AWS Step Functions, let you program using either the graphical palette or the markup language directly. While the palette is often preferred for reasons obvious to anyone who read Charles Petzold's famous April Fools joke about CSAML, the complexity of configuring integration steps in the palette means that, in practice, competent developers gain some facility with the underlying markup language itself. In effect, there is a bidirectional mapping from the graphical palette to the markup language such that changing one can immediately be reflected in the other. If I've understood the vernacular of mathematics correctly, that's what's called an isomorphism, so I'll call the resulting structure "source-diagram isomorphism," where both the palette and the markup language represent source code and can be seamlessly translated back and forth. That of course represents a developer-centric view of the world; the runtime itself only cares about the markup language. This is quite different from most software programming, where the developer directly edits the source code absent a graphical palette, a practice I'll call "source endomorphism," although you can also call it "normal" if that's easier to remember. There are tools, of course, that visualize class diagrams in Java and perhaps even let you make edits that are reflected back in the source code, but the usual activity of a Java developer is to directly edit Java source code in an IDE. The advantage of providing a graphical design palette is that it provides a way of organizing thought, a domain specific language (DSL) for integration problems, allowing you to focus on the narrow problem of wiring systems together absent extraneous complexity. Java may be better at solving general purpose problems, but the constraints of the design palette and declarative markup language purport to solve integration and workflow concerns more elegantly, in the same way that Excel functions let you solve a budgeting problem more elegantly than writing custom Java code. Similarly, in a number of contexts, I'd much prefer the calculator on my iPhone over the impressive HP 50g graphic calculator, with its support for Reverse Polish Notation and scientific calculations. [calculator] Figure 2: A good DSL removes complexity by focusing on the core problem When you buy integration tools, you are agreeing to build the actual integration itself. What you are buying is a promise that the integration can be solved more efficiently and more simply than using a general purpose language. The job of the architect then comes down to understanding in what contexts that promise is likely to hold true, and to avoid the understandable temptation to convert the "buy" decision into a mandate to use the tool outside of those contexts in order to justify its ROI. Some integration DSLs are simpler projections of the problem space, like my iPhone calculator. Others are indeed Turing complete, meaning, in a theoretical sense, they have the same algorithmic power as a general purpose language. While true, academic discussions of computability fail to account for software engineering, which a group of Googlers defined as "programming over time." If programming requires working with abstractions, then programming over time means evolving those abstractions in a complex ecosystem as the environment changes, and requires active consideration of team agreements, quality practices, and delivery mechanics. We'll examine how programming-over-time concerns affect integration in more detail shortly and how they inform the appropriate contexts for low-code integration tools. First, though, I want to challenge the idea that the primary goal of integration is wiring systems together, as I believe a broader definition allows us to better segregate the parts of the ecosystem where simplifying abstractions facilitate programming and where the additional complexity of programming-over-time concerns requires a general purpose programming language, a claim I'll defend shortly. Put most of your energy into building clean interfaces For most people, the word "integration" creates the impression of connecting systems together, of sharing data to keep systems in sync. I believe that definition of integration is insufficient to meet the demands of a modern digital business, and that the real goal of integration done well is to create clean interfaces between capabilities. When our primary focus is connecting systems, we can measure how successful our integration approach is by how quickly we can wire a new system into an existing technical estate. The systems become the primary value driver inside that estate, and integration becomes a necessary evil to make the systems behave properly. When instead we shift our primary focus to creating clean interfaces over digital capabilities, we measure success by increasing digital agility over time, and those digital capabilities become the primary value driver, arguably even more important than the systems themselves. There's a lot to unpack in that difference, starting with the emphasis on interface over implementation. Digital organizations shift the primary focus of integration from the systems to the capabilities, emphasizing clean interfaces over those capabilities. Simplifying interfaces are one of the critical elements in creating a successful product and to scaling inside a complex ecosystem. I have very little understanding of the mechanical-electrical implementation underlying the keyboard I'm typing on, for example, or the input system drivers or operating system interrupts that magically make the key I'm typing show up on my screen. Somebody had to figure that all out -- many somebodies, more likely, since the keyboard and system driver and operating system and monitor and application are all separate "products" -- but all I have to worry about is pressing the right key at the right time to integrate the thoughts in my brain to words on the screen. That, of course, has an interesting corollary: the key (no pun intended) to simplifying the interface is to accept a more complex implementation. There is nothing controversial about that statement when we think of digital products that face off with the market. Google search is unimaginably complex underneath the hood and uncannily easy for even a digitally unsavvy user to use. We also accept it for digital products that face off with business users. The sales team excited about bringing in Salesforce surely understands that, while the user interface may be more intuitive for their needs than the older CRM, it requires a significant amount of effort to maintain and evolve the product itself, which is why the subscription fees feel justifiable. Yet we treat integration differently. Intuitively, we understand that the two-dimensional boxes on our architecture diagrams may hide considerable complexity, but expect the one-dimensional lines to be somehow different. (They are different in one regard. You can buy the boxes but you can't buy the lines, because you can't buy integration.) While we have historically drawn up our project plans and costs around the boxes -- the digital products we are introducing -- the lines are the hidden and often primary driver of organizational tech debt. They are the reason that things just take longer now than they used to. [spaghetti-] Figure 3: We think of projects in terms of the applications they introduce, but the lines between those applications become the critical cost driver over time Simplifying that glue code is certainly a noble effort, and integration tools can help, but not at the expense of building clean interfaces over capabilities. Importantly, the only effective judges of how easy an interface is to use are the actual users of it. Google could have asked us for more information to make their search implementation easier -- geographical, recency, and popularity information, for example -- but instead they offered only a single text box to type a search in and had to learn how to apply those factors into their algorithm. The same concern applies to API design (which I define broadly to include synchronous calls and asynchronous events). Clean interfaces hide implementation details, and one of those implementation details in integration contexts is the choice of programming language. I have yet to see an architecture diagram that puts the primary focus on the programming languages of the systems involved: [languages-] Figure 4: Emphasizing the implementation languages in architecture diagrams is unusual Yet I have seen all too many variations of diagrams that do exactly that for integration. Such a view reinforces a tactical understanding of integration as wiring systems together, as it emphasizes the wiring toolchain instead of the digital capabilities. [integratio] Figure 5: Showing the commercial integration tool in an architecture diagram puts the emphasis on implementation details instead of interfaces and treats integration as a tactical concern Another implementation detail our API consumers would be happy to not care about is which systems the data comes from. Outside of the business users who work in SAP and the IT staff surrounding them, nobody in your organization should have to care about the quirks of the SAP system. They only care about how to get access to customer data or how to create an order. That observation is worth calling out separately, as it is one of the most commonly violated principles I see in integration strategies, and one of the strongest indicators of an implicit philosophy of integration as wiring systems together instead of creating clean interfaces over digital capabilities. You don't need an SAP API, because your API users don't care about SAP, but you might need an order management API. Abstract the capability, not the system. Your users don't stand still, and quite often good APIs add value through reuse. It's easy to over-index on reuse as a primary goal of APIs (I believe taming complexity is a more important goal) but it's still a useful aspiration. Keeping up with your users' evolving needs means breaking previous assumptions, a classic programming-over-time concern. Carrying on with my previous metaphor, the job of a keyboard is to seamlessly integrate its users thoughts into on-screen text. As a native English speaker, I've never had to struggle with the Pinyin transliteration that native Chinese speakers have to, but for several years I unnecessarily tortured myself by typing in the Colemak keyboard layout. Because my physical keyboard was incapable of magically adapting to the software layout, there was an impedance mismatch between the letters on the keyboard and what showed up on screen. Normally, that's not a problem: as a (not particularly fast) touch typist, I'm used to not looking at the keyboard. However, that impedance mismatch made the learning process painfully difficult as I constantly had to look at an on-screen mapping to QWERTY and look down at the keys while my brain worked through the resultant confusion. I'm sure there are keyboards out there that are backlit and project the letter on the physical key in consonance with the keyboard layout. The price of that improved interface, of course, is more implementation complexity, and that evolution is a programming-over-time concern. Integration interfaces that fail to adapt to users over time, or that change too easily with the underlying systems for implementation convenience, are point-in-time integrations, which are really just point-to-point integrations with multiple layers. They may wear API clothing, but show their true stripes every time a new system is wired into the estate and the API is duplicated or abused to solve an implementation problem. Point-in-time integrations add to inter-system tech debt. Treating integration as primarily about systems results in a landscape littered with point-in-time integrations, decreasing organizational agility. Of course, your creaking systems of record will resist any attempt to put them in a box. The ERP was specifically designed to do everything, so trying to externalize a new capability that still has to integrate with the ERP will be a challenge. It can require significant architectural skill to contain the resulting integration complexity and to hide it from the user, but the alternative is to increase your organizational tech debt, adding another noodle to the spaghetti mess of point-to-point or point-in-time integrations. The only way I'm aware of to pay that tech debt down is to hold the line on creating a clean interface for your users and create the needed transformations, caching, and orchestration to the downstream systems. If you don't do that, you are forcing all users of the API to tackle that complexity, and they will have much less context than you. We need to invert the mindset, from thinking of how to solve integration problems with our tools to instead thinking of how to build the right interfaces to maximize agility. Use a general purpose language to manage the interface evolution Many commercial integration tools market their ability to own the integration landscape and call out to general purpose languages as needed. While I can appreciate the marketing behind such messaging -- it promotes product penetration and lock-in -- as architectural guidance, it is exactly backwards. Instead, we should almost always manage the interface evolution in a general purpose language for at least two reasons: so we can better manage the complexity of maintaining a clean interface, and so that we avoid the gravitational pull of our tool's mental model when making strategic integration decisions. General purpose languages excel at programming over time Programming over time means making changes to source code over time, and this is one area where source-diagram isomorphism pales in comparison to normal development. The ability to "diff" changes between source code commits is a developer superpower, an invaluable debugging technique to understand the source of a defect or the context behind a change. Diffing the markup source code language of an integration tool is much harder than diffing Java code for at least three reasons: modularity, syntax, and translation. Normally, the developer is in charge of the modularity of the source code. It is of course possible to throw all logic into a single file in Java -- the classic God object -- but competent developers create clean boundaries in an application. Because they edit the textual source code directly, those module boundaries of the language correspond to filesystem boundaries. For example, in Java, packages correspond to directories and classes to files. A source code commit may change a number of lines of code, but those lines are likely to be localized to natural boundaries in the code that the team understands. With integration DSLs, the design palette has some control over the modularity of the underlying textual source code, the price you pay for source-diagram isomorphism. It is not uncommon to create, for example, the entire workflow in one file. Similarly the markup language itself may consist of syntax that makes diffing harder. The good news is that the tools I've looked at do a good job of "pretty printing" the markup language, which adds line endings to make diffing easier. However, structural changes in a workflow are still more likely to cause, for example, a re-ordering of elements in the markup language, which will make a diff show many more lines of code changed than such an operation might intuitively warrant. Additionally, some languages, XML in particular, add a significant amount of noise, obscuring the actual logic change. Finally, because you are programming at a higher level of abstraction in integration DSLs, you have a two step process to examine a diff. First, as you would with Java, you have to understand the changed lines in the context of the commit itself. With Java, since that source code is the same source code you edit, the understanding stops there. With an integration DSL, you have to make the additional mental leap of understanding what those changed lines of markup mean to the overall workflow, effectively mentally mapping them to what you would see on the design palette. The delta between source code commits can only be represented textually; graphical palettes are not designed to represent change over time. The net effect of all of this is to increase the cognitive load on the developer. Gregor Hohpe has a brilliant story demonstrating the debuggability shortcomings of low code platforms. In The Software Architect Elevator, he describes his experience when vendors shop their wares at his company. Once they've shown how simple it is to drag and drop a solution together, he asks the technical sales person if she could leave the room for two minutes while Gregor tweaks something randomly in the underlying markup language so he could then see how she debugs it when she comes back in. So far, at least as of the publication of the book, no vendor has taken him up on his offer. Commercial integration DSLs also make it harder to scale development within the same codebase. Not only is it harder to understand the context of changes over time for a single source file, it's also harder to have multiple developers edit the same source file in parallel. This isn't pain-free in a general purpose language, but is made possible by direct developer control over the modularity of the source code, which is why you rarely see teams of only one or two Java developers. With integration DSLs, given the constraints of source code modularity and the additional mental leap it takes to understand the source code -- the markup source itself and the graphical workflow abstractions they represent -- merging is considerably more painful. With such tools, it is quite common to constrain parallel development on the same codebase, and instead break the problem down into separate components that can be developed in parallel. Programming over time requires advanced testing and environment promotion practices. Many integration tool vendors go out of their way to demonstrate their support for such practices, but once again, it is an inferior developer experience. Each test run, for example, will require spinning up the runtime that interprets the XML source code into machine code. In practical terms, that friction eliminates the possibility of short test driven development "red, green, refactor" feedback loops. Additionally, you will likely be limited to the vendor's framework for any type of unit testing. The ecosystems with general purpose programming languages evolve at a rapid clip. Advances in testing tools, IDEs, observability tools, and better abstractions benefit from the sheer scale of the community such languages operate in. Low-code platforms have much smaller ecosystems, limiting the ability to advance at the same pace, and the platform constraints will almost certainly force developers to use toolchains provided by the vendor to write and test code. That naturally has implications for security concerns like supply chain and static analysis scans. Such tooling gets a lot of attention for, say, Java open source libraries, but far less attention in the walled gardens of the low-code world. Finally, integration tools offer comparatively impoverished operational support in their runtimes. Whereas observability tooling and resiliency patterns get a lot of attention for general purpose programming languages and the platforms that support them, those are not the main focus of integration tools. I've seen multiple large-scale adoptions of low code integration tools result in considerable performance concerns, a problem that grows worse over time. It is usually addressed initially by additional licensing costs, until that too becomes prohibitive. Unfortunately, by that point, there is significant platform lock-in. Low-code tools are insufficient to handle the same type of complexity that general purpose programming languages can handle. A colleague of mine described a contentious environment where he was dealing with a mandate to use TIBCO BusinessWorks, a well-known commercial integration tool. He challenged the TIBCO team to a bake-off: he would send his best Java / Spring developer to create an integration to another COTS product's web services -- SOAP interfaces coded in Apache Axis -- and they could bring their best TIBCO developers to do the same. The Java developer had a working implementation by lunch. The TIBCO team discovered that the tool did not support the older version of Apache Axis used by the COTS product, the type of legacy complexity common in large enterprises. Following the mandate would have meant going back to the vendor and changing their roadmap or adding an extension in a general programming language. Fred Brooks called such extensions "accidental complexity" in his famous No Silver Bullet essay: they add complexity due to the choice of solution, and have nothing to do with the problem. Every mandate to use low-code tools for all integration will accrue significant accidental complexity. Even more concerning than the accidental complexity needed to run all integration through commercial tooling, though, is the way such a mandate puts the emphasis on implementation over interface, on systems over capabilities. Integration tools "think" in terms of implementation Integration tools were created, and continue to thrive today, because of the complexity of unlocking data and capabilities across the spectrum of IT systems. Your actual customer master data may reside within, for example, SAP, but the early part of a customer's lifecycle exists in a Siebel CRM. The IBM mainframe system still handles core billing for some customers; an Oracle ERP for others. Now the business wants to replace Siebel with Salesforce. The business team bringing in a new product naturally understands that it will take some time to get the configuration right for adapting it to their sales intake process, but the last thing any of them want is to be told of long IT timelines just to sort out the glue between systems. It's SaaS, after all! Traditionally, those long timelines were the result of point-to-point integration, which did not allow for learning. Every new wire between systems meant teams had to re-learn how to connect, how to interpret the data, how to route between systems, and so on. Integration tools broke the problem down into smaller pieces, some of which could be reused, especially the connectivity into systems. Take a look at some of the actions available on the AWS Step Functions palette we looked at earlier: [step-funct] Figure 6: Each step in an AWS Step Functions workflow describes an implementation concern Step Functions describes all of the actions in terms of some action on some AWS service. You can configure each box in the workflow to describe, for example, the DynamoDB table name, allowing you to focus on the overall flow in the main part of the palette. While Step Functions is a relatively new integration tool with an obvious bias towards cloud native AWS services, all integration tools that I'm familiar with tend to work along similar lines with their focus on implementation concerns. The early on-prem equivalents for application integration were enterprise service buses (ESBs), which separated out system connectivity as a reusable component from orchestration and routing. You can see that separation in a simplified view of Mulesoft's ESB, so named because it aimed to remove the "donkey work" of integration: [mulesoft-e] Figure 7: ESBs separate connectivity from orchestration and routing There were some natural false starts in the ESB world as the industry aspired to have enterprise-wide canonical formats on the bus, but all of them shared the notion of adapters to the inputs and outputs of the bus -- the systems being integrated. In the happy path, you could describe your integration in a language like BPEL, which could provide a graphical design palette and source-diagram isomorphism as it described the process in XML. The industry has largely moved on from ESBs, but you can see their heritage in modern API platforms. Take a look, for example, at Mulesoft's three layer API architecture: [three-laye] Figure 8: Mulesoft's three layer architecture maintains the separation of connectivity with experience and system APIs Mulesoft sells both an API management platform and a low-code runtime for building APIs. You can and often should buy middleware infrastructure, and it is entirely possible to divorce the API gateway from the runtime, proxying to APIs built in a general purpose programming language. If you do so, the question arises: would you use Mulesoft's three layer architecture if you built all of the APIs outside the Mulesoft runtime? I quite like the idea of experience APIs. The name is less jargony than the one that's caught on in the microservice community -- backends for frontends -- although I prefer the term "channel API" over both as it more obviously covers a broader range of concerns. For example, narrowing access to core APIs in a B2B scenario is clearly a channel concern, less obviously an "experience" or "frontend" concern. Whatever the name, providing an optimized channel-specific API is a valuable pattern, one that allows the channel to evolve at a different rate than the underlying capabilities and to narrow the surface area for attackers. I'm less excited about the prescriptive separation between process and system APIs because of their focus on implementation over interface: the system layer focuses on connectivity and the process layer focuses on orchestration [1]. I've redrawn their simplified ESB picture above to show that the similarity on implementation concerns to connect systems is hard to overlook: [api-esb] Figure 9: The three layer architecture emphasizes implementation details, showing its ESB heritage Part of the value proposition of a platform like Mulesoft -- both its ESB and API runtime -- lies in the built in library of connectors to systems like SAP and Salesforce, connectors that can save you time at the edges of the system (especially the system layer). The three layer architecture simplifies use of those connectors and separates orchestration and aggregation to encourage their reuse. Conceptually, the three layer architecture serves to constrain designing APIs such that they fit inside Mulesoft's ESB heritage. In theory, the architecture allows more reuse across layers. In practice, you are limited by programming-across-time concerns of evolving process APIs to multiple consumers. In fact, I have seen many APIs that are not APIs at all, but rather ETL in API clothing, with the system layer managing the extract, the process layer managing the transform, and the experience layer managing the load. That should not be surprising, because integration tools think in terms of implementation. The allure of buying integration tools is that they make the tactical concern of wiring systems together cheaper, avoiding the usual expense and risk of custom software. Unfortunately, when we frame the problem space that way, we have allowed our tools to think for us. We're releasing this article in installments. The next installment will examine where commercial integration tools can add considerable value. To find out when we publish the next installment subscribe to the site's RSS feed, Brandon's twitter stream, or Martin's twitter stream --------------------------------------------------------------------- Footnotes 1: My main critique against Mulesoft's model is the emphasis on implementation concerns, as I believe that leads to viewing integration as a tactical concern. Praful Todkar and Ryan Murray argue for a superficially similar model in their series on building a well-factored service architecture. While I think the line between foundational capabilities and business capabilities in their model is quite fuzzy in practice, I appreciate their emphasis on classification over architectural layering and interface over implementation. Both Mulesoft's three layer architecture and Ryan and Praful's three classifications of services are useful models to think about the right ways to decompose services for composability, but I believe we get significantly more composability by focusing on digital capabilities instead of focusing on implementation concerns like orchestration and connectivity. Acknowledgements Many thanks to everyone who helped me with their critical feedback along the way, especially Ed Wilson, Rebecca Parsons, Martin Fowler, Sina Jahan, Kief Morris, Peter Gillard-Moss, Bruna Gonclaves, Ed Mangini, Ian Cartwright, Srinivasan Raguraman, Chris Ford, Charith Tangirala, Ken Collier, and Matteo Vaccari. Their feedback was essential to highlighting assumptions I otherwise failed to make explicit in the overall argument and the article is better off thanks to their review. Any inaccuracies are, of course, my own fault. Significant Revisions 07 December 2021: Published "Use a general purpose language to manage the interface evolution" 01 December 2021: Published "Put most of your energy into building clean interfaces" 30 November 2021: Published first section [ ] Topics Architecture Refactoring Agile Delivery Microservices Data Testing DSL about me About Books FAQ content Videos Content Index Board Games Photography Thoughtworks Insights Careers Products follow Twitter RSS [thoughtwor] (c) Martin Fowler | Privacy Policy | Disclosures