[HN Gopher] Why Twilio Segment moved from microservices back to ...
___________________________________________________________________
Why Twilio Segment moved from microservices back to a monolith
Author : birdculture
Score : 70 points
Date : 2025-12-13 20:30 UTC (2 hours ago)
(HTM) web link (www.twilio.com)
(TXT) w3m dump (www.twilio.com)
| develatio wrote:
| can you add [2018] to the title, please?
| pmbanugo wrote:
| have they reverted to microservices?
| Towaway69 wrote:
| Mono services in a micro repository. /s
| yieldcrv wrote:
| I feel like microservices have gotten a lot easier over the last
| 7 years from when Twilio experienced this, not just from my
| experience but from refinements in architectures
|
| There are infinite permutations in architecture and we've
| collectively narrowed them down to things that are cheap to
| deploy, automatically scale for low costs, and easily replicable
| with a simple script
|
| We should be talking about how AI knows those scripts too and can
| synthetize adjustments, dedicated Site Reliability Engineers and
| DevOps is great for maintaining convoluted legacy setups, but
| irrelevant for doing the same thing from scratch nowadays
| moltar wrote:
| Do you have any recommended reading on the topic of refinements
| in architectures? Thank you.
| eYrKEC2 wrote:
| You know what I think is better than a push of the CPU stack
| pointer and a jump to a library?
|
| A network call. Because nothing could be better for your code
| than putting the INTERNET into the middle of your application.
|
| --
|
| The "micro" of microservices has always been ridiculous.
|
| If it can run on one machine then do it. Otherwise you have to
| deal with networking. Only do networking when you have to. Not
| as a hobby, unless your program really is a hobby.
| ikiris wrote:
| Not everything you think you know is right.
|
| https://github.com/sirupsen/napkin-math
| josephg wrote:
| Well implemented network hardware can have high bandwidth
| and low latency. But that doesn't get around the complexity
| and headaches it brings. Even with the best fiber optics,
| wires can be cut or tripped over. Controllers can fail.
| Drivers can be buggy. Networks can be misconfigured. And so
| on. Any request - even sent over a local network - can and
| will fail on you eventually. And you can't really make a
| microservice system keep working properly when links start
| failing.
|
| Local function calls are infinitely more reliable. The main
| operational downside with a binary monolith is that a bug
| in one part of the program will crash the whole thing.
| Honestly, I still think Erlang got it right here with
| supervisor trees. Use "microservices". But let them all
| live on the same computer, in the same process. And add
| tooling to the runtime environment to allow individual
| "services" to fail or get replaced without taking down the
| rest of the system.
| yieldcrv wrote:
| it's not really "micro" but more so "discreet" as in special
| purpose, one off. to ensure consistent performance, as
| opposed to shared performance.
|
| yes, networking is the bottleneck between the processes,
| while one machine is the bottleneck to end users
| Nextgrid wrote:
| > one machine is the bottleneck to end users
|
| You can run your monolith on multiple machines and round-
| robin end-user requests between them. Your state is in the
| DB anyway.
| AndrewKemendo wrote:
| > Microservices is a service-oriented software architecture in
| which server-side applications are constructed by combining many
| single-purpose, low-footprint network services.
|
| Gonna stop you right there.
|
| Microservices have nothing to do with the hosting or operating
| architecture.
|
| Martin Fowler who formalized the term, Microservices are:
|
| "In short, the microservice architectural style is an approach to
| developing a single application as a suite of small services,
| each running in its own process and communicating with
| lightweight mechanisms, often an HTTP resource API. These
| services are built around business capabilities and independently
| deployable by fully automated deployment machinery"
|
| You can have an entirely local application built on the
| "microservice architectural style."
|
| Saying they are "often HTTP and API" is besides the point.
|
| The problem Twilio actually describe is that they messed up
| service granularity and distributed systems engineering processes
|
| Twilio's experience was not a failure of the microservice
| architectural style. This was a failure to correctly define
| service boundaries based on business capabilities.
|
| Their struggles with serialization, network hops, and complex
| queueing were symptoms of building a distributed monolith, which
| they finally made explicit with this move. So they accidentally
| built a system with the overhead of distribution but the tight
| coupling of a single application. Now they are making their
| foundations of architecture fit what they built, likely cause
| they poorly planned it.
|
| The true lesson is that correctly applying microservices requires
| insanely hard domain modeling and iteration and meticulous
| attention to the "Distributed Systems Premium."
|
| https://martinfowler.com/microservices/
| MrDarcy wrote:
| Reading it with hindsight, their problems have less to do with
| the technical trade off of micro or monolith services and much
| more to do with the quality and organizational structure of their
| engineering department. The decisions and reasons given shine a
| light on the quality. The repository and test layout shine a
| light on the structure.
|
| Given the quality and the structure neither approach really
| matters much. The root problems are elsewhere.
| CharlieDigital wrote:
| My observation is that many teams lack strong "technical
| discipline"; someone that says "no, don't do that", makes the
| case, and takes a stand. It's easy to let the complexity genie
| out of the bottle if the team doesn't have someone like this
| with enough clout/authority to actually make the team pause.
| mjr00 wrote:
| > Once the code for all destinations lived in a single repo, they
| could be merged into a single service. With every destination
| living in one service, our developer productivity substantially
| improved. We no longer had to deploy 140+ services for a change
| to one of the shared libraries. One engineer can deploy the
| service in a matter of minutes.
|
| If you _must_ to deploy every service because of a library
| change, you don 't have services, you have a distributed
| monolith. The entire idea of a "shared library" which must be
| kept updated across your entire service fleet is antithetical to
| how you need to treat services.
| wowohwow wrote:
| I think your point while valid, it is probably a lot more
| nuanced. From the post it's more akin to an Amazon shared build
| and deployment system than "every library update needs to
| redeploy every time scenario".
|
| It's likely there's a single source of truth where you pull
| libraries or shared resources from, when team A wants to update
| the pointer to library-latest to 2.0 but the current reference
| of library-latest is still 1.0, everyone needs to migrate off
| of it otherwise things will break due to backwards
| compatibility or whatever.
|
| Likewise, if there's a -need- to remove a version for a
| vulnerability or what have you, then everyone needs to
| redeploy, sure, but the centralized benefit of this likely
| outweighs the security cost and complexity of tracking the
| patching and deployment process for each and every service.
|
| I would say those systems -are- and likely would be classified
| as micro services but from a cost and ease perspective operate
| within a shared services environment. I don't think it's fair
| to consider this style of design decision as a distributed
| monolith.
|
| By that level of logic, having a singular business entity vs
| 140 individual business entities for each service would mean
| it's a distributed monolith.
| mjr00 wrote:
| > It's likely there's a single source of truth for where you
| pull libraries or shared resources from, when team A wants to
| update the pointer to library-latest to 2.0 but the current
| reference of library-latest is still 1.0, everyone needs to
| migrate off of it otherwise things will break due to
| backwards compatibility or whatever.
|
| No, this misses one of the biggest benefits of services; you
| explicitly _don 't_ need everyone to upgrade library-latest
| to 2.0 at the same time. If you _do_ find yourself in a
| situation where you can 't upgrade a core library like e.g.
| SQLAlchemy or Spring, or the underlying Python/Java/Go/etc
| runtime, without requiring updates to _every_ service, you
| are back in the realm of a distributed monolith.
| wowohwow wrote:
| I disagree. Both can be true at the same time. A good
| design should not point to library-latest in a production
| setting, it should point to a stable known good version via
| direct reference, i.e library-1.0.0-stable.
|
| However, the world we live in, people choose pointing to
| latest, to avoid manual work and trust other teams did the
| right diligence when updating to the latest version.
|
| You can point to a stable version in the model I described
| and still be distributed and a micro service, while
| depending on a shared service or repository.
| vlovich123 wrote:
| You can do that but you keep missing that you're no
| longer a true microservice as originally defined and
| envisioned, which is that you can deploy the service
| independently under local control.
|
| Can you imagine if Google could only release a new API if
| all their customers simultaneously updated to that new
| API? You need loose coupling between services.
|
| OP is correct that you are indeed now in a weird hybrid
| monolith application where it's deployed piecemeal but
| can't really be deployed that way because of tightly
| coupled dependencies.
|
| Be ready for a blog post in ten years how they broke
| apart the monolith into loosely coupled components
| because it was too difficult to ship things with a large
| team and actually have it land in production without
| getting reverted to an unrelated issue.
| 3rodents wrote:
| Yes, you're describing a distributed monolith. Microservices
| are independent, with nothing shared. They define a public
| interface and that's it, that's the entire exposed surface
| area. You will need to do major version bumps sometimes, when
| there are backwards incompatible changes to make, but these
| are rare.
|
| The logical problem you're running into is exactly why
| microservices are such a bad idea for most businesses. How
| many businesses can have entirely independent system
| components?
|
| Almost all "microservice" systems in production are
| distributed monoliths. Real microservices are incredibly
| rare.
|
| A mental model for true microservices is something akin to
| depending on the APIs of Netflix, Hulu, HBO Max and YouTube.
| They'll have their own data models, their own versioning
| cycles and all that you consume is the public interface.
| wowohwow wrote:
| This type of elitist mentality is such a problem and such a
| drain for software development. "Real micro services are
| incredibly rare". I'll repeat myself from my other post, by
| this level of logic nothing is a micro service.
|
| Do you depend on a cloud provider? Not a microservice. Do
| you depend on an ISP for Internet? Not a microservice.
| Depend on humans to do something? Not a microservice.
|
| Textbook definitions and reality rarely coincide, rather
| than taking such a fundamentalist approach that leads
| nowhere, recognize that for all intents and purposes, what
| I described is a microservice, not a distributed monolith.
| AndrewKemendo wrote:
| And if my grandmother had wheels she would be a bike
|
| There are categories and ontologies are real in the
| world. If you create one thing and call it something else
| that doesn't mean the definition of "something else"
| should change
|
| By your definition it is impossible to create a state
| based on coherent specifications because most states
| don't align to the specification.
|
| We know for a fact that's wrong via functional
| programming, state machines, and formal verification
| ollysb wrote:
| It's fine to have dependencies, the point is two services
| that need to be deployed at the same time are not
| independent microservices.
| wowohwow wrote:
| Yes, the user I'm replying to is suggesting that taking
| on a dependency of a shared software repository makes the
| service no longer a microservice.
|
| That is fundamentally incorrect. As presented in my other
| post you can correctly use the shared repository as a
| dependency and refer to a stable version vs a dynamic
| version which is where the problem is presented.
| andrewmutz wrote:
| Needing to upgrade a library everywhere isn't necessarily a
| sign of inappropriate coupling.
|
| For example, a library with a security vulnerability would need
| to be upgraded everywhere regardless of how well you've
| designed your system.
|
| In that example the monolith is much easier to work with.
| mettamage wrote:
| Example: log4j. That was an update fiasco everywhere.
| mjr00 wrote:
| While you're right, I can only think of twice in my career
| where there was a "code red all services must update now",
| which were log4shell and spectre/meltdown (which were a bit
| different anyway). I just don't think this comes up enough in
| practice to be worth optimizing for.
| wowohwow wrote:
| You have not been in the field very long than I presume?
| There's multiple per year that require all hands on deck
| depending on your tech stack. Just look at the recent NPM
| supply chain attacks.
| mjr00 wrote:
| You presume very incorrectly to say the least.
|
| The npm supply chain attacks were only an issue if you
| don't use lock files. In fact they were a great example
| of why you _shouldn 't_ blindly upgrade to the latest
| packages when they are available.
| wowohwow wrote:
| Fair enough, which is why I called out my assumption:).
|
| I'm referring to the all hands on deck nature of
| responding to security issues not the best practice. For
| many, the NPM issue was an all hands on deck.
| j45 wrote:
| Monorepos reasonably well designed and fleixble to grow with
| you can increase development speed quite a bit.
| reactordev wrote:
| [delayed]
| chmod775 wrote:
| In practice most monoliths turned into "microservices" are just
| monoliths in disguise. They still have most of the failure modes
| of the original monolith, but now with all the complexity and
| considerable challenges of distributed computing layered on top.
|
| Microservices as a goal is mostly touted by people who don't know
| what the heck they're doing - the kind of people who tend to
| mistakenly believe blind adherence to one philosophy or the other
| will help them turn their shoddy work into something passable.
|
| _Engineer_ something that makes sense. If, once you 're done,
| whatever you've built fits the description of "monolith" or
| "microservices", that's fine.
|
| However if you're just following some cult hoping it works out
| for your particular use-case, it's time to reevaluate whether
| you've chosen the right profession.
| Nextgrid wrote:
| Microservices were a fad during a period where complexity and
| solving self-inflicted problems were rewarded more than
| building an actual sustainable business. It was purely a
| career- & resume-polishing move for everyone involved.
|
| Putting this anywhere near "engineering" is an insult to even
| the shoddiest, OceanGate-levels of engineering.
| readthenotes1 wrote:
| Some this sounds like the journey to ejb's and back.
| nyrikki wrote:
| > Once the code for all destinations lived in a single repo, they
| could be merged into a single service. With every destination
| living in one service, our developer productivity substantially
| improved. We no longer had to deploy 140+ services for a change
| to one of the shared libraries. One engineer can deploy the
| service in a matter of minutes.
|
| This is the problem with the undefined nature of the term
| `microservices`, In my experience if you can't develop in a way
| that allows you to deploy all services independently and without
| coordination between services, it may not be a good fit for your
| orgs needs.
|
| In the parent SOA(v2), what they described is a well known anti-
| pattern: [0] Application Silos to SOA Silos
| * Doing SOA right is not just about technology. It also requires
| optimal cross-team communications. Web Service Sprawl
| * Create services only where and when they are needed. Target
| areas of greatest ROI, and avoid the service sprawl headache.
|
| If you cannot, due to technical or political reasons, retain the
| ability to independently deploy a service, no matter if you
| choose to actually independently deploy, you will not gain most
| of the advantages that were the original selling point of
| microservices, which had to do more with organizational scaling
| than technical conserns.
|
| There are other reasons to consider the pattern, especially due
| to the tooling available, but it is simply not a silver bullet.
|
| And yes, I get that not everyone is going to accept Chris
| Richardson's definitions[1], but even in more modern versions of
| this, people always seem to run into the most problems because
| they try to shove it in a place where the pattern isn't
| appropriate, or isn't possible.
|
| But kudos to Twilio for doing what every team should be,
| reassessing if their previous decisions were still valid and
| moving forward with new choices when they aren't.
|
| [0] https://www.oracle.com/technetwork/topics/entarch/oea-soa-
| an... [1]
| https://microservices.io/post/architecture/2022/05/04/micros...
| yearolinuxdsktp wrote:
| I would caution that microservices should be architected with
| technical concerns first---being able to deploy independently
| is a valid technical concern too.
|
| Doing it for organizational scaling can lead to insular vision
| with turf defensive attitude, as teams are rewarded on the
| individual service's performance and not the complete product's
| performance. Also refactoring services now means organizational
| refactoring, so the friction to refactor is massively
| increased.
|
| I agree that patterns should be used where most appropriate,
| instead of blindly.
|
| What pains me is that a language like "Cloud-Native" has been
| usurped to mean microservices. Did Twilio just stop having a
| "Cloud-Native" product due to shipping a monolith? According to
| CNCF, yes. According to reason, no.
| mikert89 wrote:
| "Microservices is the software industry's most successful
| confidence scam. It convinces small teams that they are "thinking
| big" while systematically destroying their ability to move at
| all. It flatters ambition by weaponizing insecurity: if you're
| not running a constellation of services, are you even a real
| company? Never mind that this architecture was invented to cope
| with organizational dysfunction at planetary scale. Now it's
| being prescribed to teams that still share a Slack channel and a
| lunch table.
|
| Small teams run on shared context. That is their superpower.
| Everyone can reason end-to-end. Everyone can change anything.
| Microservices vaporize that advantage on contact. They replace
| shared understanding with distributed ignorance. No one owns the
| whole anymore. Everyone owns a shard. The system becomes
| something that merely happens to the team, rather than something
| the team actively understands. This isn't sophistication. It's
| abdication.
|
| Then comes the operational farce. Each service demands its own
| pipeline, secrets, alerts, metrics, dashboards, permissions,
| backups, and rituals of appeasement. You don't "deploy" anymore--
| you synchronize a fleet. One bug now requires a multi-service
| autopsy. A feature release becomes a coordination exercise across
| artificial borders you invented for no reason. You didn't
| simplify your system. You shattered it and called the debris
| "architecture."
|
| Microservices also lock incompetence in amber. You are forced to
| define APIs before you understand your own business. Guesses
| become contracts. Bad ideas become permanent dependencies. Every
| early mistake metastasizes through the network. In a monolith,
| wrong thinking is corrected with a refactor. In microservices,
| wrong thinking becomes infrastructure. You don't just regret it--
| you host it, version it, and monitor it.
|
| The claim that monoliths don't scale is one of the dumbest lies
| in modern engineering folklore. What doesn't scale is chaos. What
| doesn't scale is process cosplay. What doesn't scale is
| pretending you're Netflix while shipping a glorified CRUD app.
| Monoliths scale just fine when teams have discipline, tests, and
| restraint. But restraint isn't fashionable, and boring doesn't
| make conference talks.
|
| Microservices for small teams is not a technical mistake--it is a
| philosophical failure. It announces, loudly, that the team does
| not trust itself to understand its own system. It replaces
| accountability with protocol and momentum with middleware. You
| don't get "future proofing." You get permanent drag. And by the
| time you finally earn the scale that might justify this circus,
| your speed, your clarity, and your product instincts will already
| be gone."
|
| -DHH
| untwerp wrote:
| Also from DHH: microservices were a zero-interest rate
| phenomena https://youtu.be/iqXjGiQ_D-A?t=924
| maxdo wrote:
| Both approaches can fail. Especially in environments like Node.js
| or Python, there's a clear limit to how much code an event loop
| can handle before performance seriously degrades.
|
| I managed a product where a team of 6-8 people handles 200+
| microservices. I've also managed other teams at the same time on
| another product where 80+ people managed a monolith.
|
| What i learned? Both approaches have pros and cons.
|
| With microservices, it's much easier to push isolated changes
| with just one or two people. At the same time, global changes
| become significantly harder.
|
| That's the trade-off, and your mental model needs to align with
| your business logic. If your software solves a tightly connected
| business problem, microservices probably aren't the right fit.
|
| On the other hand, if you have a multitude of integrations with
| different lifecycles but a stable internal protocol,
| microservices can be a lifesaver.
|
| If someone tries to tell you one approach is universally better,
| they're being dogmatic/religious rather than rational.
|
| Ultimately, it's not about architecture, it's about how you build
| abstractions and approach testing and decoupling.
| strken wrote:
| Can you explain a bit more about what you mean by a limit on
| how much code an event loop can handle? What's the limit,
| numerically, and which units does it use? Are you running out
| of CPU cache?
| rozap wrote:
| To me this rationalization has always felt like duct tape over
| the real problem, which is that the runtime is poorly suited to
| what people are trying to do.
|
| These problems are effectively solved on beam, the jvm, rust,
| go, etc.
| btown wrote:
| Some important context to this 2018 article is given here:
| https://www.twilio.com/en-us/blog/archive/2018/introducing-c...
|
| TL;DR they have a highly partitioned job database, where a job is
| a delivery of a specific event to a specific destination, and
| each partition is acted upon by at-most-one worker at a time, so
| lock contention is only at the infrastructure level.
|
| In that context, each worker can handle a similar balanced
| workload between destinations, with a fraction of production
| traffic, so a monorepo makes all the sense in the world.
|
| IMO it speaks to the way in which microservices can be a way to
| enforce good boundaries between teams... but the drawbacks are
| significant, and a cross-team review process for API changes and
| extensions can be equally effective and enable simplified
| architectures that sidestep many distributed-system problems at
| scale.
| blatherard wrote:
| (2018)
| ShakataGaNai wrote:
| Too much of anything sucks. Too big of a monolith? Sucks. Too
| many microservices? Sucks. Getting the right balance is HARD.
|
| Plus, it's ALWAYS easier/better to run v2 of something when you
| completely re-write v1 from scratch. The article could have just
| as easily been "Why Segment moved from 100 microservices to 5" or
| "Why Segment rewrote every microservice". The benefits of
| hindsight and real-world data shouldn't be undersold.
|
| At the end of the day, write something, get it out there. Make
| decisions, accept some of them will be wrong. Be willing to
| correct for those mistakes or at least accept they will be a pain
| for a while.
|
| In short: No matter what you do the first time around... it's
| wrong.
___________________________________________________________________
(page generated 2025-12-13 23:00 UTC)