[HN Gopher] Disasters I've seen in a microservices world
___________________________________________________________________
Disasters I've seen in a microservices world
Author : gregdoesit
Score : 123 points
Date : 2021-06-13 13:51 UTC (9 hours ago)
(HTM) web link (world.hey.com)
(TXT) w3m dump (world.hey.com)
| sackerhews wrote:
| I was once working with an engineer so hell bent on splitting
| everything up into microservices that at one point logging became
| incompatible with his solution.
|
| He then argued that our banking solution didn't need logging,
| because it was so well tested that failure rates would be
| extremely low.
|
| I'm not making this up.
| deckard1 wrote:
| I was going to comment on this very thing in the other thread
| about software glue[1]. In that article there is a youtube video
| on Multics vs Unix[2] which really outlines why microservices
| were always doomed.
|
| Someone should coin a new law for how programmers have to
| rediscover Brooks's law every 5-10 years. The issue with
| microservices, as it has always been, is that you need an
| _enormous_ company to brute force the communication pathways and
| maintenance overhead for it to all work. And by work, I don 't
| mean function efficiently (as the Multics vs Unix video shows). I
| mean _just function_. Just work at all. The Multics team had all
| the devs and the Unix team was two guys doing laps around the
| Multics team. Because they had the mathematics on their side.
|
| Remember the bad old days of memory thrashing? That's what
| happens to teams that do not have enough bandwidth to properly
| maintain the dozens of services they are responsible for. Your
| organization gets frozen.
|
| This is what we all get for taking advice from the Googles and
| Facebooks of the world. Google has like a billion lines of code
| in a monorepo. They do not do things remotely like 99% of the
| businesses out there. They are sitting on huge piles of money
| that lets them be incredibly inefficient _for decades_.
|
| [1] https://news.ycombinator.com/item?id=27482832
|
| [2] https://www.youtube.com/watch?v=3Ea3pkTCYx4
| helge9210 wrote:
| > I've seen many engineers ignoring these because it's "an edge
| case", to realize later they have a massive data integrity
| problem.
|
| Massive optimism detected at the part of "to realize later".
|
| Also, if you have "edge case" with high (close to 1 but lower
| than 1) probability of successful outcome, with more and more
| tries probability of all outcomes being successful moves to zero
| (multiplication of lower than 1 values) and probability of at
| least single failed outcome goes to 1 (addition of lower than 1
| values). You still have to handle low probability edge cases.
| spaetzleesser wrote:
| I recently got talked into developing a new project with
| Kubernetes and microservices. It's an interesting journey but the
| complexity this adds is just enormous. Debugging is hard,
| refactoring is hard once it touches service boundaries,
| coordinating releases between services is hard and so on. I
| highly doubt that we will ever scale to a size where the
| complexity pays off.
|
| I feel this kind architecture of especially appeals to people who
| like to write only new code instead of understanding existing
| code. They don't like to read old code to see where new
| functionality may fit in so they spin up new services.
|
| We are complaining about maintenance of old COBOL code but God be
| with the poor people who in 20 years will have to maintain the
| monstrosities we are creating today.
| mech422 wrote:
| >>We are complaining about maintenance of old COBOL code but
| God be with the poor people who in 20 years will have to
| maintain the monstrosities we are creating today.
|
| I'm hoping for some major tax changes right before my
| retirement so I can dust off my COBOL and get phat lewt!
|
| I wonder how many people that have actually written COBOL will
| be left by 2030? No one mentions it here, but I'm sure there
| must be programmers deep in the bowels of banks and insurance
| companies that are learning COBOL even now?
|
| Oh - And RPG... wonder if there's any of that left ?
| yurishimo wrote:
| There are definitely new devs learning COBOL. I'm not sure
| where I saw it, but I could have sworn I saw an article or
| blog post about these mission critical companies paying
| insane wages and signing bonuses to devs who would sign-on to
| learn & maintain their infrastructure. And since these jobs
| would likely be on prem for security reasons, they don't have
| to pay insane Silicon Valley wages either.
|
| Imagine living in Cincinnati making $250k maintaining COBOL?
| Some people are totally fine with that paradigm and will keep
| the system running as long as it needs to.
| mech422 wrote:
| I was down until the on-prem thing..
|
| Still, I wouldn't be surprised about insane wages for
| COBOL. You make money at the bleeding edge of tech, or way
| back on the tail end - scarcity breeds profit :-)
| gravypod wrote:
| > One could not just look into a project in their IDE, but it
| required to have multiple projects open simultaneously to make
| sense of all that mess
|
| Why not both? You can set things up so your go-to-def can
| understand your API calls and head to the right place. This is
| very easy to do with a monorepo + protobuf setup.
|
| > How much does it cost to spin 200 services in a cloud provider?
| Can you do it? Can you also spin up the infrastructure needed to
| run them?
|
| Assuming 256mb RAM/service you're still well within 1 machine
| territory. Once you get above 1 machine territory you can set
| things up so that you can:
|
| 1. Build really good integration testing tooling so that devs
| don't really need to interface with all services. In a test spin
| up everything you need + deps, run an API call, tear everything
| down. This can be cached if your build system does that. You can
| run into issues if you have situations where 1 API call hits
| _every_ service but if you 've done that you've already messed
| up. In those cases the best you can do is mock a step in the
| chain there, run a few test hits the entire chain before release,
| but then have devs run against the mock in their integration
| tests.
|
| 2. Hybrid environments. You run a dev cluster that has all of
| your basic infrastructure that doesn't change much and provide a
| way for developers to launch new tasks that don't get routed to
| unless the driver has a feature flag flipped. Essentially you
| have a "dev" cluster that is continuously delivered from your
| master repo, each developer has the ability to launch new tasks
| in this cluster, and they can say "all traffic from alice for
| FoobarService should go to `{namespace=bob,service=Foobar}`.
|
| > As you can imagine, end-to-end tests have similar problems to
| development environments. Before, it was relatively easy to
| create a new development environment using virtual machines or
| containers. It was also fairly simple to create a test suite
| using Selenium to go through business flows and assert they were
| working before deploying a new version.
|
| Why is it not simple anymore? I've implemented this at more than
| one company.
|
| > Aside from being an obvious single-point-of-failure, defeating
| some of the service-oriented architecture's principles, there's
| more. Do you create a user per service? Do you have fine-grained
| permissions so service A can only read or write from specific
| tables? What if someone removes an index unintentionally? How do
| we know how many services are using different tables? What about
| scaling?
|
| Come up with a convention for your company and stick to it. If
| you can automate it that's better. If you build some way for your
| task you are running to know "who" it is they can inject that
| information into other libraries. For example you can inject the
| following environment variables into a container:
| FOOBAR_DB_ABCD=pg FOOBAR_DB_ABCD_PASSWORD=...
| FOOBAR_DB_ABCD_HOST=... FOOBAR_DB_ABCD_PORT=...
|
| You can then have some library you write expose a
| `OpenDatabase("abcd")` that connects in and injects everything. A
| security operator can then provision accounts and everything
| transparently. If you generate those env vars from some automated
| config management tool you don't even have to see the passwords.
|
| > Instead of having a monolith getting all of the traffic, now
| you have a home-made Spring Boot service getting all of it! What
| could go wrong? Engineers quickly realize this is a mistake, but
| as there are many customizations, sometimes they cannot
| substitute this piece for stateless, scale-friendly ones.
|
| I don't think this is a single point of failure. This is a single
| point of failure for a specific _subset_ of your infrastructure
| and at that it should be a very simple (mostly-pass-through)
| component. If your mobile gateway dies, your backend one shouldn
| 't. If all of your API gateways die then your integrations to
| third parties that are required for legal compliance should stay
| up, etc.
|
| > I've seen teams using circuit breakers and then increase the
| timeouts of an HTTP call to a service downstream.
|
| You should always decrease timeouts for operations if you're
| attempting to retry calls. You can also use a load balancer that
| already knows the liveness state of all of your instances.
|
| Aside from this bit I mostly agree with this section about
| timeouts, retries, etcs is actually correct. If you are tackling
| a single problem that's simple don't break it into a distributed
| system. If you are saying "I want to have X but we should
| implement a Y" where Y is a completely different thing that
| doesn't need to talk to X directly, then why not implement it in
| a separate binary? There's no reason they can't share code to
| make the burden of operation low.
| jaredcwhite wrote:
| The problem with the hype around microservices was that it's the
| web app deployment equivalent of writing device drivers in C.
| Sure, some people can do it. Some people _have_ to do it. Yet
| most people shouldn 't even make the attempt.
|
| I've been on teams where we can't even reliably deploy code over
| time to _one_ service. The idea of our team maintaining multiple
| services is madness. That 's not a knock on any individual's
| technical merits. Deploying code to the cloud is just hard,
| period--and that's even when talking about a traditional
| monolith!
|
| I'm glad the pendulum is swinging back. The microservices pattern
| is useful for the areas in which it's useful...it just so happens
| that problem space is _way_ smaller than the hype cycle cared to
| admit a few years ago.
| dwaite wrote:
| > I've been on teams where we can't even reliably deploy code
| over time to one service. The idea of our team maintaining
| multiple services is madness. That's not a knock on any
| individual's technical merits. Deploying code to the cloud is
| just hard, period--and that's even when talking about a
| traditional monolith!
|
| If devops has a good philosophy here it is - if something is
| hard, you should do it more often. When something is
| intermittent pain it can be avoided or considered a one-off; if
| you are say deploying your web app once a week or more people
| will start to figure out how to automate the manual processes
| and optimize the slow ones, rewrite troublesome components and
| come up with strategies for things like incremental database
| migrations.
|
| More generally - everything is a trade-off, and you shouldn't
| blindly accept more complexity when you don't need the benefits
| that it is supposed to provide. But sometimes you need to
| embrace the complexity of doing things the right way when you
| are hitting up against limitations of doing it the wrong way
| (in this example - slow, painful, error-prone deployments)
|
| Microservices are useful when the monolith becomes too complex
| for local reasoning and management, so you instead
| compartmentalize things into components with contracts and
| reason about those, while you are responsible for managing just
| your own component. You're taking on complexity (and latency,
| and additional resource utilization) because your system
| started to hit up against the limits of everyone working within
| the same project.
|
| If you haven't hit the point where your database is falling
| over in production because of concurrent loads, or where
| development is hindered by the infrastructure/dependency or
| time requirements of doing a local test, or various other
| problems - you likely don't need to invest in doing
| microservices.
| jameshart wrote:
| I guess you just won't get traction on HN writing 'Disasters I've
| seen in a monolithic world'.
|
| If you have never gone through a significant dependency upgrade
| on a large monolithic codebase, you might not appreciate the
| value of microservice architecture. I've been at companies where
| the number one technical achievement for a tech organization for
| an entire calendar year was 'successfully moved the app from .net
| 2 to .net 4'. Have you ever gone through the process of
| integrating an acquired company's monolith into an acquiring
| company's monolith? It's futile. With a microservice architecture
| though I've seen acquisitions integrate significant systems
| (things like billing and auth) in a matter of weeks where
| monolith projects dragged on for years.
|
| Sure, there's no silver bullets. But there are a _lot_ of
| problems with monoliths which microservices eliminate. Not
| without trade offs, naturally.
| jeffshek wrote:
| In those organizations, I highly doubt microservices or
| monolithic decisions would have made a difference.
|
| Disorganized management won't magically get fixed if the root
| problems of ASAP, priority, techinal debt, etc features aren't
| fixed.
|
| The fact that the organization was trying to combine two large
| monolithic codebase into one is an obvious smell that technical
| decisions are made by someone non-technical.
| jameshart wrote:
| Well yes, acquisitions are frequently decided by nontechnical
| people. That's where the money comes from.
|
| For better or worse, nontechnical people sometimes make
| decisions. One of the jobs of a software architecture is to
| be robust in accommodating the consequences of them.
| Monolithic architectures frequently _aren't_.
| dspillett wrote:
| _> I guess you just won't get traction on HN writing 'Disasters
| I've seen in a monolithic world'._
|
| Though I think mostly that is because the discussion there has
| been well trodden already. Microservice Architectures are
| relatively new, less completely explored, and less familiar to
| many.
|
| There are new and interesting mistakes to make (the biggest two
| being to use this model where it is not at all appropriate, or
| cargo-culting and not _really_ understanding what its
| advantages are so not actually taking advantage of them) where
| most of the "interesting" problems to be had in the monolithic
| world have been well documented for quite some time (which is
| why new models pop up, to try remove the potential for these
| problems in some, many, or even most, cases (anyone who says
| method X is better in _all_ cases should be treated with deep
| suspicion)).
| mavelikara wrote:
| A response someone wrote to this article:
| https://medium.com/productboard-engineering/countering-micro...
| cratermoon wrote:
| > Timeouts, retries, and resilience
|
| At a previous employer I was responsible for a critical service
| that was starting to show strain as traffic ramped up. It used
| Hystrix as the circuit breaker for calls to backend services,
| including the DB, and at peak times the thread pool would fill
| and start rejecting additional requests. I was tasked with fixing
| that.
|
| There's a very simple formula for tuning the number of threads:
|
| > requests per second at peak when healthy x 99th percentile
| latency in seconds + some breathing room
|
| The catch is that getting good RPS and latency numbers is in a
| distributed system deployed across three geographically separated
| datacenters is the opposite of simple. In particular, the legacy
| of the system meant that we had one write instance of the DB, in
| one datacenter, meaning that latency was different depending on
| which DC was the source of the call, so there was no one setting
| that worked for all instances.
| ferdowsi wrote:
| It's interesting to hear about stability concerns. Overall I
| think my organization moving to microservices improved our
| resiliency story. It allowed us to freeze sensitive legacy
| services and gradually build other surrounding services that
| incrementally replaced those legacy services with better-
| performing Go services. Rolling out new services is not onerous
| due to our Kubernetes platform (which was nowhere near as
| difficult to build on as some might suggest).
|
| Strong service boundaries helped us, they didn't hold us back.
| e67f70028a46fba wrote:
| Putting a network connection between your application
| abstractions was always a dicey proposition. (See EJB 1.0)
|
| Making it the entire basis for application abstraction is lunacy,
| the sort of extremely clever idiocy that can only occur in the
| tech world.
| discreteevent wrote:
| The funny thing is that Martin Fowler had a First Law of
| Distributed Objects: Don't distribute your objects. But then he
| switched and helped popularize microservices (he's rowing back
| a bit nowadays).
|
| I can only assume it's because he saw his clients doing it and
| for a consultant the client is always right. (And I think the
| client was doing it to indulge their hard to get developers who
| wanted complexity and the freedom to use their programming
| language of the week)
|
| It feels like a cowboy industry when this is the 'leadership'
| we have.
| e67f70028a46fba wrote:
| Yep.
|
| Look at the REST/JSON API debacle that has unfolded over the
| last two decades. It is _trivially obvious_ that REST is both
| difficult to implement and largely pointless outside of a
| hypermedia system, but the thought leaders never said a thing
| about it.
| kevmo314 wrote:
| > Some teams were suffering from servicitis. Even worse than
| that, it generated a lot of friction while developing. One could
| not just look into a project in their IDE, but it required to
| have multiple projects open simultaneously to make sense of all
| that mess.
|
| This is real: I've worked in projects where purely
| transformational code was offloaded into a "service". Refactoring
| it into a library reduced lines of code, computational cost, and
| code complexity dramatically.
|
| But wouldn't it be cool if there were a framework where the
| developer didn't have to demarcate where services started and
| ended? In principle, any pure asynchronous function could be
| abstracted out to a service. It would be neat if the compiler did
| that for me and deployment of the application was more like
| "deploy the cluster" instead of deploying each individual
| service.
| brown wrote:
| Next up, leftpad.io, leftpad as a service.
| kevmo314 wrote:
| Haha someone actually made this: http://left-pad.io/
| coding123 wrote:
| This is comedy gold:
|
| > `left-pad.io` is 100% REST-compliant as defined by some
| guy on Hacker News with maximal opinions and minimal
| evidence.
| ai_ja_nai wrote:
| Erlang?
| wrnr wrote:
| That is what lots of actor frameworks are trying, first specify
| a DAG of tranformations and then instantiate that graph on a
| physical network of computers such that the async boundaries
| can be scaled independently. It's cumbersome, like writing the
| same program squared just to get scaling for "free".
|
| Personally I think the Persistency-as-a-library and
| Consistency-as-a-library architecture is a saner alternative.
| aprdm wrote:
| Lol, that sounds like Java in 2011... put an annotation on a
| function and it's a service :) SOAP and all that.
|
| Maybe they had something right! It did cause problems when
| users put something on a tight loop that was actually a remote
| call but not easy to see.
|
| Anyho... what's old is new!
| kitd wrote:
| The one I was thinking of was SCA. Define the interface and
| implement the backend, either as a service or library, and
| the SCA layer in between would handle how to call it.
| rsynnott wrote:
| Also Java in 1996; remember RMI?
| ourmandave wrote:
| I used RMI but put a queue in front of it.
|
| One at a time, please _wait_ your turn...
| slver wrote:
| > But wouldn't it be cool if there were a framework where the
| developer didn't have to demarcate where services started and
| ended?
|
| Erlang/Elixir, for JVM Akka, for .NET Akka.NET.
| jerf wrote:
| No, it's a bit of a myth that those techs just magically
| scale everything. They don't make that problem go away at
| all. You still need to find homes for the services to run on.
| It does make it a single function call to potentially start
| up a new service on a remote node, but if you need to be
| careful about resource use you're hardly any better off than
| you are with anything else. In a way that can end up being
| _too_ easy to just add work to systems rather willy-nilly,
| the ceremony in kubernetes or any other more explicit system
| can be a _good_ thing.
| slver wrote:
| The question wasn't "how to magically scale everything".
|
| The question was why not eliminate the difference between
| in-process async call and a service, and that's basically
| what Erlang and actors do.
| jillesvangurp wrote:
| People do microservices for the wrong reasons. There are only a
| few somewhat valid reasons:
|
| 1) something has different run time needs than something else.
| Think CPU, memory, network. Breaking stuff up allows you to make
| different choices here. Although I have dodged this by simply
| deploying the same monolith and configure it to do different
| things.
|
| 2) Something needs to be developed by a different team and for
| whatever reasons you don't want those teams to be too dependent.
| It's a bad reason but it's valid in a lot of companies where
| certain teams just need to be engineered around or where there is
| a fundamental lack of trust between different parts of the org
| chart. Conway's law is a thing. It's the most common reason to do
| micro services.
|
| 3) You have two things depending on each other (cyclical
| dependency) and you want to reuse that thing. Extracting it to a
| third thing is a common way out. It's true for almost any
| component technology. If you have two components, you'll find a
| reason to create a third. And a fourth. And so on. However,
| consider using something less dramatic. E.g. code libraries are a
| valid choice. Or having an extra module in your source tree.
|
| Everything else is just needlessly/prematurely increasing
| overhead, deployment friction, etc. You get more things to
| monitor, deploy, manage the roadmap off, worry about, specialize
| in, etc. Big bloated organizations do micro services because they
| are big and bloated. Many smart startups keep this nonsense to a
| minimum. Of course some startups start out being over funded and
| bloat too early. VC money is great and sometimes requires over
| engineering like this (i.e. impress the suits). I've heard more
| than a few CTOs boast their multi cloud strategy and micro
| service architecture. In my mind that translates to we funnel a
| lot of VC money to Amazon and pay people full time to do just
| that. Ridiculous monthly bills and no users or traction is a
| common pattern in that world.
| Diggsey wrote:
| There are plenty more reasons than the three you've
| mentioned...
|
| However, the one thing that's _not_ a reason to use
| microservices and which the article brought up a lot, is to
| increase resilience. Microservices do not increase resilience:
| at best they can avoid reducing it.
|
| One of the most compelling reasons for me to use microservices
| is to limit the potential damage of bad decisions.
|
| In a monolithic application, one developer can have a bad day
| and write some code that eg. leaks a database object into an
| area of the program which should deal with business logic only.
| Or perhaps they try out some new technology that turns out to
| have problems.
|
| If that manages to get through code review, and is then copied
| elsewhere by other developers who are just following the
| example, you can quickly end up in a situation where fixing the
| problem requires everyone to stop feature work so that a
| significant rewrite or at least refactoring can take place.
|
| In most cases, the business cannot afford to stop all feature
| work in this way, and so the problem will persist forever,
| becoming a permanent drop in productivity. It will also make
| your developers miserable.
|
| In a microservices architecture, a single mistake can only grow
| to the size of a single service. In the worst case you have to
| stop feature work on that one service for a period of time, but
| work on other areas of the product can continue uninterrupted.
|
| Of course, you could still make a mistake whilst defining the
| boundaries between services. Luckily those decisions are much
| less frequent and involve many more people, so there's less
| chance of a freak bad decision. And even if you do get it
| wrong, you're only looking at two or three affected services
| rather than your whole product.
| macspoofing wrote:
| >you can quickly end up in a situation where fixing the
| problem requires everyone to stop feature work so that a
| significant rewrite or at least refactoring can take place.
|
| But that's normal and expected on any non-trivial codebase
| that has been evolving over some amount of time. We call that
| 'maintenance'. You'll never have 100% of your development
| resources pushing new features out. And if you do, well... we
| call that 'incurring technical debt'. You should be investing
| in this kind of maintenance continuously, because as your
| product evolves, new problem areas will emerge.
|
| >In most cases, the business cannot afford to stop all
| feature work in this way, and so the problem will persist
| forever, becoming a permanent drop in productivity.
|
| Yes.. That's called 'technical debt'. If your business does
| not invest in maintenance today, then it'll will cost them in
| the future. This isn't limited to software. If you're
| maintaining physical infrastructure, same deal. You're always
| fixing things. Microservices are not a solution to this.
|
| >In a microservices architecture, a single mistake can only
| grow to the size of a single service.
|
| Not necessarily. Microservices tend to be tightly coupled to
| other microservices. I've worked on a system that suffered
| from sporadic 'failure storms' where a failure in one service
| propagated across the entire system. It took days for us to
| track down the root cause. Microservices aren't a panacea. In
| fact, everything is easier with monolithic systems.
| Diggsey wrote:
| > But that's normal and expected on any non-trivial
| codebase that has been evolving over some amount of time.
|
| Having to stop all feature work is not normal. A healthy
| business should be able to have a certain proportion of its
| engineers working on tech debt at all times, it shouldn't
| have to stop all feature work to address tech debt.
| candiddevmike wrote:
| A single mistake can only impact one service? If a
| microservice depends on said broken microservice, you could
| experience a cascade of failures, or possibly data
| corruption, because microservice development is typically
| predicated on trusting the other components following their
| API spec.
| yxhuvud wrote:
| Well there are different cases of resilience. Creating a
| separate service for some part that have the potential for
| network traffic spikes can be fine and can make the system
| more resilient against that.
|
| But you are right in that it makes the code resilience worse,
| as it makes logic discovery and overview harder.
| jmchuster wrote:
| Hmm, i don't think our needs fall into any of those three. But
| our reasons may just be, as you say, the wrong reasons.
|
| We split up our services when the majority of changes made to a
| service can be made independently of everyone else. So then
| each of our services is a different codebase, and they each
| have a completely different rate of change from each other. You
| very rarely would ever make changes across all services as
| once, only when changing what's being communicated between some
| set of services. Some services are large, some are small, and
| many of them fall into the category of -- developed for a while
| then stabilized and now basically rarely every touched just
| works.
|
| So then the advantage is that you always have a small mental
| working set. You're focusing on a service at a time, have less
| to worry about breaking everything else with your changes. And
| then when you deploy, even if everything goes horribly wrong,
| it's just your one service that is down, and everything else is
| up and running, and you'll just have to process your queued
| messages once you came back online.
|
| And then of course each service is smaller, so less code, less
| tests, faster to compile, faster to run through the pipeline,
| faster to release. And you're only ever doing rolling deploys
| on a small sub-section of your infrastructure, never the whole
| thing at once.
| AlexCoventry wrote:
| Modular interfaces are great, but why require communication
| over those interfaces to go via network traffic? Can't you
| just use a library/package?
| spaetzleesser wrote:
| That's what I am always. People can't manage libraries so
| they think services will solve this problem. It seems to me
| that a lot of microservices are creating technical debt
| that will come due when the requirements change in a way
| that requires a big overhaul of the system.
| jmchuster wrote:
| Because there is state?
| DasIch wrote:
| Big organizations do microservices because it reduces the
| number of people who you need to coordinate changes with to a
| reasonably small number per service. This is critical for the
| ability to effectively maintain services and make progress at a
| reasonable pace.
|
| Important to note here is that a single microservice can
| actually be quite large with anywhere from 3-12 people working
| on just one or a few services. A single service could be larger
| than a startups monolith.
|
| I would argue that this is really the only reasonable use of
| microservices. If you can fully understand your monolith, you
| shouldn't change to microservices. If you fully understand or
| even know about all microservices in your organization, chances
| are that you're doing it wrong.
| yakshaving_jgt wrote:
| As far as I can tell, there's no reason why any of the
| constraints you have described could not be solved with
| libraries, and a function call is always going to be less
| complex than a network request.
| raffraffraff wrote:
| How do you ensure that everybody uses the correct version
| of the library? If a library change comes with a database
| schema change, how do you coordinate? With microservices,
| the owner of the service is responsible for the database
| schema change and service upgrade. Nobody else needs to be
| involved. (Genuine question, I'm an infrastructure guy, not
| a software engineer)
| yakshaving_jgt wrote:
| Where I work, we use Haskell's type system to keep all of
| these boundaries in sync. All of the "services" we have
| extracted are indeed libraries. These libraries use type
| classes to interface with the core of our application
| which manages data persistence. This also means that each
| "service" can be written mostly in isolation and use a
| different data store (like an in-memory DB or something),
| depending on our needs.
| DasIch wrote:
| There are many reasons for why a library might not be
| sufficient. The most common one is that what you're doing
| is stateful and requires some sort of data store.
|
| That has a huge impact because now you need to set up the
| data store and potentially perform migrations, which means
| you may need to change how you do deployments. It may
| introduce a bottleneck such as number of connections to
| that store, so you now have limitations to think about when
| scaling.
| Tabular-Iceberg wrote:
| Why can't a library be stateful or have a data store?
|
| If you're using libpng to edit an image on your disk you
| are working with both state and a data store. Yet plenty
| of applications do this every day without feeling a need
| to call it over a network.
|
| And if you have to set up and migrate a client/server
| DBMS for the library, wouldn't you also have to do the
| exact same steps as if it was running as its own process
| somewhere the network?
| spaetzleesser wrote:
| "Important to note here is that a single microservice can
| actually be quite large with anywhere from 3-12 people
| working on just one or a few services. A single service could
| be larger than a startups monolith.".
|
| This seems quite reasonable In some companies it seems it's
| more like 1 developer managing 3-12 services :-(
| slver wrote:
| Disaster #1 is too small services, and Disaster #4 is huge,
| shared databases (between many services).
|
| Which reaffirms my overall opinion that most people writing
| services have no idea what's a service and what it encapsulates
| (it encapsulates its own state, for example, it's basically
| distributed OOP).
|
| BTW, remember when "your service should be 100 lines of code top"
| was considered a best practice?
|
| Why is it so hard for most to resist this hype nonsense wave when
| its oncoming and it's so hard to resist the anti-hype wave that
| inevitably follows it? Because hype and anti-hype are simply the
| oscillation of a sea of empty minds in look for a solution to a
| problem they don't understand.
|
| I've been writing service oriented apps for decades. In my
| "world" nothing has changed.
| jeffbee wrote:
| No, I don't remember when 100-line services were the "best
| practice" because I don't elevate every stupid idea that some
| inexperienced kid blogs about to "best practice". I know 1000s
| of people in the industry and exactly none who would agree that
| is a best practice. Successful large-scale systems are built
| around much larger services, like "deliver this e-mail message"
| or "store this blob of data".
| slver wrote:
| You legit called 1000s of people in the industry to ask them
| what they think about the 100 lines meme. Great, appreciate
| your effort there. /s
| yxhuvud wrote:
| I think one major contributing reason is that the amount of
| programmers is still rising so fast, with each new generation
| being larger than the previous. Then every generation see the
| tail end of the previous generation and will then rebel against
| it to solve the issues. That they will have moderated after a
| couple oscillations doesn't matter as there is so few of them
| in most work places.
| unlimit wrote:
| > Why is it so hard for most to resist this hype nonsense wave
| when its oncoming and it's so hard to resist the anti-hype wave
| that inevitably follows it?
|
| Because then how will one do career development? We are
| building now for a client and the client wants it, I can feel
| it in my bones that what we are building is far more complex
| and will be difficult to maintain. I am no expert in
| microservices but this hunch is just coming from common sense.
| slver wrote:
| In your situation it seems like a non-technical person in
| charge of technical decisions. Those decisions are by
| definition poor quality.
|
| But the really bad moment is when the developers themselves
| make those bad choices entirely on their own.
| unlimit wrote:
| > In your situation it seems like a non-technical person in
| charge of technical decisions. Those decisions are by
| definition poor quality.
|
| The client is to blame. The client let go of all the people
| who knew the technical side of the product and has hired an
| architect to re-architect everything. And the client is in
| a hurry.
| aprdm wrote:
| Really good write up! I think the suite spot in a < 50 eng
| organization is either a monolithic or a microservice per domain
| (instead of per functionality)
|
| Once you have thousands of engineers, then you either need
| extreme discipline and a huge team maintaining the "devops"
| pipeline that everything goes through, or, it's basically
| everyone for themselves and a "devops" team trying to help others
| setting standards, best practices and whatnot.
| FpUser wrote:
| In places where micro/services were *really* needed people/orgs
| were implementing those even decades back. I personally was doing
| it in the 90s. For myself the criteria for making something as a
| service was simple: it will really cost the organization not to
| have it as a service.
|
| Now as with many things that nothingburger got suddenly overhyped
| and ended up being shoved into every hole disregarding of any
| technical rationale.
___________________________________________________________________
(page generated 2021-06-13 23:02 UTC)