[HN Gopher] The Modern Observability Problem
___________________________________________________________________
The Modern Observability Problem
Author : kiyanwang
Score : 49 points
Date : 2022-11-20 19:03 UTC (3 hours ago)
(HTM) web link (failingfast.io)
(TXT) w3m dump (failingfast.io)
| a_c wrote:
| People are seeing observability as a separate subject from
| product making. I see it as two side of the same "does this thing
| work" coin. To a product person, does it work mean are people
| using the tool to solve their problem. To engineer, it means the
| tool is solving the problem correctly. The former is answered by
| various analytics tool while the latter is answered by QA and
| observability tools.
|
| IMO this is doubling effort. Given all the logging,metrics and
| traces, can we not tell how good the product is performing? Given
| we know how a feature is used by users, all the clicks and APIs,
| can we not guaranteed future changes not breaking a user journey?
| I'm at the moment prototyping a tool to capture my thinking in
| product and software development. Having OtEL is making my idea
| so much easier to realise
| jahewson wrote:
| Mixing up these two concerns is a bad idea. Your product
| manager now has to convince engineers to implement the metrics
| they want. Data that is irrelevant to engineers now needs
| dragging along to places inside the system so that events are
| useful to both engineers who car about data and latency and
| product managers who care about individual user behaviour.
|
| So while there is indeed overlap, the two types of metrics
| involve different people, different teams, different concerns,
| different data, different privacy concerns, and different
| reliability/uptime concerns.
| a_c wrote:
| Structure follows strategy. There are various reasons product
| manager exist but the role doesn't exist unless the team
| grows to a certain scale. All founders serve as a product
| manager some point in their start up journey because they
| need to know if the thing the built works. There are
| companies wanting to spend energy on building things that
| work and skipping a layer of communication is one way of
| doing it.
|
| That's also how compiler speed up a program _sometime_ , by
| inline expansion or by loop unrolling. It doesn't work all
| the time. But as an option.
|
| Right now there seem to be an orthodox in product development
| vrnvu wrote:
| Observability is the other side of statistics.
| mkl95 wrote:
| > People are seeing observability as a separate subject from
| product making. I see it as two side of the same "does this
| thing work" coin.
|
| I have noticed the same thing with performance and UX. Most
| SaaS vendors only care about client / frontend UX and neglect
| the backend aspect of it. After a few years of intense
| development, they sell good looking software that is also
| mindbogglingly slow and in some extreme cases unusable.
| nickstinemates wrote:
| Increasingly, the network is a failure boundary people take for
| granted.
|
| Micro/3rd party services exacerbate this problem. You may see
| latency increase for a particular call, but what tells you why
| latency is increasing? What's measuring all of these tech
| choices, how do you know your 3rd party api is serving you
| traffic reliably?
| harryvederci wrote:
| Adding telemetry plus yet another service to fix the complexity
| of having a lot of services sounds a bit like de-escalating a
| conflict by slapping someone in the face.
|
| It always depends on the specific use case of course, but maybe
| it's also worth investigating if reducing the overall complexity
| of the system + amount of microservices could be a solution.
| dboreham wrote:
| I've become a proponent of OtEL tracing in recent months having
| used it successfully to diagnose some performance issues in
| multi-language, multi-service systems. I've found it also useful
| in single-process scenarios where heavy use of "async" prevails.
| Async-ish things (Kotlin coroutines and Scala futures in this
| particular case) make it hard to reason about the linear behavior
| of code using traditional debugging tools, I find. Disclosure:
| I've also made a couple of very small contributions to the
| project.
| afandian wrote:
| I'm curious, do you have more details?
| Joel_Mckay wrote:
| In general, Erlang/Elixir solved this challenge in an interesting
| way: with peer state awareness and channels (functionally
| equivalent to most micro-services use cases). It is commonly the
| back-end secret in a lot of low-latency game back-ends.
|
| If you are stuck on a polyglot loving project, than RabbitMQ or
| Kafka can bolt on most functions with the standard AMQP services.
| Erlang/Elixir is weird, but is a single kind of weird... which
| even has built-in profiling tools without external dependencies.
|
| Best of luck, =)
___________________________________________________________________
(page generated 2022-11-20 23:00 UTC)