[HN Gopher] OpenTelemetry in 2023
___________________________________________________________________
OpenTelemetry in 2023
Author : kevinslin
Score : 265 points
Date : 2023-08-28 14:58 UTC (8 hours ago)
(HTM) web link (bit.kevinslin.com)
(TXT) w3m dump (bit.kevinslin.com)
| throwaway894345 wrote:
| I'd be very interested to hear others experience (either as
| comments or good blog posts / write ups that they've read) with
| using Open Telemetry. I haven't used OTEL stuff directly, but
| I've had always been very disappointed with the telemetry vendors
| in the past (although as far as I can tell the problems weren't
| "these vendors don't talk to each other" kinds of problems that I
| think OTEL aims to solve).
| vmarchaud wrote:
| I've used OpenTelemetry since it's original alpha in 2020.
| Originally the main issue i had was supporting tracing across
| common libraries (there wasn't a lot of libraries supported
| back then). Now (and i recently worked with it) i would say is
| which protocol is supported by which component: your sdk
| generate spans/metrics in a specific format, then you send that
| to a collector that accept a range of protocol versions and
| finally you can send that to your vendor ... but you need to
| know which protocol/version it supports.
|
| That's not actually something you can do much about considering
| the sheer size of opentelemetry (both in term of implementation
| and vendors working on it) and i expect for people implementing
| nowadays, proto should be pretty stable and my experience
| should _theorically_ not be the case anymore.
| dboreham wrote:
| Experience has been positive. We had to understand (and in
| places fix/enhance) the agent used in our system but the
| benefits of it not being an expensive black box were huge. E.g.
| you can run the whole stack on a laptop and therefore use the
| same tooling for perf analysis in dev as production.
| bilalq wrote:
| OpenTelemetry is being pushed as a replacement for AWS X-Ray SDKs
| by AWS, but it's in such a broken state for Lambda right now. A
| 200-500% performance penalty for using it is insane[1][2].
|
| [1]: https://github.com/aws-observability/aws-otel-
| lambda/issues/...
|
| [2]: https://github.com/open-telemetry/opentelemetry-
| lambda/issue...
| CSDude wrote:
| I love OpenTelemetry and we want to trace almost every span
| happening. We'd be bankrupt if we went any vendor. We wired
| opentelemetry with Java magic, 0 effort and pointed to a self
| hosted Clickhouseand store 700m+ span per day with a 100$ EC2.
|
| https://clickhouse.com/blog/how-we-used-clickhouse-to-store-...
| flaviut wrote:
| I've got a small personal project submitting
| traces/logs/metrics to Clickhouse via SigNoz. Only about
| 400k-800k spans per day (https://i.imgur.com/s0J6Mzo.png), but
| running on a single t4g.small with CPU typically at 11% and
| IOPS at 4%. I also have everything older than a certain number
| of GB getting pushed to a sc1 cold storage drive.
|
| w/ 1 month retention for traces: +-parts.tabl
| e-----------------+------rows-+-disk_size--+-engine----+-compre
| ssed_size-+-uncompressed_size-+----ratio-+ |
| signoz_index_v2 | 26902115 | 17.06 GiB |
| MergeTree | 6.21 GiB | 66.74 GiB | 0.0930 |
| | durationSort | 26901998 | 5.44 GiB |
| MergeTree | 5.40 GiB | 53.02 GiB | 0.10190 |
| | trace_log | 123185362 | 2.64 GiB |
| MergeTree | 2.64 GiB | 37.96 GiB | 0.0695 |
| | trace_log_0 | 120052084 | 2.46 GiB |
| MergeTree | 2.45 GiB | 37.60 GiB | 0.06528 |
| | signoz_spans | 26902115 | 2.21 GiB |
| MergeTree | 2.21 GiB | 76.73 GiB | 0.028784 |
| | query_log | 16384865 | 1.91 GiB |
| MergeTree | 1.90 GiB | 18.31 GiB | 0.10398 |
| | part_log | 17906105 | 846.73 MiB |
| MergeTree | 845.39 MiB | 3.84 GiB | 0.21521 |
| | metric_log | 4713151 | 820.92 MiB |
| MergeTree | 806.13 MiB | 14.56 GiB | 0.05405 |
| | part_log_0 | 15632289 | 702.82 MiB |
| MergeTree | 701.70 MiB | 3.34 GiB | 0.20490 |
| | asynchronous_metric_log | 795170674 | 576.24 MiB |
| MergeTree | 562.50 MiB | 11.11 GiB | 0.049429 |
| | query_views_log | 6597156 | 461.35 MiB |
| MergeTree | 459.75 MiB | 6.36 GiB | 0.07060 |
| | logs | 6448259 | 408.59 MiB |
| MergeTree | 406.65 MiB | 5.99 GiB | 0.06627 |
| | samples_v2 | 949110122 | 345.01 MiB |
| MergeTree | 325.31 MiB | 22.09 GiB | 0.014382 |
|
| If I was less stupid I'd get a machine with the recommended
| Clickhouse specs and save myself a few hours of tuning, but
| this works great.
|
| Downsides:
|
| - clickhouse takes about 5 minute to start up because my tiny
| sc1 drive has like 4 IOPS allowed
|
| - signoz's UI isn't amazing. It's totally functional, and
| they've been improving very quickly, but don't expect datadog-
| level polish
| pranay01 wrote:
| Thanks for mentioning SigNoz, I am one of the maintainers at
| SigNoz and would love your feedback on how we can improve it
| further.
|
| If anyone wants to check our project, here's our GitHub repo
| - https://github.com/SigNoz/signoz
| flaviut wrote:
| I hope I'm not coming across as negative! Y'all are just
| have a much younger product, and have not had time to do
| all the polish and tiny tweaks. I'm also much more familiar
| with Datadog, and sometimes a learning curve feels like
| missing features.
|
| - I really like your new Logs & Traces Explorers. I spend a
| lot of time coming up with queries, and having a focused
| place for that is great. Especially since there's now a way
| to quickly turn my query into an alert or a dashboard item.
|
| - You've also recently (6mo?) improved the autocomplete
| dramatically! This is awesome, and one of my annoyances
| with Datadog
|
| Other feedback, and honestly this is all very minor. I'd be
| perfectly happy if nothing ever changed.
|
| - where do I go see the metrics? There's no "Metrics" tab
| the way there's a "Logs" and "Traces" tab. A "Metrics
| Explorer" would be great.
|
| - when I add a new plot, having to start out with a blank
| slate is not great. Datadog defaults to a generic
| system.cpu query just to fill something in, I find this
| helpful.
|
| - when I have a plot in a dashboard and I see it is
| trending in the wrong direction, it would be nice to be
| able to create an alert directly from the chart rather than
| have to copy the query over.
|
| - the exceptions tab is very helpful, but I've only
| recently discovered the LOW_CARDINAL_EXCEPTION_GROUPING
| flag. It'd be super nice if the variable part of exceptions
| was automatically detected and they were grouped
|
| - once nice thing in DD is being able to preview a span
| from a log or logs from a span without opening a new page.
| Or previewing a span from the global page. Temporary
| popping this stuff up in a sidebar would be great.
|
| - I'm not sure if there's a way to view only root spans in
| the trace viewer.
|
| - This might be a problem with the spring boot
| instrumentation, but I can't see how to figure out what
| kind of span it is. Is it a `http.request`, `db.query`,
| etc?
| _boffin_ wrote:
| Are you making sure that you're doing a sample rate, but send
| over all errors?
|
| At a former place, we were doing 5% of non-error traces.
| fastest963 wrote:
| How do you send all errors? The way tracing works, as I
| understand it, is that each microservice gets a trace header
| which indicates if it should sample and each microservice
| itself records traces. If microservice A calls microservice B
| and B returns successfully but then A ends up erroring, how
| can you retroactively tell B to record the trace that it
| already finished making and threw away? Or do you just accept
| incomplete traces when there are errors?
| sigwinch28 wrote:
| You can do head-based sampling and tail-based sampling.
|
| With head sampling, the first service in the request chain
| can make the decision about whether to trace, which can
| reduce tracing overhead on services further down.
|
| With tail-based sampling, the tracing backend can make a
| determination about whether to persist the trace _after_
| the trace has been collected. This has tracing overheads,
| but allows you to make decisions like "always keep errors".
| phamilton wrote:
| https://opentelemetry.io/docs/concepts/sampling/ describes
| it as Head/Tail sampling, but in practice with vendors I
| see it as Ingestion sampling and Index sampling. We send
| all our spans to be ingested, but have a sample rate on
| indexing. That allows us to override the sampling at index
| and force errors and other high value spans to always be
| indexed.
| fastest963 wrote:
| Maybe the Go client doesn't support that? https://opentel
| emetry.io/docs/instrumentation/go/sampling/
| phillipcarter wrote:
| It does, but the docs aren't clear on that yet.
| TraceIdRatioBased is the "take X% of traces" sampler that
| all SDKs support today.
| grogenaut wrote:
| Careful, we've had systems go down under increased load just
| emitting errors if they didn't emit much in non error state
| _boffin_ wrote:
| Can you go into more detail about your comment, please.
| cwp wrote:
| Not the GP, but:
|
| Imagine you're sampling successful traces at, say, 1%,
| but sending all error traces. If your error rate is low,
| maybe also 1%, your trace volume will be about 2% of your
| overall request volume.
|
| Then you push an update that introduces a bug and now
| _all_ requests fail with an error, and _all_ those traces
| get sampled. Your trace volume just increased 50x, and
| your infrastructure may not be prepared for that.
| Longwelwind wrote:
| I think what they means is that if you provisioned your
| system to receive spans for 5% of non-error requests and
| a few error requests, if for some random act of god, all
| the requests yield an error, your span collector will
| suddenyl receive spans for all requests.
| Topgamer7 wrote:
| We've seen problems with memory usage on failure too.
| Python implementation sends data to the collector in a
| separate thread from the http server operations. But if
| these start failing, its configured for exponential
| backoff, so it can hold onto a lot of memory, and start
| causing issues with container memory limits.
| CSDude wrote:
| Normally yes, but we do a lot of data collection and
| identifying what's an error is usually hard because of
| partial errors. We also care about performance, per tenant
| and per resource with lots of dimensionality and sampling
| reduces that information for us.
| push-to-prod wrote:
| That's a really informative post, the ClickHouse thing sounds
| interesting!
| [deleted]
| annanay wrote:
| This is really interesting, thanks for sharing. What's also
| cool was the low effort needed for this setup (Java
| autoinstrumentation + Clickhouse exporter + Grafana Clickhouse
| Plugin).
| podoman wrote:
| The reality is that most people don't want to manage their own
| Clickhouse store, and not all engineers can operate with SQL as
| efficiently as with code (me included). Nonetheless, this is
| pretty cool!
| hnarn wrote:
| > not all engineers can operate with SQL as efficiently as
| with code
|
| I don't mean for this to sound insulting but I honestly do
| not think this is an acceptable take to have as a developer.
|
| Not knowing SQL is like refusing to learn any language that
| has classes in it, simply because you don't like it.
|
| I've heard stories of huge corporations failing product
| launches because some code was written to SELECT * from a
| database and filtering it in-app instead of doing the queries
| correctly, and what's so fun with these types of issues is
| that they usually don't appear until weeks later when the
| table has grown to a size where it becomes a problem.
|
| When you're saying that you'd rather find the data in-app
| than in-database, you're putting the work on an inferior
| party in the transaction simply because you can't be
| bothered.
|
| The code will never* find the correct data faster than the
| database.
|
| * there may be exceptions, but they're far enough between to
| still say "never".
| xyzzy_plugh wrote:
| Dropping down to SQL to write a really complex query is, in
| my professional experience, always a poor use of time. It's
| far simpler to just write the dumb for-loops over your
| data, if you can access it.
|
| Of course not all engineers can operate with SQL as
| efficiently as code -- that's the whole point. Otherwise
| why would we be writing code? Learning SQL intimately
| doesn't change that fact.
| nrr wrote:
| It's unclear whether you mean that it's simpler to make a
| query and iterate over the rows to massage the result in
| your application or to make a query and then iterate over
| the returned rows to make more single-row queries. (Or
| perhaps some secret third thing I'm not considering.)
|
| I'll admit I'm a little curious about what exactly you
| mean here.
| qaq wrote:
| Did your for loop solution include concurrent access by
| multiple clients? I highly doubt "engineers can not
| operate with SQL as efficiently as code" can implement
| anything even remotely as robust as what SQL DBMS offer
| even for basic use cases. Are you mutating data? What
| will happen if the system crashes in the middle of the
| mutation? How are you handling concurrent writes and
| reads?
| duggan wrote:
| Obviously it greatly depends on what you're doing. If
| you're using a relational database as a glorified key-
| value store for offline or batch processing of a few
| hundred megabytes of data, sure. Hell, just serialize and
| unserialize a JSON document on every run if it's small
| and infrequent enough -\\_(tsu)_/-
|
| If you've got a successful data hungry web service with a
| reasonably normalized schema and moderately complex
| access patterns though, you're not going to be looping
| over the whole thing on every page load.
| simonw wrote:
| SQL is code.
| hnarn wrote:
| > Dropping down to SQL to write a really complex query
| is, in my professional experience, always a poor use of
| time.
|
| We're not talking about Assembly here, "dropping down" to
| SQL is something that anyone should be expected to do as
| soon as you're grabbing or modifying any data from a
| database in any scenario where performance or integrity
| matters. The errors you can see in situations like this
| are extremely complex and databases literally exist to
| solve them for us.
|
| Also, if we just completely disregard the performance for
| a second and focus on data security instead, how do you
| ensure sensitive data isn't passed to the wrong party if
| you don't care about what queries are being sent?
|
| I mean, it doesn't matter if it's not "in the end"
| displayed to an end user in the application you're
| writing, or if its not stored in the intermediary node
| where your code is running, that data is now
| unnecessarily on the wire in a situation where it never
| should have been in the first place. If you end up mixing
| one customers data with another's and sending all of it
| in such a way that it could even theoretically be
| accessed by a third party, that's a lawsuit waiting to
| happen regardless of whether it was "displayed" or
| "forwarded" or not.
|
| Imagine if you sniffed the packets going to some
| logistics app you use on your phone and you saw meta-data
| for all packages in your zip code in the response, or if
| some widget showing you your carbon footprint actually
| was based on a response containing the carbon footprint
| of every customer in the database. Even if it's just
| [user_id,co2] it's still completely unacceptable.
|
| Never mind scenarios where you're modifying, adding or
| deleting data, those are even worse and no explanation
| should be necessary for why.
| tkiolp4 wrote:
| There's a difference between writing olap and oltp sql
| queries. Hell, in the industry we even have a dedicated
| role for people who, among other things, write olap
| queries: data analysts. I'm assuming here that we are
| talking about writing complex analytical queries.
| jauntywundrkind wrote:
| "Don't want to manage their own" has for so long been a valid
| excuse but cloud costs haven't been going down for so long -
| in many cases prices have increased - and hardware keeps
| getting more badass. In so many cases it's fear speaking.
|
| A decent sized server will host a hugely capable instance
| that you may not have to think about for years. The scoffing
| down at DIY has made sense to some degree, but it just works
| brilliantly keeps getting to be a stronger & stronger case &
| most just assume reality can't actually work that well, that
| it'll be bad, and those folks won't always be right.
| qaq wrote:
| With current SSD prices a box that will have 30 million
| IOPS can cost you 10K. 30 mil IOPS in a cloud would be
| crazy $$$$
| CSDude wrote:
| We just left ours running for months in a Docker container.
| The volume is external, we just replace container image with
| new one, it takes 5 seconds to update, and spans are treated
| ephemeral. We store only 7d of data. We could use S3 but we
| have no use for that data in the long run.
|
| To be fair, we wanted to get experience on ClickHouse and
| it's a special database need special attention to details on
| both ops and schema design.
| klysm wrote:
| SQL is code and absolutely worth learning.
| simonw wrote:
| I'm beginning to sound like a broken record at this point,
| but if you don't know SQL very well but know how to use
| GPT-4, you have access to enough SQL to get a lot more done
| than you might think.
| podoman wrote:
| Disclaimer: I'm the founder of an observability company
| (Highlight.io).
|
| OpenTelemetry has been INCREDIBLY valuable to us. Not only has it
| made it super fast to build out SDKs for our customers, but the
| fact that its maintained actively gives us confidence that we're
| rolling out stable logic to customers' environments.
|
| I agree with the author that OpenTelemetry has succeeded, and its
| pretty obvious from the fact that most major observability
| vendors support.
|
| In short, to a developer it may not seem like its particularly
| valuable because a metrics/logs/traces API is quite simple
| whether or not you use OTEL. But the fact that this is an
| industry wide spec is where it becomes powerful.
|
| A few
| shayel wrote:
| [flagged]
| monlockandkey wrote:
| I'm new to telemetry. Can someone please explain the relationship
| between otel, Prometheus, Grafana agent?
| bboreham wrote:
| OpenTelemetry is primarily definitions of protocols, APIs and
| semantic conventions for instrumentation data (traces, metric
| and logs, in decreasing order of maturity). OpenTelemetry also
| ships the Collector, which is like a patch-bay accepting many
| different data formats and sending on in more formats.
|
| Grafana Agent is a bundling of a bunch of things for collecting
| instrumentation. The idea is to simplify deployment and allow
| opinionated setup. The Agent can upload to Grafana Cloud, or to
| any compatible backend (all the basic components are open
| source).
|
| In particular it bundles Prometheus Agent, which does metrics
| collection from anything Prometheus-compatible but not queries,
| and OTel Collector.
|
| It also bundles Promtail for logs.
|
| (I work for Grafana Labs)
| wdb wrote:
| It's a standardised way to send metrics to Prometheus. You can
| have a metrics exporter for Prometheus, e.g.
| https://opentelemetry.io/ecosystem/registry/?s=prometheus&co...
|
| [1]
| https://opentelemetry.io/ecosystem/registry/?s=prometheus&co...
| [2]
| https://opentelemetry.io/docs/specs/otel/metrics/sdk_exporte...
| podoman wrote:
| Not only prometheus fwiw. There's lots of providers that
| support it, e.g.: - datadog - honeycomb.io - highlight.io
| (I'm a founder) - sentry.io
| skrtskrt wrote:
| Prometheus: metrics collector/databases with a defined metrics
| format.
|
| Many metrics/timeseries databases make an effort to be
| "Prometheus-compatible" as it was kind of the unofficial
| standard.
|
| OTEL: new open standards which are supposed to provide vendor-
| independent formats which are compatible and composable between
| metrics, logs, and traces, and easily enable things like
| deriving metrics from traces (latency would be an obvious one
| here).
|
| Grafana Agent: a metrics? maybe also logs and traces? collector
| that supports Prometheus, OTEL and other metrics formats, which
| can forward, sample, and transform the data. Made by Grafana,
| open source, etc.
|
| Grafana's metrics DB Mimir and maybe some others are
| essentially "more scalable prometheus", and use prometheus
| metrics format on disc, so one large concern of the Grafana
| Agent would be converting OTEL metrics to prometheus metrics
| format for ingestion into the Grafana databases - but the agent
| has a whole bunch of other functions supported as well.
|
| OTEL Collector - non-vendor-specific collector, like Grafana
| Agent but largely just concerned with allowing
| collection/ingestion of OTEL and coverting other formats into
| OTEL. Allows extensions and plugins to be added for other
| purposes.
| _boffin_ wrote:
| Has anyone found a solution to automatically adding spans
| automatically for nodejs applications that doesn't involve
| littering startSpan or startActiveSpans everywhere?
|
| I'm currently looking into monkey patching, but that seems dirty.
| phillipcarter wrote:
| There isn't a way to automatically add spans for arbitrary
| functions in Node, no. There's currently a proposal to add some
| stuff that makes creating the span easier to make it auto-
| closing based on scope.
|
| FWIW the philosophy here is that observability is a part of the
| application rather than something separate. That's distinctly
| different from the APM philosophy, which is that a separate
| process "does the observability" and your app is "clean" from
| that. I think there's quite a benefit to manually instrumenting
| in your codebase intentionally rather than having an automated
| process do it for you. But I can understand not wanting to go
| through and do that.
| CubsFan1060 wrote:
| That's an interesting distinction. Given some of the older,
| larger code bases, manually instrumenting seems like it would
| require a reasonably large effort. I think that's one of the
| huge benefits that some of the APM's bring.
|
| I wonder if there is any reason OTEL can't have both.
| phillipcarter wrote:
| OTel kind of has both today. Most languages support
| autoinstrumentation via either an agent, libraries, or
| both. So for example, if you've got a big Spring Boot app,
| it'll instrument requests/responses and DB calls for you.
| Some languages also have lightweight "sprinkle some spans
| on it" things you can put in code, like Java method
| annotations. But none have truly automatic "span for every
| method call" instrumentation.
| CubsFan1060 wrote:
| Yeah, I've used the Ruby one, and the results were...
| painful.
| _boffin_ wrote:
| For stuff on the JVM, you just load the APM plugin and
| you're pretty much good to go as it inspects the bytecode,
| i believe.
| CubsFan1060 wrote:
| Oh, right. I guess my question was more around "why can't
| OTEL _also_ have something that does that"?
| pondidum wrote:
| Depends what you're trying to accomplish.
|
| We're using middleware to start spans for http and message
| handlers, and then adding `startSpan` where we need.
|
| I don't see a problem with `startSpan` everywhere as it's not
| much noiser than the `log.info` that would be there instead if
| we didn't have otel.
| Spivak wrote:
| I think my major pain point with them being everywhere is
| unlike a log they push a new stack frame and closure. They
| make tracebacks so much more annoying, yes I know the error
| happened in a start span, thank you.
|
| I wish observability actually observed more than it
| contributed. Once https://peps.python.org/pep-0669/ is
| available I'm gonna try my damndest to get otel working
| through it. Just give me a config file that says what
| functions you're interested in and I'll do the rest.
| pondidum wrote:
| Agreed the amount of stuff showing in stack traces can be
| annoying - but it does depend on language.
|
| I agree 100% in javascript/typescript its annoying, and I
| would love to get rid of them, In go however, there isn't
| an extra stack frame. Nor in C# thinking about it.
|
| The config file of functions to trace is a really
| interesting idea. How would you handle wanting to add data
| to the spans from inside those functions though? e.g. I
| want to add all kinds of props to the spans that can't be
| known except inside the function executing.
| _boffin_ wrote:
| I started my APM journey when using Spring w/ Java and fell
| in love with how i can trace the entire flow through the
| entire application and then just config on how many spans i
| want to send over and what sample rate i want collected.
|
| I'd love to accomplish that.
|
| What you mentioned is all nice and well (optimal route), but
| right now, i'm working with some applications that needs it,
| but has i don't even know how many methods / classes that i'd
| need to go through and implement it on.
| vadman97 wrote:
| Have you seen the OTEL auto instrumentation package[1]? It
| supports a number of common frameworks (as middleware) and
| libraries (via monkeypatches) setting up spans for you to
| capture things like external http requests, database/cache
| queries, etc. [2]
|
| [1] https://opentelemetry.io/docs/instrumentation/js/automatic/
| [2] https://github.com/open-telemetry/opentelemetry-js-
| contrib/t...
| _boffin_ wrote:
| Tested those before, but still doesn't get what i'm looking
| for, which is inside the black box without manually adding
| the spans. The implementations for bunyan or winston would
| come the closet, but still doesn't achieve what i'm looking
| for.
| eep_social wrote:
| I think the Ruby auto-instrumentation library does wrap
| practically every method but we ended up finding it too
| verbose. I think this is a bit of a goldilocks problem
| insofar as getting it just right is not easy and varies per
| application.
| _boffin_ wrote:
| One of the things i think that can be done is also limit
| the number of spans collected via configuration.
| no_wizard wrote:
| The OTEL official project libraries don't work well on the web
| frontend yet. No way of correlating errors to source-maps for
| instance, at least out of the box.
|
| The web browser collector published by the OTEL project uses
| Zone.js to hijack just about everything in the browser into
| contexts. If you used modern Angular before, you may recognize
| zone.js, its a real pain sometimes, and messes with globals,
| which isn't great, as it can create situations where behavior
| isn't predictable.
|
| I don't know that OTEL has any standard around things like
| session replays either. Lots of telemetry platforms support this
| (Sentry, Rollbar, DataDog etc)
|
| I think alot of backend teams have really like it. I do like the
| cross boundary nature of spans where you can follow them by a
| unique tag across your entire system.
|
| I personally find its extremely verbose at times, in terms of the
| payload it generates, some logging platforms are more compact in
| this regard, but in practice, I haven't noticed it to be an issue
| pranay01 wrote:
| Do check this doc from Otel
| https://opentelemetry.io/docs/instrumentation/js/getting-sta...
|
| It is not completely solving the issue, but a starting point
| no_wizard wrote:
| this is exactly what I referencing, its not really a starting
| point. I read through all the docs quite thoroughly, these
| are missing features and design choices
| renke1 wrote:
| I think with native Promises (and async/await?) there is
| currently no way to implement something like Zone.js properly.
| I've tried to instrument my code manually, but it's really
| error-prone and verbose. We really need something like
| https://nodejs.org/api/async_context.html#class-asynclocalst...
| to be implemented in the browser.
| vmarchaud wrote:
| You can follow [0] which is currently stage 2 to fix this
|
| [0]: https://github.com/tc39/proposal-async-context
| no_wizard wrote:
| In addition to this, is the new (stage 3 even!)explicit
| resource management proposal[0], supported by TypeScript
| version >= 5.2[1]
|
| Though I agree that async context is better fit for this
| generally, the ERM should be good for telemetry around
| objects that have defined lifetime semantics, which is a
| step in the right direction you can use today
|
| [0]: https://github.com/tc39/proposal-explicit-resource-
| managemen...
|
| [1]: https://www.totaltypescript.com/typescript-5-2-new-
| keyword-u...
| renke1 wrote:
| Thanks! I was only aware of a Zones proposal which was
| withdrawn I think.
| jauntywundrkind wrote:
| DataDog's front end instrumentation (Real User Monitoring) is
| also notably unrefined. Playing with Duplo blocks level of
| finesse.
|
| Does anyone have an even part way start at doing front end
| tracing?
| no_wizard wrote:
| As with most of the DataDog platform, I have found that once
| you go underneath the sheen it leaves alot to be desired.
|
| I've had better runs with Bugsnag for pure error reporting
| and more recently Sentry, which can do RUM / Session Replay
| collections.
|
| None are what you expect though. If you want really good user
| behavior analytics FullStory is still top notch
| streblo wrote:
| I'm hugely disappointed with OpenTelemetry. In my experience, its
| an over-engineered mess and the out-of-the-box experience is
| super user hostile. What it purports to be is so far away from
| what it actually is. Otel markets itself as a universal
| tracing/metrics/logs format and set of plug and play libraries
| that has adapters for everything you need. It's actually a bunch
| of half/poorly implemented libraries with a ton of leaky
| internals, bad adapters, and actually not a lot of functionality.
| iofiiiiiiiii wrote:
| Agreed, I find myself having to think orthogonally to common
| sense whenever I try to use one of its SDKs. Nothing works the
| way you expect it to, everything has 3 layers of unnecessary
| abstraction and needs to be approached via the back door. Many
| features have caveats about when it works, where it works, how
| much it works, during what phase of the moon it works and how
| long your strings can be when Jupiter is visible in the sky.
|
| That said, if we disregard the leaky SDK APIs and half-
| implemented everything, it does somewhat deliver on the
| pluggability promise. Before OTel, you had bespoke stacks for
| everything. Now there is some commonality - you can plug in
| different logging backends to one standard SDK and expect it to
| more or less work. Yes, it works less well than a vertically
| integrated stack but this is still something. It enables
| competition and evolution piece by piece, without having to
| replace an observability stack outright (never going to be a
| convincing proposition).
|
| So while the developer experience is pretty unpleasant and I am
| also disappointed with the actual daily usage, from an
| architectural perspective it opens up new opportunities that
| did not exist before. It is at least a partial win.
| fshr wrote:
| Ok, then do you have a suggestion for an alternative, or do you
| just put up with OT?
| cratermoon wrote:
| Okay but can you point to some specific experience and suggest
| improvements?
| caust1c wrote:
| Try implementing an OTEL Tracer. This interface is insane,
| and should really just be a struct.
|
| https://github.com/open-telemetry/opentelemetry-
| go/blob/trac...
| skrtskrt wrote:
| how is this interface insane?
|
| It's a list of the behaviors you need to implement if
| you're rolling your own OTEL Tracer Span implementation,
| and not using one of the multiple available.
|
| In contrast, OpenTracing's interfaces had hardly any
| required methods, so you had to do a runtime type-cast to
| the whichever implementation you were using in order to
| access anything useful on the Span like the OperationName.
| ramenmeal wrote:
| Why were you implementing your own tracer? Don't they
| publish an implementation?
| phillipcarter wrote:
| Yes, all SDKs have a tracer you can just use. While you
| can technically create your own tracer, you're officially
| in Hard Mode territory - there's no highly extensible
| system I'm aware of that makes swapping core concepts
| (that already have a default) easy.
| oaiey wrote:
| I am very happy with the progress of OpenTelemetry. When I pushed
| for it years ago, my developers where hesitant (it was new and
| they never heard about it) but when I revisited the topic a year
| ago, OpenTelemetry was everywhere in our systems and the
| log/tracability vendor we have was switching over to it.
|
| Thanks to this amazing group!
| hu3 wrote:
| Speaking about OpenTelemetry, has anyone used https://uptrace.dev
| ?
|
| From a quick glance it seems to be simple, free and open-source
| deployed as a single Go binary. They use ClickHouse to store
| data. Almost too good to be true.
|
| I'm contemplating them for a new project.
| podoman wrote:
| Fwiw, we do the same (https://highlight.io). Heard good things
| about uptrace as well.
| sheshe42 wrote:
| [flagged]
| paulddraper wrote:
| Two problems with OpenTelemetry:
|
| 1. It doesn't know what the hell it is. Is it a semantic
| standard? Is a protocol? It is a facade? It is a library? What
| layer of abstraction does it provide? Answer: All of the above!
| All the things! All the layers!
|
| 2. No one from OpenTelemetry has actually tried instrumenting a
| library. And if they have, they haven't the first suggestion on
| how instrumenters should actually use metrics, traces, and logs.
| Do you write to all three? To one? I asked this question two
| years ago, zero answers :( [1]
|
| [1] https://github.com/open-telemetry/opentelemetry-
| specificatio...
| donutshop wrote:
| I totally read that with kubernetes in mind.
| withinrafael wrote:
| 1. Agreed. It's the sink and the house attached to it, and the
| docs are thin and confusing as a result.
|
| 2. I had a similar experience to you. I wanted to implement a
| simple heartbeat in our desktop app to get an idea of usage
| numbers. This is surprisingly not possible, which greatly
| confuses me given the name of the project. The low engagement
| on my question put me off and I abandoned my OpenTelemetry
| planning completely [1][2].
|
| [1] https://github.com/open-
| telemetry/community/discussions/1598
|
| [2] https://github.com/open-telemetry/semantic-
| conventions/issue...
| jauntywundrkind wrote:
| Good deck of questions but I don't think they matter. I don't
| think those are answerable questions for observability, be it
| OpenTelemetry or other proprietary systems.
|
| You can go read the leading observability companies web pages
| and they'll have a 4 page writeup on custom instrumentation.
| That's not much, just covers very elementary basics! It's not
| like OTel is behind. The answer just tends heavily towards "it
| depends."
|
| Once you have experience - OTel or other - you can work through
| these things that might confound a neophyte.
| morelisp wrote:
| The problem is they keep making the OTel tooling worse for
| working through these things, because the people writing the
| OTel tooling broadly aren't the people actually trying to
| monitor things. Even before OTel, plain Prometheus client
| libraries suffered from this.
| coxley wrote:
| Agreed. Some things they suggest aren't actually possible with
| their SDKs.
|
| For example, you cannot define a histogram's buckets near where
| you define the histogram. You have to give the global exporter
| (or w/e the type is) a list of "overrides" that map each
| histogram name => their buckets. This makes it extremely ugly
| when you have libraries that emit metrics.
|
| https://github.com/open-telemetry/opentelemetry-go/issues/38...
| nickstinemates wrote:
| I run a full OSS otel stack in my home lab. It was a lot of fun
| to set up, with redundancy and all of the extras. It hasn't been
| particularly useful, but centralized logging and metrics are
| pretty to look at in dashboards, which was my motivation.
|
| Let's hear some great debugging stories that have been powered by
| OTEL. I'd love to hear from the horses mouth without marketing
| speak how it was worth $$$$ collecting and storing all of this
| info.
| pradn wrote:
| Most of the comments on this thread are talking about using
| OpenTelemetry to send metrics/logs to self-hosted collector jobs.
| Though using a standard library supported by a bunch of collector
| tools like ClickHouse is useful it itself, there's another
| benefit too. The specification allows transferring trace IDs
| across system boundaries. If you and your dependencies all
| implement the OpenTelemetry spec, then you get spans that reveal
| in granular detail what happened in the journey. For example, you
| could learn that it was your database loading a page from disk
| that took so long, or that a Cloud service's metadata plane was
| the responsible span for high latency.
| phito wrote:
| OTEL is awesome. I was part of the team integrating it at my job,
| it went really smoothly, and it saved us so much time debugging
| our microservice application.
|
| I would say the hardest part was getting other devs to use it, a
| lot of them are stuck in their own way and did not want to go
| through the relatively small learning curve...
| dottedmag wrote:
| Ugh, protobuf and gRPC as blessed (?!) transports? Thanks, I
| always wanted to bring 100MB dependency into my code just to send
| metrics.
| Xeoncross wrote:
| If you're using Rust, Go, C, Java, or some other performant
| language then it shouldn't be much. If you're using a slower
| scripting language like Ruby or JavaScript then you might have
| issues.
| kortex wrote:
| Where are you getting 100MB from? You don't need the whole
| protoc toolchain. Well I've never worked with C or C++ and
| gRPC, but insofar as rust, go, and python are concerned, the
| increase in container/binary size is a few MB.
| podoman wrote:
| Just a quick clarification. OTEL is not just a transport, its a
| specification.
| wdb wrote:
| Yeah, but you have the same problem when using Prometheus and
| the native histograms? So far I know it is not supported by the
| text / openmetrics format only in their protobuf-basaed format
| version
| kevinslin wrote:
| author of the post here - was inspired to write this post after
| working with OTEL for a few months - realized that OTEL had a
| ridiculously large surface area that most people (at least
| myself) might not be aware of
|
| I see a lot of comments about how overly complex OTEL is. I don't
| disagree with this. in some sense, OTEL is very much the k8 for
| observability (good and bad)
|
| The good is that it is a standard that can support every
| conceivable use case and has wide industry adoption The bad is
| that there is inherent complexity in needing to support the wide
| array of use cases
| jensneuse wrote:
| We're soon launching a GraphQL Analytics, Tracing and Metrics
| stack on top of OTEL. We've built a custom OTEL exporter to
| Clickhouse in go, so we can export OTEL traces and Prometheus
| metrics all to Clickhouse. We've built this stack for our
| federated GraphQL Gateway but it could essentially work for any
| OTEL service. If you want to learn more, here's some info:
| https://wundergraph.com/cosmo Were soon going to open source
| this, so just follow me/us of you're interested.
|
| What I really like about this stack is that you can use our end
| to end solution, but you're not locked into it. We provide a full
| service, but you can also just use your own OTEL backend if you
| want to eject.
| jeffchao wrote:
| Otel is great for avoiding vendor lock-in and the spec stability
| is a long time coming -- great to see. Also good there's
| architecture options aside from side car (there's also proxy
| mode), which is nice for situations like deploying on a PaaS.
|
| That said the main trouble I've had in the past is instability in
| the SDKs. Java one's decent. Go, not so much, for example.
| Traditionally Otel has been awesome for tracing but not so much
| logs, events, and even metrics (obviously depending on your
| language).
|
| Other than that, auto-instrumentation has been nice. We had this
| exact functionality when I worked on observability at Netflix and
| made starting up and maintaining microservices easy and really
| helped with adoption.
| renke1 wrote:
| A few of my colleagues and I had the silly (?) idea that you
| don't really need logs anymore. Instead of log messages you just
| attach span events [0]. You then just log the span title and a
| link to that span in Jaeger; something like [1]. I've only really
| tried that in my private project, but it felt pretty good. The UI
| of Jaeger could be a bit better to support that usage, though.
|
| Edit: Actually, those colleagues are doing a talk about that
| topic. So, if you are in Germany and Hannover area, have a look
| at [2] and search for "Nie wieder Log-Files!".
|
| [0]:
| https://opentelemetry.io/docs/instrumentation/ruby/manual/#a...
|
| [1]: tracing.ts:38 Usecase: Handle Auth
| tracing.ts:47
| http://localhost:16686/trace/ec7ffb1e23ddbb8dd770a3f08028666b
| tracing.ts:38 Adapter: Find Personal Board tracing.ts:47
| http://localhost:16686/trace/e22d342316ab0d7d23230864008e27bc
| tracing.ts:38 Adapter: Find Starred Board List
| tracing.ts:47
| http://localhost:16686/trace/129f89cee26d54cfdc38abea368d9b4e
| tracing.ts:38 Adapter: Find Personal Board List
| tracing.ts:47
| http://localhost:16686/trace/97948127d77501ff0c65a5db21b21b5a
|
| [2]: https://javaforumnord.de/2023/programm/
| klysm wrote:
| I've thought about doing this as well, but I like being able to
| use dumb tools to get an idea of what's going on. There's a lot
| that has to be working correctly to use traces. Or maybe I'm
| just scared of the tooling because I don't have enough
| experience with it yet idk
| pquerna wrote:
| You can also just, Log the spans as they are being created to
| stderr/stdout -- I've done this on a previous project with
| this approach of "spans first".
|
| It made it debuggable via output if needed, but the primary
| consumption became span oriented.
| klysm wrote:
| Good idea yeah, but do the same notions of log level apply?
| kiitos wrote:
| You don't even really need to ship traces anywhere. You can
| just keep them in-process, and build an API on top of that in-
| memory trace data.
| phillipcarter wrote:
| Depending on how greenfield a project is, you don't even need
| span events unless you absolutely require a timestamp for a
| specific operation with no duration. Just using spans for every
| meaningful operation is like having more powerful structured
| logs.
|
| This approach isn't possible for a lot of systems that have
| existing logs they need to bring along, but if you're
| greenfield enough, I'd recommend it.
| glenjamin wrote:
| Using only spans is surprisingly effective!
|
| I did a talk about this at QCon London last year
|
| https://www.infoq.com/presentations/event-tracing-
| monitoring...
| paulddraper wrote:
| Agreed.
|
| Same is true for metrics derived from spans. (Though for
| metrics, you don't need to sample, and for spans you might.
| So keep in mind.)
| morelisp wrote:
| This was the idea behind Stripe's Veneur project - spans, logs,
| and metrics all in the same format, "automatically" rolling up
| cardinality as needed - which I thought was cool but also that
| it would be very hard to get non-SRE developers on board with
| when I saw a talk about it a few years ago.
|
| https://github.com/stripe/veneur
| enahs-sf wrote:
| Someone needs to build the Segment of APM. Datadog lock-in is
| real and the bill is high.
| waynesonfire wrote:
| like a virus, the complexity spreads to fill head count.
| zaat wrote:
| Like a reflex, comments blaming solutions for complex
| challenges are being emitted. Alas, no alternatives are ever
| offered. Shall we limit all technology to what can be contained
| in a single box? should we just ban countries, companies and
| communities from growing past the 500k head count? Please, go
| on, do say what shall we do with the inherent complexity of the
| universe, shall we stop physics at Newton and ban particle
| accelerators? De-legalize storage systems exceeding petabyte?
| Please, let us hear how complexity can be put back to its old
| box
| nerdjon wrote:
| Slight tangent for a moment, I really really hate the "subscribe"
| popup that comes up on this blog. It is not clear at all that you
| can just close it and not give your email since there is no "x"
| button. Instead it has the incredibly unintuitive "continue
| reading" under the subscribe button that I did not think would
| work. Clicking out of it also did not work. Seriously we can and
| should be better than this.
|
| On the topic of open telemetry. I have been long wanting to play
| with it and see if it offers all of the capabilities when we send
| the data to datadog. But I have been reluctant to add in another
| thing to manage and train on if it means that the datadog agent
| is still necessary for anything outside of the basics.
|
| Has anyone else actually tried hooking this up to datadog?
|
| Edit: just to be clear, my driving goal of this is not
| necessarily to keep it with datadog. But that is currently where
| much of our alerting and logs are now. So the idea would be to
| switch to open telemetry which would then allow us to
| (theoretically) move to something else down the line.
| monocasa wrote:
| WRT the popup, that's a medium.com thing. I agree that it's
| very annoying.
| luispauloml wrote:
| It's not Medium, it's Substack.
| baz00 wrote:
| Is there yet any way of having a front end for this which doesn't
| significantly dent your revenue stream either in staffing,
| infrastructure or license fees? We land over 2000 requests/second
| and it's expensive just keeping logs.
| memset wrote:
| I am building https://scratchdb.com/ to address this. It's an
| HTTP wrapper around Clickhouse, and uses pay-as-you-go pricing.
| It's also open source and easy to host yourself (single go
| binary.) Currently have users sending in the 1000 requests per
| second so definitely have capacity.
|
| Typically seeing a 0.1 compression ratio on data before other
| optimizations.
|
| I have it connected to Fly.io here:
| https://scratchdb.com/blog/fly-logs-to-clickhouse/
|
| I'd be really grateful to learn more about what you're looking
| for (how are you even managing logs today?) Even if you end up
| not using scratchdb it'll help me figure out the next thing to
| build!
| singron wrote:
| I recommend sampling traces if you aren't. I've been
| unimpressed with datadog apm, which has no affordable
| configuration. We've been running our own Jaeger stack with
| 0.1% sampling, and it's negligible to run compared to datadog
| apm.
|
| For metrics and logs, sampling isn't so useful, so I don't have
| a good answer. Datadog has 80% gross margin, so at most 20% of
| what you pay them is the infra, so you stand to save a lot of
| money running your own open source stacks if your labor costs
| would be less than that 80%. With datadog, we have a project
| every 3 months to reduce usage, so it's not like we aren't
| constantly babysitting it anyway.
| phamilton wrote:
| > I've been unimpressed with datadog apm, which has no
| affordable configuration
|
| FWIW we sample in datadog APM and it works fine to control
| costs, I'm not sure what issues you hit.
| pranay01 wrote:
| If you're looking for open source APM stack which is
| OpenTelemetry native and you can self host - you can check
| out SigNoz ( https://github.com/SigNoz/signoz)
| paulddraper wrote:
| > datadog apm, which has no affordable configuration
|
| In my experience, Datadog's ingestion sampling works pretty
| well.
|
| And there's retention filters you can use to override.
| pondidum wrote:
| Honeycomb's pricing is pretty reasonable - but looking at your
| volume, some sampling might help also
| yourapostasy wrote:
| If you're using Honeycomb, how did you find their solution's
| developer/DevOps engineer ergonomics, their support, and
| their overall experience?
| pondidum wrote:
| The only thing I can recall being a bit of a stumbling
| point at the beginning was finding "my" trace from a local
| environment. I just didn't know where to find it. Once I
| figured that out I've not had problems.
|
| The high-level view and being able to draw a box around
| something that looks weird on my graph, and honeycomb tells
| me what's different inside and outside the box is amazing
| (called bubble up, if you're searching).
|
| It's faster than any other tool I've used; usually data is
| available by the time I've switched to their ui from the
| curl command to our API. It's mind blowing actually.
|
| Other saas and self hosted options I've tried have all been
| awful in some way or other; honeycomb is a breath of fresh
| air, and going back to other tools after using honeycomb is
| painful.
|
| I'm not sponsored or working for them btw, they just make
| one of the few products that I genuinely love using.
| AndreasHae wrote:
| Super nitpick, but you meant profit, not revenue, right?
| secondcoming wrote:
| Why is this downvoted? This is a pain point for us too, except
| we're at 500k requests/second. We're currently Datadog but
| everyone knows they're too expensive.
| spockz wrote:
| We trace everything and first collects all spans in their
| trace. Then we sample the successful traces and we retain all
| traces ending in a failed state. This greatly reduces the
| need for storage.
| phillipcarter wrote:
| Cost management via sampling is still largely a vendor
| concern. Each vendor has a different solution, and while some
| are better than others, all can be effective at bringing
| costs down.
| lukaszwojtow wrote:
| Try coralogix.
| jdwyah wrote:
| I'm working on giving you more options for saving money on
| logging with dynamic log levels:
| https://prefab.cloud/features/log-levels/
|
| The hypothesis being that you can save money by turning things
| down, but easily turn them back up when you're actively
| investigating. Or turn the volume up for a targeted sub-segment
| of your traffic.
|
| We've done some exploration into providing the same for APM and
| the rest of OTEL and I think it's pretty doable. hmu if you
| want to talk.
| candiddevmike wrote:
| I would ask this a different way--is there a single OSS project
| that handles collecting all of the OTEL metrics/logs/traces?
| Folks keep saying you can't do this in data store/format,
| Elastic seems to, and having to manage separate
| storage/infrastructure for all of these tools is intense.
|
| Could Prometheus be augmented to store metrics, logs, and
| traces somehow? I don't really mind if it doesn't scale well on
| a single instance or is highly available, I'll just add more
| instances and aggregate them.
| kortex wrote:
| It's really not _that_ intense. I basically set up my last co
| 's telemetry infrastructure all by myself, using terraform,
| otel-python, jaeger, and AWS elasticsearch.
|
| This TF project does most of the heavy lift.
| https://github.com/telia-oss/terraform-aws-jaeger
| candiddevmike wrote:
| Jaeger for tracing, Elasticsearch for logs? What are you
| using for metrics?
| mmclean wrote:
| Likely Prometheus - Jaeger for tracing, Elastic for logs,
| Prometheus for metrics is a pretty common and effective
| OSS observability stack
| Svenstaro wrote:
| I guess you could take a look at this:
| https://openobserve.ai/
|
| It's in Rust to add some HN catnip.
| candiddevmike wrote:
| Why .ai?
| CubsFan1060 wrote:
| I've been watching this with a lot of interest:
| https://signoz.io/
| candiddevmike wrote:
| That looks really promising, thank you for sharing it
| pranay01 wrote:
| Great to hear that. I am one of the maintainers at SigNoz
| - if you have any queries as you implement , do ask in
| our slack - https://signoz.io/slack
| rdoherty wrote:
| Grafana seems to be an option? Handles metrics, logs and
| traces. I don't know what storage costs look like though if you
| are self hosting.. https://grafana.com/
| Thaxll wrote:
| Sampling is the answer. Sample 1% of success and all the
| errors.
|
| Cost is one thing but you would be surprised how heavy
| observability can be on a service, it's uses a lot of %cpu.
| [deleted]
| kiitos wrote:
| Sampling is fine at the query layer, but if you sample at the
| ingest layer -- and therefore drop a majority of your
| telemetry data outright -- that's a total hack, and I can't
| see how the resulting system is anything but unsound.
| BHSPitMonkey wrote:
| You use sampling rates that are statically significant
| enough for the system you're observing. You can always make
| exceptions to the default rate based on other heuristics.
| What's wrong with that, for the kinds of insights it
| provides?
| kiitos wrote:
| General-purpose observability systems serve two use
| cases: presenting a high-level summary of system
| behavior, _and_ allowing operators to inspect telemetry
| data associated with a specific e.g. request.
|
| The former use case is often solved by a specific and
| narrow kind of observability data, which is metrics. A
| common tool for that purpose is Prometheus. You certainly
| can't query Prometheus for individual requests, which is
| fine, and accepting that invariant allows Prometheus to
| treat input data as statistical, in the sense that you
| mean in your comment.
|
| But if we're talking about general-purpose telemetry,
| we're talking about more than just high-level summaries
| of system behavior, we also need to be able to inspect
| individual log events, trace spans, etc. If a user made a
| request an hour ago with reqid 123, I expect to be able
| to query my telemetry system for reqid 123 and see all of
| the metadata related to that request.
|
| A telemetry system that samples prior to ingest certainly
| delivers value, but it can only ever solve the first use
| case, and never the second.
| number101010 wrote:
| Most observability vendors support OTEL at this point. To plug
| the OSS project I work on that supports OTEL ingestion:
|
| https://github.com/grafana/tempo/
| baz00 wrote:
| Thanks I had no idea Tempo even existed despite using
| Grafana. Will read into it.
| candiddevmike wrote:
| IMO, Jaeger is easier to setup/manage and has a better
| interface than Grafana/Tempo. It's easy to add Jaeger to
| your local dev stack so you can have tracing while
| developing.
|
| FWIW I now use Tempo because I have everything else in
| Grafana (Prometheus, Loki), but I do miss using Jaeger.
| number101010 wrote:
| > It's easy to add Jaeger to your local dev stack so you
| can have tracing while developing.
|
| Tempo can be spun up with docker compose using a local
| disk for ephemeral storage/querying: https://github.com/g
| rafana/tempo/blob/main/example/docker-co...
|
| Maybe this meets your needs?
|
| > Jaeger is easier to setup/manage and has a better
| interface than Grafana/Tempo
|
| What do you enjoy about the Jaeger interface? Perhaps
| it's a gap in Tempo we can improve.
| sofixa wrote:
| Jaeger can use multiple backends for storage, including
| Tempo, so it's not an either/or situation.
|
| I'm fairly sure there was an official Grafana-provided
| Jaeger gRPC plugin for Tempo, but can't easily find it,
| only this one: https://github.com/flitnetics/jaeger-tempo
| kortex wrote:
| Define "significant". At $lastco, we routed traces to Cassandra
| and stored them in an AWS elasticsearch domain. Jaeger was used
| to visualize traces. We also wrote some elasticsearch queries
| to generate some basic reports, eg finding the most sluggish
| queries. Pretty standard stuff if you follow the OTEL/jaeger
| tutorials.
|
| Traces came on the order of hundreds/second, but we didn't have
| downsampling turned on, just collected all. Traces were saved 7
| days (configurable). Very little actual optimization at the
| point where I left.
|
| I think it cost on the order of dozens to hundreds of dollars a
| month.
|
| There's an environment variable which you can set on your
| containers which defines how the tracing sampler behaves. It's
| in the docs. See _OTEL_TRACES_SAMPLER_
|
| https://opentelemetry.io/docs/specs/otel/configuration/sdk-e...
| pphysch wrote:
| So I have a VictoriaMetrics (i.e. Prometheus) setup that I am
| happy with, but haven't touched OpenTelemetry; why should I care
| about it? Is it a serious solution for log aggregation, but do I
| need a separate database for logs?
| hagen1778 wrote:
| You shouldn't unless you want to use the new open source
| standard for telemetry. You won't benefit from simplicity or
| performance improvements. It would be quite the opposite. You
| can check what is the actual cost of open telemetry adoption
| here [0]
|
| But if you ever decide to go this path - VictoriaMetrics
| supports OpenTelemetry protocol for metrics [1]
|
| [0]
| https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2570
|
| [1] https://docs.victoriametrics.com/Single-server-
| VictoriaMetri...
| kiitos wrote:
| OpenTelemetry is a marketing-driven project, designed by
| committee, implemented naively and inefficiently, and guided by
| the primary goal of allowing Fortune X00 CTOs to tick off some
| boxes on their strategy roadmap documents.
|
| It's not something that anyone with a choice in the matter should
| be using.
| anbotero wrote:
| What should people use?
|
| With basic parameters in place so it doesn't eat your billing,
| it's been working great with me for years. Initially with New
| Relic, then Datadog, now a setup with OpenTelemetry is good
| enough.
| kiitos wrote:
| Instrumentation isn't solved by any single specific thing.
| It's a praxis that you apply to your code as your write it,
| like I guess error handling; it's not a product that you can
| deploy, like I guess Splunk or New Relic or whatever else.
|
| You should "use" metrics, logs, and traces thru dependencies
| that are specific to your organization. The interface between
| your business logic and operational telemetry should be
| abstract, essentially the same as a database or a remote HTTP
| endpoint or etc. The concrete system(s) collecting and
| serving telemetry data are the responsibility of your dev/ops
| or whatever team.
|
| Main point: instrumentation is part of the development
| process, not something that's automatic or that can be
| bolted-on.
| pgwhalen wrote:
| Have you worked with OTEL before? Basically all of your
| points about instrumentation are actually quite sympathetic
| to OTEL's view of the world. The whole point of OTEL is to
| provide some standards around how these pieces fit together
| - not to solve for them automatically.
| kiitos wrote:
| I've been deeply involved with OTel from even before it
| was a CNCF jam. My experiences with the project, over
| time, have made me basically abandon the project as
| unsound and infeasible since a year or two. Those
| experiences also inform comments like the ones I've made
| here.
| pgwhalen wrote:
| Can you elaborate on what unsound and infeasible mean?
| I'm newer to OTel than you (~6 months of working with it
| in depth), and don't really understand what you're
| getting at. It's solving real problems in my
| organization, with only a "regular" amount of pain for a
| component of its size.
| klysm wrote:
| Okay so what's the interface? Sounds like what OTEL
| provides to me
| kiitos wrote:
| There are well-defined interfaces for specific sub-
| classes of telemetry data. Prometheus provides a set of
| interfaces for metrics which are pretty battle-tested by
| now. There are similar interfaces for logs and traces,
| authored by various different parties, and with various
| different capabilities, trade-offs, etc.
|
| There is no one true interface! The interface is a
| function of the sub-class of telemetry data it serves,
| the specific properties of the service(s) it supports,
| the teams it's used by, the organization that maintains
| it, etc. etc.
|
| OTel tries to assert a general-purpose interface. But
| this is exactly the issue with the project. That
| interface doesn't exist.
| klysm wrote:
| OTEL is a set of interfaces, so I'm not sure your last
| point applies. I do agree that battle tested things like
| Prometheus work great, but why not have a set of
| standardized interfaces? There is clearly a cost to
| having them; for some projects this may be too much. For
| the projects I've used it in it let me spin up all the
| traces and telemetry without thinking hard.
| KronisLV wrote:
| > What should people use?
|
| I recall Apache Skywalking being pretty good, especially for
| smaller/medium scale projects: https://skywalking.apache.org/
|
| The architecture is simple, the performance is adequate, it
| doesn't make you spend days configuring it and it even
| supports various different data stores: https://skywalking.ap
| ache.org/docs/main/v9.5.0/en/setup/back...
|
| The problems with it are that it isn't super popular
| (although has agents for most popular stacks), the docs could
| be slightly better and I recall them also working on a new UI
| so there is a little bit of churn:
| https://skywalking.apache.org/downloads/
|
| Still better versus some of the other options when you need
| something that just works instead of spending a lot of time
| configuring something (even when that something might be
| superior in regards to the features): https://github.com/apac
| he/skywalking/blob/master/docker/dock...
|
| Sentry comes to mind (OpenTelemetry also isn't simpler due to
| how much it tries to do, given all the separate parts),
| compare its complexity to Skywalking:
| https://github.com/getsentry/self-
| hosted/blob/master/docker-...
|
| I wish there was more self-hosted software like that out
| there, enough to address certain concerns in a simple way on
| day 1 and leave branching out to more complex options like
| OpenTelemetry once you have a separate team for that and the
| cash is rolling in.
| podoman wrote:
| > OpenTelemetry is a marketing-driven project, designed by
| committee, implemented naively and inefficiently, and guided by
| the primary goal of allowing Fortune X00 CTOs to tick off some
| boxes on their strategy roadmap documents.
|
| I'm the founder of highlight.io. On the consumer side as a
| company, we've seen a lot of value of from OTEL; we've used it
| to build out language support for quite a few customers at this
| point, and the community is very receptive.
|
| Here's an example of us putting up a change:
| https://github.com/open-telemetry/opentelemetry-js/pull/4049
|
| Do you mind sharing why you think no-one should be using it?
| Some reasoning would be nice.
| dboreham wrote:
| Well, roll up your sleeves and fix the performance bugs that
| affect you (source: I have).
| kiitos wrote:
| I have no reason to do so, because I don't believe that
| OpenTelemetry is a project that was created, or is
| maintained, in good faith to its stated goals.
| Dopameaner wrote:
| Care to elaborate a bit more on the goals contrast?
| l-albertovich wrote:
| That's overly harsh, they are doing good work I think their
| data model is a step forward in the right direction.
|
| Their processors are quite capable and the entire receiver and
| exporter contrib collection is pretty good.
|
| I'm not saying it's the best solution out there because that
| clearly depends on each use case but I don't think such harsh
| criticism makes sense.
|
| Disclaimer: I'm part of the fluent-bit maintainer team.
| jrockway wrote:
| I don't think that's true. It seems like it's more of a "oh
| shit, all this open source software emits Prometheus metrics
| and Jaeger traces, but we want to sell our proprietary
| alternatives to these and don't want to upstream patches to
| every project". (Datadog had a literal army of people adding
| datadog support to OSS projects. Honestly, probably a great
| early-career job; diving into unfamiliar codebases is a
| superpower.)
|
| OTel lets the open source projects use an abstraction layer so
| that you can buy instead of self-host.
|
| None of this has ever made me feel super great, but in the end
| I would probably consider OTel today for services that people
| other than my company operate. That way if some user wants to
| use Datadog, we're not in their way.
|
| I used OTel in the very very early days and was rather
| disappointed; the Go APIs were extremely inefficient (a
| context.Context is needed to increment a counter? no IO in my
| request path please), and abstracted leakily (no way to set
| histogram buckets when exporting to Prometheus). I assume they
| probably fixed that stuff at some point, though.
| morelisp wrote:
| OTel is the backend, in-program equivalent of "we need all of
| five analytics systems on our frontend to figure out that
| users bounce when our page takes 10s to load because it has
| five analytics systems in it".
| the_duke wrote:
| What helps hosted data collectors helps self-hosting setups
| just as much.
|
| More and more solutions are getting built in OTEL support,
| which means you can relatively seamlessly switch between
| backends without changing anything in your application code.
| morelisp wrote:
| This only makes sense if you're in a world where you're
| switching backends more than once, which means you're not
| seriously programming, you're just burning VC money for
| lottery tickets.
| jrockway wrote:
| I agree with this. For internal apps, pick a system and
| stick with it. If you're excited by Datadog's marketing
| pitch, just buy it and use it. It will not make or break
| your startup; like if your Datadog bill is what's
| standing between you and profitability, then you probably
| didn't actually find product/market fit. Switching to
| Prometheus at that point also won't help you find
| product/market fit.
|
| In the 2 jobs where I've set up the production
| environment, I just picked Prometheus/Jaeger/cloud
| provider log storage/Grafana on day 1 and have never been
| disappointed. You explode the helm chart into your
| cluster over the course of 30 minutes, and then move on
| to making something great (or spending a week debugging
| Webpack; can't help you with that one).
| hankchinaski wrote:
| I kind of agree with you. Clueless managers just asking "open
| telemetry" on the roadmap without contextualising
| costs/benefits
| brainbag wrote:
| It looks like every other comment in this thread is favorable
| to very positive, can you go into more detail about what
| specifically isn't good about it?
| arp242 wrote:
| Not the previous poster, but I had to implement it a few
| years ago, and I found it unbelievable complex with dense and
| difficult to read specifications. I've implemented plenty of
| protocols and formats from scratch using just the
| specification, but rarely have I had such difficulty than
| with OpenTelemetry.
|
| I guess this is something you don't notice as merely a
| "user", but IMHO it's horribly overengineered for what it
| does and I'm absolutely not a fan.
|
| I also disliked the Go tooling for it, which is "badly
| written Java in Go syntax", or something along these lines.
|
| This was 2 years ago. Maybe it's better now, but I doubt it.
|
| In our case it was 100% a "tick off some boxes on their
| strategy roadmap documents" project too and we had much much
| better solutions.
|
| OTel is one of those "yeah, it works ... I guess" but also
| "ewwww".
| phillipcarter wrote:
| I'd recommend trying it out today. In 2021, very few things
| in OTel were GA and there wasn't nearly as much automatic
| instrumentation. One of the reasons why you had to dive
| into the spec was because there was also very little
| documentation, too, indicative of a heavily in-progress
| project. All of these things are now different.
| arp242 wrote:
| I'll be happy to take your word that some implementation
| issues are now improved, but things like "overengineered"
| and "way to complex for what it needs to do" really are
| foundational, and can't just be "fixed" without starting
| from scratch (and presumably this is all by design in the
| first place).
| phillipcarter wrote:
| That's fair. I find that to be a bit subjective anyways,
| so I don't have much to comment on there. Most languages
| are pretty lightweight. For example, initializing
| instrumentation packages and creating some custom
| instrumentation in Python is very lightweight. Golang is
| far more verbose, though. I see that as part and parcel
| of different cultures for different languages, though
| (I've always loved the brevity of Python API design and
| disliked the verbosity of Go API design).
| infogulch wrote:
| Telemetry seems like it would be a great candidate for columnar
| storage formats like Parquet or arrow. In particular I expect
| that it would compress very well, which could reduce telemetry
| bandwidth consumption / allow for a bigger sample rate.
|
| Does anybody have any experience with the intersection of these
| technologies?
| solumos wrote:
| Honeycomb built their own columnar database[0] to support their
| product.
|
| [0] - https://www.honeycomb.io/resources/why-we-built-our-own-
| dist...
| annanay wrote:
| Grafana Tempo also switched from Protobuf storage format to
| Apache Parquet last year. It's fully open source, and the
| proposal (from April 2022) is here:
| https://github.com/grafana/tempo/blob/main/docs/design-propo...
|
| The relevant code for parquet storage backend can be found
| here:
| https://github.com/grafana/tempo/tree/main/tempodb/encoding
|
| disclosure: I work for Grafana!
| infogulch wrote:
| Cool thanks for sharing. Can you say something about how it's
| worked out? Has it reduced bandwidth or CPU usage?
| vmarchaud wrote:
| Someone from F5 worked on this with OpenTelemetry [0] for
| Arrow, another effort was done for Parquet but was dropped [1]
|
| [0]: https://github.com/open-telemetry/oteps/pull/171
|
| [1]: https://github.com/open-telemetry/opentelemetry-
| proto/pull/3...
| infogulch wrote:
| Oh nice, thank you (and also solumos) for the links! It looks
| like oteps/pull/171 (merged June 2023) expanded and
| superseded the opentelemetry-proto/pull/346 PR (closed Jul
| 2022) [0]. The former resulted in merging OpenTelemetry
| Enhancement Proposal 156 [1], with some interesting results
| especially for 'Phase 2' where they implemented columnar
| storage end-to-end (see the Validation section [2]):
|
| * For univariate time series, OTel Arrow is 2 to 2.5 better
| in terms of bandwidth reduction ... and the end-to-end speed
| is 3.1 to 11.2 times faster
|
| * For multivariate time series, OTel Arrow is 3 to 7 times
| better in terms of bandwidth reduction ... Phase 2 has [not
| yet] been .. estimated but similar results are expected.
|
| * For logs, OTel Arrow is 1.6 to 2 times better in terms of
| bandwidth reduction ... and the end-to-end speed is 2.3 to
| 4.86 times faster
|
| * For traces, OTel Arrow is 1.7 to 2.8 times better in terms
| of bandwidth reduction ... and the end-to-end speed is 3.37
| to 6.16 times faster
|
| Pretty exciting results! The OTEL-Arrow adapter has
| subsequently been donated to the otel community; here's a
| comment that does a good job of summarizing the results and
| the recommendations that came out of the test [3].
|
| [0]: https://github.com/open-telemetry/opentelemetry-
| proto/pull/3...
|
| [1]: https://github.com/open-
| telemetry/oteps/blob/main/text/0156-...
|
| [2]: https://github.com/open-
| telemetry/oteps/blob/main/text/0156-...
|
| [3]: https://github.com/open-
| telemetry/community/issues/1332#issu...
| podoman wrote:
| We're big users of clickhouse at https://highlight.io. Some
| more details here if you're interested:
| https://www.highlight.io/blog/how-we-built-logging-with-clic...
| candiddevmike wrote:
| CNCF needs an OTEL log aggregator
| lokar wrote:
| The otel collector/agent can do this. Not storage/query, but
| aggregation and processing.
| [deleted]
| chrisweekly wrote:
| Non-paywalled URL: https://archive.is/NULpZ
| TwentyPosts wrote:
| Can someone tell me roughly at which point a tool like
| OpenTelemetry becomes interesting? It seems complicated, at which
| point should you bother with it instead of just handrolling
| simple stuff and eyeballing it?
| paulddraper wrote:
| OpenTelemetry is several different things, so you'd have to be
| more specific.
|
| But, for example, if you write a library and you want your
| downstream users to be able to see the telemetry. OpenTelemetry
| provides a standardized interface, so you don't need to make
| assumptions.
| skrtskrt wrote:
| since I added OTEL instrumentation to an app, I already easily
| swapped out the backends from Prometheus and Jaeger to Grafana
| Mimir and Tempo.
|
| The lack of lock-in is fantastic, don't know if I've ever just
| switched technologies that easily, even SQL databases.
___________________________________________________________________
(page generated 2023-08-28 23:01 UTC)