[HN Gopher] LLM Observability in the Wild - Why OpenTelemetry Sh...
___________________________________________________________________
LLM Observability in the Wild - Why OpenTelemetry Should Be the
Standard
Author : pranay01
Score : 46 points
Date : 2025-09-27 18:56 UTC (4 hours ago)
(HTM) web link (signoz.io)
(TXT) w3m dump (signoz.io)
| CuriouslyC wrote:
| A full observability stack is just a docker compose away: Otel +
| Phoenix + Clickhouse and off to the races. No excuse not to do
| it.
| perfmode wrote:
| Phoenix as in Elixir?
| mindcrime wrote:
| I imagine they meant:
|
| https://github.com/Arize-ai/phoenix
| pranay01 wrote:
| one of the cases we have observed is that Phoenix doesn't
| completely stick to OTel conventions.
|
| More specifically, one issue I observed is how it handles span
| kinds. If you send via OTel, the span Kinds are classified as
| unknown
|
| e.g. The Phoneix screenshot here - https://signoz.io/blog/llm-
| observability-opentelemetry/#the-...
| CuriouslyC wrote:
| If it doesn't work for your use case that's cool, but in
| terms of interface for doing this kind of work it is the
| best. Tradeoffs.
| 7thpower wrote:
| I've found phoenix to be a clunky experience and have been
| far happier with tools like langfuse.
|
| I don't know how you can confidently say one is "the best".
| a_khan wrote:
| Curious what you prefer from langfuse over Phoenix!
| ijk wrote:
| Spans labeled as 'unknown' when I definitely labeled them in
| the code is probably the most annoying part of Phoenix right
| now.
| dcreater wrote:
| Is phoenix really the no-brainer go to? There are so many
| choices - langfuse, w&b etc.
| ram_rar wrote:
| The article makes a fair case for sticking with OTel, but it also
| feels a bit like forcing a general purpose tool into a domain
| where richer semantics might genuinely help. "Just add
| attributes" sounds neat until you're debugging a multi-agent
| system with dynamic tool calls. Maybe hybrid or bridging
| standards are inevitable?
|
| Curious if others here have actually tried scaling LLM
| observability in production like where does it hold up, and where
| does it collapse? Do you also feel the "open standards" narrative
| sometimes carries a bit of vendor bias along with it?
| mrlongroots wrote:
| I think standard relational databases/schemas are underrated
| for when you need richness.
|
| OTel or anything in that domain is fine when you have a
| distributed callgraph, which inference with tool calls does. I
| think the fallback layer if that doesn't work is just say
| Clickhouse.
| bfung wrote:
| TL;DR - follow https://opentelemetry.io/docs/specs/semconv/gen-
| ai/
| olliem36 wrote:
| We've built a multi-agent system, designed to run complex tasks
| and workflows with just a single prompt. Prompts are written by
| non-technical people, can be 10+ pages long...
|
| We've invested heavily in observability having quickly found that
| observability + evals are the cornerstone to a successful agent.
|
| For example, a few things measure:
|
| 1. Task complexity (assessed by another LLM) 2. Success metrics
| given the task(s) (Agin by other LLMS) 3. Speed of agent runs &
| tools 4. Errors of tools, inc time outs. 5. How much
| summarizaiton and chunking occurs between agents and tool results
| 6. tokens used, cost 7. reasoning, model selected by our dynamic
| routing..
|
| Thank god its been relatively cheap to build this in house.. our
| metrics dashboard is essentially a vibe coded react admin site..
| but proves absolutely invaluable!
|
| All of this happed after a heavy investment in agent
| orchestration, context management... it's been quite a ride!
___________________________________________________________________
(page generated 2025-09-27 23:00 UTC)