[HN Gopher] Some notes on Grafana Loki's new "structured metadata"
___________________________________________________________________
Some notes on Grafana Loki's new "structured metadata"
Author : valyala
Score : 106 points
Date : 2025-03-16 10:38 UTC (3 days ago)
(HTM) web link (utcc.utoronto.ca)
(TXT) w3m dump (utcc.utoronto.ca)
| kbouck wrote:
| It's also not ideal to have a different query language for
| different Grafana datastores (LogQL, PromQL, TraceQL). Are there
| any plans on making a unified Grafana query language?
| pjd7 wrote:
| Unifying things slows engineers down, so probably not (for some
| time).
| Wheaties466 wrote:
| well, not having a unified language slows down things on the
| other end.
| jakozaur wrote:
| There is an effort in OpenTelemetry to create a standard query
| language for observability. There were a lot of discussions
| with a lot of opinions; there were even several talks during
| KubeConEU about that:
|
| https://sched.co/1tcyx
|
| https://sched.co/1txI1
|
| We are still waiting for a compelling implementation that will
| show the way.
| ople wrote:
| Why not just use SQL? With LLMs evolving to do sophisticated
| text-to-SQL, the case for a custom language for the sake of
| simplicity is diminishing.
|
| I think that expressiveness, performance and level of fluency
| by base language models (i.e. the amount of examples in
| training set) are the key differentiators for query languages
| in the future. SQL ticks all those boxes.
| pbh101 wrote:
| I think I'm probably not interested in this. PromQL is already
| relatively dense to learn, but reasonably well fit to the
| domain model and internally consistent, unlike most other
| metric querying tools I've tried over the years.
|
| Maybe that would work as well with traces and logs but IMO the
| problem space is quite different and not sure how much value
| we'd get from a unified language where some subsets only apply
| to parts, ie traces and logs and metric, as opposed to
| spiritually similar but distinct languages.
| DeathArrow wrote:
| Did someone use both Grafana Loki and Kibana? Does it have any
| advantages over Kibana? I am mostly interested in resource usage
| and versatility of filtering.
|
| In Kibana, if something is there I will find it with ease and it
| doesn't take a lot of time to investigate issues in a
| microservice based application. It is also quite fast.
| weitzj wrote:
| From the Enterprise Perspektive at least for my use cases(fine
| grained permissions using extra id) , elasticsearch with kibana
| always had a solution available.
|
| For grafana cloud and Loki you can close to a good usability
| with LBAC (label based access control) but you still need have
| many data sources to map onto each "team view" to make it user
| friendly.
|
| What is missing for me is like in elastic a single datasource
| for all logs which every team member across all teams can see
| and you scope out the visibility level with LBAC
| feydaykyn wrote:
| Compared over Kibana, we experience: - 3x reduced costs - no
| more index corruption because a key changed type - slower
| performance for queries over 1 day, especially when non
| optimized without any filtering - non intuitive ui/ux
|
| So good but not perfect! When we have the time we'll look for
| alternatives
| kbouck wrote:
| Re: storage, Kibana (Elastic) has a new (as of v8.17)
| "logsdb" index mode which claims to be ~2.5x more storage
| efficient than previous options.
| valyala wrote:
| Did you try VictoriaLogs? It is easier to configure than Loki
| and Elasticsearch, and it uses less CPU and RAM. See
| https://itnext.io/how-do-open-source-solutions-for-logs-
| work...
| Unroasted6154 wrote:
| Loki was much cheaper to run in my experience, using S3
| storage. And you could scale the parts you needed dynamically
| in K8s.
|
| Elastic was kind of a resource hog and much more expensive for
| the same amount of data.
|
| That might be dependent on your use case though.
| suraci wrote:
| 1. It is extremely resource-efficient.
|
| 2. It has a convenient and simple query language.
|
| 3. It works very well with traces and metrics.
|
| the pain part:
|
| 1. It struggles to query logs over a wide time range.
|
| 2. Its indexing (or labeling) capabilities are very limited,
| similar to Prometheus.
|
| 3. Due to 1 and 2, it is difficult to configure and use
| correctly to avoid errors related to usage limits (e.g.,
| maximum series limits).
| arcanemachiner wrote:
| Which its is it that your its are referring to?
| suraci wrote:
| loki
| ohgr wrote:
| Kibana + ElasticSearch was a mess for us. Was glad to get rid
| of it. Cost a fortune to run and was time consuming. Loki
| conversely doesn't even show up on our costs report (other than
| the S3 bucket) and requires very little if any maintenance!
|
| Also out of box configuration sinks 1TB/hr quite happily in
| microservices mode.
| valyala wrote:
| How much CPU and RAM do all the Loki components use for your
| workload?
|
| Could you share Loki config, which can deal with 1TB/hr
| volume of logs?
| ohgr wrote:
| No idea. It's their out of box microservices set up. I
| think the only thing we pumped up was the chunks cache to
| 30Gb.
| jakozaur wrote:
| Kibana is great, but managing Elasticsearch is pain. Slow
| ingestion, query performance, takes a lot of space, harder to
| tune up.
|
| Modern columnar SQL such as ClickHouse are 10+ times more
| efficient in real-world use cases.
|
| I'm a CEO and founder of Quesma, which, let's use Kibana with
| ClickHouse: https://quesma.com/
|
| Forever free, source-available license.
| remram wrote:
| ELK could never deal with my logs which are sometimes-JSON.
| Loki can ingest and query it just fine. Also the
| query/extraction language makes a lot more sense to me.
| cortesoft wrote:
| Couldn't you use the logstash part of ELK to process the
| JSON?
| remram wrote:
| JSON no problem, sometimes-JSON no. It would drop the not-
| JSON message entirely.
| valyala wrote:
| Elasticsearch can store arbitrary text in log fields,
| including JSON-encoded string. Elasticsearch can also
| tokenize JSON-encoded string and provide fast full-text
| search over such string in the same way like it does for a
| regular plaintext string.
|
| why do you need storing JSON-encoded string inside log field?
| It is much better parsing the JSON into separate fields at
| log shipper and storing the parsed log fields into
| Elasticsearch. This gives better query performance and may
| also reduce disk space usage, since values for every parsed
| field are stored separately (this usually improves
| compression ratio and reduces disk read IO during queries if
| column-oriented storage is used for per-field data).
|
| I tried explaining this at https://itnext.io/why-
| victorialogs-is-a-better-alternative-t...
| kbouck wrote:
| If your source emits logs in OpenTelemetry format, using an
| OTel Collector inbetween you could do sometimes-JSON parsing
| of log content before the backend.
| parliament32 wrote:
| Yes, we switched metrics and logs from an Elastic stack to
| Prometheus/Thanos/Loki/Grafana about two years ago. On the logs
| side specifically, resource usage is WAY lower (300eps is like
| 1.5 cores and 4gb of memory), not to mention going from
| persistent volumes (disks) to blob storage / S3 is far cheaper
| and doesn't require any maintenance. Queries are slower,
| however, because Elastic pre-indexes while Loki searches on-
| demand, so it really comes down to query volume and your need
| for query performance (does it matter if your search takes
| 300ms vs 3s?). I've also found running Elastic yourself
| requires constant maintenance, while Loki has been very hands-
| off. Strongly recommend.
| jiveturkey wrote:
| (2024)
|
| important because the title includes _new_
| slekker wrote:
| A disclaimer is that OP is CEO of another company in the same
| sector
| NewJazz wrote:
| Chris Siebenmann? What company is he the CEO of?
| fsiefken wrote:
| No, the OP of the HN thread is from VictoriaMetrics (open
| source), he's not Chris Siebenmann, unix systems
| administrator at the University of Toronto's Computer Science
| Labs.
| duckydude20 wrote:
| i find him everywhere theres is mention of loki. trying so
| hard to push down loki. sure loki have issues but i don't
| feel this is a good approach to promote your product.
| valyala wrote:
| I'm trying to push Loki developers to fix these issues to
| make Loki users happy. What's the point to hide these
| issues? The won't disappear on their own over time.
|
| I'm very glad to know about VictoriaLogs issues, so they
| could be addressed quicker. If you hit such issues, then
| please file them at
| https://github.com/VictoriaMetrics/VictoriaMetrics/issues
| or publish an article highlighting these issues.
| NewJazz wrote:
| Ah, poster not article author gotcha.
| ohgr wrote:
| Not much I agree with in this article. Seems to be based on
| little operational experience with the product, particular
| indicated by a couple of major mistakes and assumptions in the
| article (compacting does happen, didn't read the manual about
| deployment configurations clearly).
|
| Loki has its idiosyncrasies but they are there for a good reason.
| Anyone who has sat there waiting hours for a Kibana or Splunk
| query to run to get some information out will know what I'd
| referring to. You don't dragnet your entire log stream unless
| your logs are terrible, which needs to be fixed, or you don't
| know when something happened, which needs fixing. I watch many
| people run queries that scan terabytes of data with gay abandon
| on a regular basis on older platforms and still never get what
| they need out.
|
| The structured metadata distinction is important because when you
| do a query against that you are not using an index, just parsed
| out data. That means explicitly you're not filtering, you're
| scanning and that is expensive.
|
| If you have a problem with finding things, then it's not the
| logging engine, it's the logs!
| xyz-x wrote:
| Does anyone have experience using Apache Iceberg, Hudi or
| DeltaLake for storing logs instead of Loki?
| pbh101 wrote:
| danluu.com mentioned this approach (or just, 'big data'
| systems) for traces and metrics iirc. Not sure if for logs too.
| Aren't all of those relatively tabular? What would you be
| looking for from those tools to help with logs?
| ptman wrote:
| @valyala , as others have noted, you are CEO of VictoriaMetrics
| and have written (most of?) VictoriaLogs. How is VictoriaLogs
| coming along? This is an older blog post.
| Dusseldorf wrote:
| I switched our team over to VictoriaLogs from ELK when VL1.0
| was released a few months back and we've been very happy with
| it. Nowhere near as much finicky performance tuning, no more
| logs failing to ingest because a string looked a bit too
| numeric, and the query language has fewer weird gotchas.
|
| At the end of the day ELK was throwing us a bunch of roadblocks
| in order to solve problems we didn't need solved. Maybe if we
| were trying to build some big analysis layer on top of our logs
| that would've been nice. VL has worked great for our use case
| of needing to centralize and view logs.
| valyala wrote:
| VictoriaLogs is free from issues mentioned in the referred
| article. It supports log fields with big number of unique
| values (such as user_id, trace_id, ip, etc.) from the
| beginning, and it doesn't need any configuration for working
| with such fields. It automatically indexes all the log fields
| and provides fast full-text search over all the ingested log
| fields.
|
| This is explained in more details at https://itnext.io/why-
| victorialogs-is-a-better-alternative-t...
___________________________________________________________________
(page generated 2025-03-19 23:01 UTC)