[HN Gopher] Show HN: LogsQL - opinionated query language for logs
___________________________________________________________________
Show HN: LogsQL - opinionated query language for logs
I don't like the existing query languages for Elasticsearch and
Grafana Loki, because they are too awkward to use for typical logs'
investigation cases. So I designed new query language - LogsQL -
and wrote reference implementation for it as a part of VictoriaLogs
- an open source database for logs. LogsQL is based on the
following principles: - Simplicity. It is easy to write typical
queries over logs in it. For example, a single word `error` is a
valid LogsQL query, which returns all the logs with the `error`
word. Another example is `_time:5m error`, which returns all the
logs with the 'error' word over the last 5 minutes. - Composable
building blocks similar to Unix pipes, which allow powerful
filtering, transforming and calculating stats over the selected
logs. For example, `_time:5m error | stats count() as rows` returns
the number of logs with the `error` word over the last 5 minutes.
- Readability. Typical LogsQL queries must be easy to read and
understand even for persons unfamiliar with it. Take a look at
LogsQL docs [1] and try using VictoriaLogs [2] in production. If
you like Unix way and KISS design principle, then you'll enjoy
LogsQL :) [1]
https://docs.victoriametrics.com/victorialogs/logsql/ [2]
https://docs.victoriametrics.com/victorialogs/
Author : valyala
Score : 47 points
Date : 2024-05-23 01:28 UTC (2 days ago)
(HTM) web link (docs.victoriametrics.com)
(TXT) w3m dump (docs.victoriametrics.com)
| bdcravens wrote:
| You have to feed your logs into the VictoriaLogs database in
| order to use LogsQL, right?
|
| "LogsQL is a simple yet powerful query language for
| VictoriaLogs."
| BillFranklin wrote:
| Interesting to see a new approach!
|
| You wrote that you don't like Loki's LogQL, but it looks quite
| similar (Victoria's LogQL first): log.level:error
| _stream:{app!~"buggy_app|foobar"} {app!~"buggy_app|foobar"}
| | "log.level:error"
|
| The pipes are arguably a bit noisy in Loki queries (compared to
| spaces in Victoria's), but I find they do make the queries a bit
| more readable, and it's easier to understand under the hood how
| the queriers will handle the query, and coming from PromQL I
| found Loki's approach quite intuitive:
| https://grafana.com/docs/loki/latest/query/.
|
| Maybe I missed something fundamental though, interested to hear
| more about the differences, since I only read the couple links
| you shared!
| BillFranklin wrote:
| As an addendum, I think the main flaws with Loki is log
| presentation in the Grafana UI, rather than necessarily
| querying syntax (extracting data from log lines with regex
| etc/pattern matching is gross, and if you're doing that you
| have a bigger problem).
|
| With the Grafana Loki UI the main issue is if you log perfect
| JSON / logfmt and Loki parses it, the Grafana UI can struggle a
| bit with rendering views of logs in the way you'd want.
| Kibana's columnar views with nice filtering UX is much easier
| to use. I think Elastic still has this over Loki (that, and the
| capacity to build big expensive indexes if you have the stomach
| to manage them).
|
| I wrote a custom CLI for Loki to work around this, because the
| Loki CLI has a similar problem. All that said, I'd still
| recommend Loki, it's really good IME.
| deepsun wrote:
| I believe pipes for logs were invented by SumoLogic 10+ years
| ago. Or maybe someone before that.
| dangoodmanUT wrote:
| IMO SQL is great for logs, ClickHouse and DuckDB SQL in
| particular are awesome
| lmeyerov wrote:
| I find SQL nice for a happy path - highly structured, non-
| nested, and simple log scenarios, such as some heavily curated
| app log subset like an app-level authentication audit log
|
| When we have arbitrary logs from all over, and especially
| system.. I appreciate the sloppy and dynamic nature of SPL and
| its derivatives
|
| We have ported our app tier to OTel metrics/logs/traces, and
| I've been curious what will be more 'right' in this new era.
| Likewise, with louie.ai, we have genAI do the text2query for us
| by default, which shifts things a bit as well.
| ople wrote:
| Recently there seems to be an bunch of SPL (Splunk) -like query
| languages popping up: PQL, PRQL, Grafana Explore Logs syntax,
| Kusto.. Probably others as well. Does yet another similar but
| slightly different language make sense? Why not leverage an
| existing one?
| rixrax wrote:
| Am I the only one that feels that EVERYTHING is wrong in this
| ELK, Splunk, etc. Grafana world? The user interfaces that these
| monstrosities present us with are barely useable, everyone has
| their own query language, they force us to install their own
| agents own our hosts and servers, when I upload logs, many can't
| even take random JSON logs and input them in a structured way
| without defining pipeline rules or what now. And did I say that
| the Logstashes and Promtails and Vectors and what not pipeline
| tools with their Grok etc. filters feel like somebody wanted to
| really make busywork cool.
|
| I am happy that in my day to day work I can dump my mostly Linux
| logs to rsyslog, and eventually forward them to S3 glacier for a
| few years.
|
| So I am guessing the question I am asking is that what in the
| world are you doing with these observability or SIEM platforms
| and is anyone actually deriving some REAL value from using them?
| fragmede wrote:
| Yes there's real value there. That everyone's got their own
| flavor is a bunch of extra work because we haven't solved the
| coordination problem yet is annoying, but that's solved by
| choosing one and sticking with it. Learn that query language
| really really well, and don't touch anything else. Splunk is
| useful as all hell once you get over the hump of learning their
| proprietary query language. it's really friggin useful. it's
| useful to the tune of Cisco buying them for $28 billion. people
| are deriving real value from them, the question is what are
| your problems that this can solve for you, but do you even have
| those problems in the first place? If you've not found it
| useful then why are you stuffing logs into S3? just send them
| to /dev/null instead
| rixrax wrote:
| >> just send them to /dev/null instead
|
| I wish. But 'regulatory compliance'. And 'we might need them'
| - just not sure for what - but we'll try another data analyst
| next quarter. Thankfully because of the GDPR (and maybe other
| reasons) there is a healthy pressure to also cleanse us from
| the logs we've collected.
|
| That said, I agree, based on my trials (and mostly errors),
| Splunk seems one of the better ones. Not considering the cost
| anyway. My trouble is that I am not a data analyst, but I get
| asked more than I would like about these things.
| hijinks wrote:
| Take a look at quickwit. Its basically a clone of elasticsearch
| but in rust.
|
| I have around 380 TBs of logs currently in s3 and have sub 1s
| searches for needle in the haystack searches. It handles all
| that with just 5 search nodes running on kubernetes with 6gig
| of RAM each.
|
| I'm ingesting around 20TBs of logs a day on it.
| Starlevel004 wrote:
| > And did I say that the Logstashes and Promtails and Vectors
| and what not pipeline tools with their Grok etc. filters feel
| like somebody wanted to really make busywork cool.
|
| The worst part about Promtail/Vector is that you have to write
| code in YAML. Why.
| evil-olive wrote:
| Vector at least supports TOML, not just YAML [0]
|
| that, plus having support for built-in support for "unit
| testing" processing pipelines [1] are two features that made
| me immediately want to ditch our existing Promtail configs
| and switch to Vector.
|
| 0: https://vector.dev/docs/reference/configuration/#formats
|
| 1: https://vector.dev/docs/reference/configuration/unit-
| tests/
| paulddraper wrote:
| None of those require agents?
|
| But for most people, an agent is simpler.
| piterrro wrote:
| I'm a https://logdy.dev (logs to UI interface) author and been
| recently thinking about how to enable users use a query language
| to search throught logs beyond usual filter. I was looking at
| LogsQL but then I felt that is just another QL a user will need
| to learn. My next though was on SQL, but it was not designed for
| this purpose. Any ideas? I would appreciate any recommendation
| (peter at logd.dev)
|
| [1] https://github.com/logdyhq/logdy-core
___________________________________________________________________
(page generated 2024-05-25 23:01 UTC)