[HN Gopher] Preview of Explore Logs, a new way to browse your lo...
       ___________________________________________________________________
        
       Preview of Explore Logs, a new way to browse your logs without
       writing LogQL
        
       Author : matryer
       Score  : 188 points
       Date   : 2024-04-09 14:23 UTC (8 hours ago)
        
 (HTM) web link (grafana.com)
 (TXT) w3m dump (grafana.com)
        
       | candiddevmike wrote:
       | Why have explore logs as a separate app instead of bundled with
       | Loki? It would be nice if Loki had the same kind of barebones
       | querying/debugging functionality as Prometheus...
        
         | skrtskrt wrote:
         | Loki is just the backend just like Prometheus is just the
         | backend
        
           | pluies wrote:
           | Yeah but Prometheus has a web ui where you can run PromQL
           | queries and it'll give you basic graphs back, which is handy
           | for throwing a quick query at it before putting it into
           | something more long-term like a Grafana dashboard or an
           | alerting rule.
        
             | skrtskrt wrote:
             | wow I always thought that Prom UI was just a tacked-on part
             | of alertmanager or something because it's so rudimentary.
             | In my experience, everyone just uses Grafana Explore since
             | that's what Grafana was originally purpose-built for and
             | it's crazy easy to set up. Just pull down a container or
             | helm chart or whatever.
             | 
             | Since Grafana built Loki, it doesn't make any sense why
             | Grafana would create a separate querying UI app for Loki
             | when they already have Grafana Explore. Prometheus (and I
             | assume its UI) was created by Google [edit: sorry, created
             | by SoundCloud, inspired by a Google Borg tool] before
             | Grafana became the de facto Prometheus query UI, so it's
             | not really analogous.
        
               | richardwhiuk wrote:
               | Prometheus was created by SoundCloud, not Google, but was
               | inspired by the Google Borgmon tool.
        
       | jdoss wrote:
       | I recently setup Victoria Metrics +
       | https://github.com/prometheus/snmp_exporter + Grafana to get
       | start tracking bandwidth on my top of rack switches in my
       | Datacenter rack which has been a pretty awesome setup. The way
       | you can auto generate a config for your SNMP MIBs with SNMP
       | Exporter was unexpectedly not a terrible experience.
       | 
       | My next task is to get centralized logging going with Victoria
       | Logs + Vector, I'll have to check this out once I get everything
       | setup. I believe I can use LogQL with Victoria Logs but I haven't
       | tried it out yet.
       | https://docs.victoriametrics.com/victorialogs/logsql/
        
         | NortySpock wrote:
         | I've been eyeing a VictoriaLogs setup for my docker container
         | fleet, but I haven't quite spotted where docker's remote
         | logging export options overlap with VictoriaLogs ingestion
         | options.
         | 
         | Wrinkle: two docker remote logging plugins I tried (e.g. loki,
         | elastic) didn't seem to work on ARM processors out of the box.
        
           | jdoss wrote:
           | Check out Vector for shipping logs from Docker. It might work
           | out for you https://vector.dev/docs/reference/configuration/s
           | ources/dock...
           | 
           | I use Podman for all of my container stuff and there are
           | issues with how Podman produces JSON logs
           | 
           | https://github.com/vectordotdev/vector/issues/6807
           | https://github.com/containers/podman/issues/16317
           | 
           | which needs to get fixed before I can use it for my
           | workloads.
        
         | nklmilojevic wrote:
         | This is what I've been doing on my cluster:
         | 
         | https://github.com/nklmilojevic/home/blob/main/kubernetes/ap...
         | 
         | https://github.com/nklmilojevic/home/tree/main/kubernetes/ap...
         | 
         | Here you have Vector in aggregator + agent mode and several
         | sources. VictoriaLogs also recently added Grafana datasource so
         | it is fairly easy to set it up:
         | 
         | https://github.com/nklmilojevic/home/blob/main/kubernetes/ap...
         | 
         | I'm a big fan of VictoriaMetrics as well and we use it
         | extensively in my company at high scale.
        
       | vbezhenar wrote:
       | I thought it was a standalone web app, but it's integrated into
       | Grafana. I'm confused a bit. There's already Explore
       | functionality in Grafana for Loki. Seems like spreading the
       | efforts for no reason.
        
       | umanwizard wrote:
       | I'm not really a cloud expert so maybe I'm fundamentally missing
       | something about how I'm "supposed to work", but honestly all I
       | have ever wanted to do, when looking at logs, is see the log from
       | one process, from beginning to end, as a text file. You can of
       | course do this using kubectl but only for the most recent two
       | instances of a given pod which isn't helpful when investigating
       | an incident that happened a while ago.
       | 
       | It seems nobody else cares about this use case and wants you to
       | use LogQL and the incredibly clunky Grafana web UI instead,
       | because it makes it possible to aggregate across many different
       | processes, slice and dice by various labels, etc., which as I
       | said, I have never (or almost never) actually wanted to do.
       | 
       | Hopefully this new UI is a step in the right direction as people
       | won't need to futz around with LogQL anymore, but it seems like
       | it still doesn't quite do what I want.
        
         | nine_k wrote:
         | Could LogQL do.something like                 select * from
         | stdout, stderr       where session_id = 123456
         | 
         | ? If not, why?
        
           | Matthias247 wrote:
           | yes it can, if you tag your log stream correctly - either by
           | having the stream externally tagged via attributes, or
           | internally by following certain conventions in the log line.
           | 
           | You can also do something like
           | 
           | select client_ip from requests where elapsed_ms > 10000
           | 
           | which is incredibly powerful
        
           | skrtskrt wrote:
           | yep, with the caveat that you probably don't want to have the
           | backend of whatever log system you use (not exactly sure how
           | Loki does it) to have an index on something as high-
           | cardinality as session id so that query could get slow.
           | 
           | But these log query systems can also optimize these queries
           | for instance by by sampling, using distributed trace ids to
           | ensure you get shown corresponding, allowing you to get only
           | logs where at least one step in the trace errored, etc.
        
           | westurner wrote:
           | strace and gdb can trace and close and reopen process file
           | handles 0,1,2.
           | 
           | ldpreloadhook has an example of hooking write() with
           | LD_PRELOAD=, which e.g. golang programs built without libc
           | don't support.
           | 
           | When systemd is /sbin/init, it owns all subprocess' file
           | handles already, so there's no need to close(0), time,
           | open(0) with gdb.
           | 
           | Without having to logship (copy buffers that are flushed
           | and/or have newline characters in the stream) to a network or
           | local Arrow database files and or SQLite vtables,
           | 
           | journalctl (journald) supports pattern matching with: -t
           | syslogidentifier, -u unit; and -g grepexpr of the MESSAGE=
           | field:                 journalctl -u <TAB>       journalctl
           | -u init.scope --reverse       journalctl -u unit.scope -g
           | "Reached target"  # and then "/sleep" to search and highlight
           | with less              journalctl -u auditd.service
           | # this is slow because it's a full table scan, because
           | # journald does not index the logfiles;       # and -g/--grep
           | is case insensitive if the query is all lowercase:
           | journalctl -g avc --reverse       journalctl -g AVC --reverse
           | # this is faster:       journalctl -t audit -g AVC -r
           | # this is still faster,       # because it only searches the
           | current boot:       journalctl -b 0 -t audit -g AVC
           | # these are equivalent:       journalctl -b 0 --dmesg -t
           | kernel       journalctl -k            #        journalctl -b
           | 0 --user | grep -i -C "xyz123"
           | 
           | There is a GNOME Logs viewer that has 'All' and a few
           | mutually exclusive filter/reports in a side pane, and a
           | search expression field to narrow a filter/report like All or
           | Important.
           | 
           | There is a Grafana Loki Docker Driver that logships from all
           | containers visible on that DOCKER_HOST docker socket to
           | Grafana for querying with Loki:
           | https://grafana.com/docs/loki/latest/send-data/docker-
           | driver...
           | 
           | Podman with Systemd doesn't need the Grafana Docker Driver
           | (or other logshippers like logstash, loggly, or fluentd)
           | because systemd spawns containers and optionally pipes their
           | stdout/stderr logs to journald.
           | 
           | Influx has Telegraf, InfluxDB, Chronograf, and Kapacitor.
           | Chronograf is their WebUI which provides a query interface
           | for configurable chart dashboards and InfluxQL.
           | 
           | Grafana supports SQL, PromQL, InfluxQL, and LogQL.
           | 
           | Graylog2 also indexes logfiles.
           | 
           | But you can't query stdout and stderr you or /sbin/init
           | haven't logged to a file.
        
         | Topgamer7 wrote:
         | I use LogQL a fair amount. Often times even just negative
         | filtering is quite useful.
         | 
         | I do a fair amount of tracking down of issues with LogQL.
         | Looking for logs specific to a customer support ticket.
         | Filtering for logs by a traceId for distributed traces.
         | 
         | I have serious doubts this new UI is something I will care
         | about at all.
         | 
         | The explore ui for setting labels is atrocious and painful, and
         | I'd rather just give me the text input for LogQL*
         | 
         | *: Please FFS someone fix the Ctrl+f creating a vscode like
         | find dialog that only finds inside the text input. I never want
         | to do a find specifically isolated to my LogQL
        
         | fellerts wrote:
         | Just want to chip in and say that I wholeheartedly agree with
         | you. I'm not a cloud developer either, but I'm regularly forced
         | into what's apparently called "Google Cloud's operations suite"
         | to grovel through logs. Compared to working with Linux journals
         | using the tried and true text manipulation tools, it feels like
         | looking through a straw with oven mitts on. I'd happily
         | download a 500 MB text file instead, but there is an arbitrary
         | limit to how much I can grab (10k lines IIRC). Maybe we're just
         | out of touch.
        
           | debarshri wrote:
           | It fairly risky to download 500MB of log and analyse it
           | locally in the machine. I know People do it anyways. Just
           | saying.
        
             | dstroot wrote:
             | Risky how exactly? If it has data in it that it shouldn't
             | it's a problem no matter where it resides.
        
               | ta1243 wrote:
               | In theory a logfile could contain privileged information
               | (indeed it almost certainly will - IP addresses etc),
               | putting that on a laptop increases risk of losing it.
        
               | debarshri wrote:
               | It is not about what data is in the log. It is about the
               | fact that once the data is download, it most likely going
               | to stay in the machine.
               | 
               | Logs often contain privileged info, if not reveal a bit
               | about how the application behaves. It is risky to do
               | that.
        
           | remus wrote:
           | > but I'm regularly forced into what's apparently called
           | "Google Cloud's operations suite" to grovel through logs
           | 
           | Is this google cloud logging? If so, personally I quite like
           | it, especially for looking through logs from multiple sources
           | at the same time. Being able to put all your logs through
           | there, and then search them with a simple query language,
           | feels very convenient.
        
             | pas wrote:
             | it's ridiculously slow. and compared to how expensive it is
             | ... it's robbery in daylight.
        
         | dzikimarian wrote:
         | Dump the logs into elastic, loki or whatever, along with pod
         | name as a label. Usually I use Kibana, so I don't want to speak
         | for Loki, but seems pretty straight forward.
        
           | hyperpape wrote:
           | You missed the key criterion, which is being able to see the
           | logs from that process "as a text file", or the way I'd
           | rephrase it "with the same ease of a text file."
           | 
           | Kibana is ok (definitely beats grep) when you want to look
           | across a fleet and determine if a specific thing is
           | happening. But when you have a specific symptom that happens
           | on a particular instance, what you want to do is see logs in
           | the order they happened, and Kibana isn't close. Querying and
           | viewing logs are just slow and cumbersome relative to
           | less/grep.
        
             | dzikimarian wrote:
             | Well, honestly I don't understand what's missing - you just
             | pick time window, instance and have logs displayed line-by-
             | line as they happened.
             | 
             | Best to configure view for this to limit columns and maybe
             | pre-configure some filters. Plus annotate your logs with
             | timestamp, so you rely on time of event and not time of
             | ingestion.
             | 
             | But these both are one-time configuration thing and then
             | you can simply scroll.
        
               | umanwizard wrote:
               | > Well, honestly I don't understand what's missing - you
               | just pick time window, instance and have logs displayed
               | line-by-line as they happened.
               | 
               | What's missing is that I don't want to learn and use some
               | clunky web UI in order to do this. I want the UI to be
               | "download this text file" and then use the tools I
               | already know and understand (local text processing
               | utilities and text editors)
        
               | adra wrote:
               | This seems like a solution for pets. If you have a lot of
               | pets, this sounds totally reasonable, but it isn't some
               | universal truism. People are moving away from pets as
               | they're often harder to work with cattle. That also means
               | you need an observability aggregation which can make
               | sense of what's happening everywhere, not just one
               | instance of on machine.
        
         | tkone wrote:
         | If you're debugging something simple or non-distributed, this
         | product isn't for you.
         | 
         | If you're working on anything distributed, log aggregation
         | becomes a must. But, also, if you're working on anything
         | distributed and you're looking at logs, you're desperate.
         | Distributed traces are so much higher quality.
        
           | umanwizard wrote:
           | When I formed these opinions I was working on Materialize,
           | which is basically the polar opposite of "simple and non-
           | distributed". However it was still quite common that I knew
           | exactly which process was doing something weird and
           | unexpected.
        
             | jhrmnn wrote:
             | Maybe it's the difference between tracking a bug (abnormal
             | operation) vs understanding behavior of a complex system
             | (normal operation)?
        
           | mason55 wrote:
           | Yup and the reason no one markets something like "tail the
           | logs for server X" is because, if you're talking in the
           | context of an individual server, you're too small for anyone
           | to care about.
        
             | ta1243 wrote:
             | I've got logs from hundreds of servers that I use standard
             | tools to look at, and that's a small system. Centralising
             | logs has been a thing for decades.
        
               | mason55 wrote:
               | Which is fine, I'm just saying you're not the target
               | market for the big observability vendors.
               | 
               | The current generation of observability tools is built
               | for distributed systems that are basically too complex to
               | reason about, and so you have other ways of monitoring
               | and debugging them. When you have 10's of k's of
               | ephemeral containers running hundreds of services, you
               | can't just look at some logs for a server to understand
               | what's going on (ignoring the fact that servers aren't
               | even a primitive in this system).
               | 
               | 10's of GBs of logs a day just doesn't move the needle on
               | pricing. They want the customers that are going to
               | generate 7 figures in revenue and those customers aren't
               | talking about aggregating logs from a few hundred
               | servers.
        
           | zo1 wrote:
           | Sorry, did plenty of "distributed" tracing back in the day
           | and this is just not the case. I can't help but feel like
           | you're after-the-fact rationalizing as if you need this for
           | diagnosing anything "distributed" or "complicated".
           | 
           | Distributed anything is actually easier in most cases because
           | you will always have input and output. Sure, if you're
           | debugging a complicated and coordinated "dance" between two
           | concurrent threads/processes then yeah fully agreed, but then
           | you're deep in uncharted territory and you need all the help
           | you can get.
        
         | andres wrote:
         | If you get a chance, please check out kubetail
         | (https://github.com/kubetail-org/kubetail). It's an open source
         | log viewer for Kubernetes. Currently you can use it to look at
         | pod logs from beginning to end, grouped together by workload
         | (e.g. Deployment, CronJob) with basic filtering available (e.g.
         | node-id, AZ). It doesn't let you look at historical logs yet
         | but that's where we're headed. We just launched so we're eager
         | for feedback and we like to build out new features quickly.
        
           | bennine wrote:
           | Interesting. Will be following this tool.
           | 
           | There is a CLI tool with the same name that does something
           | similar - https://github.com/johanhaleby/kubetail
        
         | pluies wrote:
         | Fwiw this is how I use Loki most of the time. Pick an app
         | label, pick a time period, look at raw logs. The LogQL for this
         | ends up something like `{app="workload-foo"}`. Loki is
         | excellent at that.
         | 
         | Then if I know which pod I'll filter down to a specific pod
         | with `{pod="workload-foo-1234"}`, sometimes I'll search for a
         | specific term (error message etc) with `{pod="workload-
         | foo-1234"} |= "error message"` then look at the logs around
         | that. There's really no point writing complicated queries
         | unless you need to.
        
           | umanwizard wrote:
           | That will, if I understand correctly, get the logs for one
           | pod, not for one process. For example if the pod restarted 10
           | times you will not get 10 separate files from that query.
        
             | sofixa wrote:
             | You'd have the label shown in the output that indicates the
             | log line in question is from a different
             | process/pod/container/host/whatever.
        
               | umanwizard wrote:
               | How so? The pod, container, and host labels should be the
               | same for a process that crashes and is automatically
               | restarted, no?
        
             | liampulles wrote:
             | Even more than that, if you are running multiple instances
             | of the app in multiple pods concurrently, then all of those
             | logs will be joined together.
        
               | remram wrote:
               | That's definitely false
        
               | Matthias247 wrote:
               | Why? If the pod is defined to spawn multiple containers,
               | and each container runs the same application, then this
               | seems true to me? Unless you would add an additional
               | filter on the container name.
        
               | remram wrote:
               | Well yes obviously you have to filter on container if you
               | want a single container (just like kubectl logs -l
               | <...>). The parent comment was phrased as a limitation of
               | Loki, of course if you request all logs for an
               | application you'll get all containers, or if you request
               | all logs for an applications or a namespace you will get
               | that.
               | 
               | Not being able to filter between multiple processes or
               | multiple restart of a container was a genuine issue, not
               | being able to filter between pods of a deployment is not.
        
               | Matthias247 wrote:
               | I actually didn't understand it being phrased as a
               | limitation. It could also be a feature - maybe one would
               | prefer to look at logs for multiple services within a
               | single query?
               | 
               | Anyhow, the nice thing about the system is that one can
               | get anything that is preferable as long as the logs are
               | annotated correctly (with pod and container id).
        
               | remram wrote:
               | No, once again, the trouble is that you can't get the
               | logs for a specific execution. If a container in your pod
               | restarts, that is invisible to Loki, you have to look for
               | whatever the container writes on startup and cut there
               | manually. If you want a specific process in your
               | container, it's mixed with the rest.
        
               | liampulles wrote:
               | Its certainly true in my environment, maybe not others
               | though? Apologies!
        
         | ta1243 wrote:
         | I'm an old fart so I use things like "cat" and "grep", and
         | maybe "sed" and "cut" if the lines are particularly long.
         | 
         | I have one log file per day per host on my syslog server and
         | can use "sort" to order across multiple files.
         | 
         | Loki was sold to me at fosdem a couple of years ago as this,
         | but I still haven't got round to working it out, seems a very
         | high barrier to entry compared with running cat.
        
           | 392 wrote:
           | You may be amazed at how hard these tools are to get started
           | with relative to that. I have been thoroughly unimpressed
           | with and unable to really get started with any of these tools
           | because of the overemphasis on cloud. Not sure what people
           | were doing before, but sshing to the prod box kinda sucks.
        
           | abhijat wrote:
           | For analysing text logs lnav is pretty good, if you need to
           | work with a live updated view of the log in response to
           | commands.
        
           | sofixa wrote:
           | > seems a very high barrier to entry compared with running
           | cat
           | 
           | It really isn't. It's a single binary with a relatively
           | simple configuration file, you throw logs at it via an API
           | (which a bunch of logging agents support, and syslogs can be
           | sent to it).
           | 
           | Then the actual queries aren't all that complex, it's just a
           | difference of cd-ing to the correct folder for the
           | date/server to be able to cat and grep vs writing a query
           | that selects by server name and filters by date.
           | 
           | The learning curve and maintenance of Loki are quite minimal,
           | but the value add is quite significant _in most cases_. Being
           | able to do cross-host queries, metrics from logs (how many
           | times did error X occur in the logs), as well as easy
           | visualisations is pretty useful.
        
             | ta1243 wrote:
             | "to be able to cat and grep"
             | 
             | Admitably I learned how to use basic tools 25 years ago,
             | but that's an investment that can be used for decades.
             | cat *web*log | "grep 34.5.22.4" | sort -n | less
             | 
             | is hardly a complex thing to learn. Sure you can then build
             | on that pipeline -- "cut -b -10|uniq -c" and if you want
             | something really complex then you can use awk, or perl, or
             | python, and do all sorts of things with the data.
             | 
             | Will whatever today's favoured log query/filter/etc be
             | around in 25 years? Last time I looked at this people were
             | going on about logstash and elasticsearch. Nobody could
             | show me how to to the above command without touching the
             | mouse.
             | 
             | Now sure, cat and grep can be sluggish on millions of lines
             | (which is the main reason I'm tempted by loki or similar),
             | and there's always some twat that comes along with "useless
             | use of cat" [0], but the kind of pipeline processing serves
             | me well and it seems a very different way to think about
             | things when you need to access things from a database.
             | Maybe I'm in a local maximum, but it's good-enough for me
             | to find out what's going on.
             | 
             | [0] https://stackoverflow.com/questions/11710552/useless-
             | use-of-...
        
               | sofixa wrote:
               | "it's just a difference of cd-ing to the correct folder
               | for the date/server" to be able to cat/grep.
               | 
               | You have to connect to your server, get to the correct
               | folder, and then run the cats and greps which are easy
               | (if you have to do some more advanced filtering with awk
               | it gets more complicated.)
               | 
               | Connecting to Grafana and running a simple label query is
               | practically the same in terms of complexity and time, but
               | with vastly more features available.
               | 
               | > Will whatever today's favoured log query/filter/etc be
               | around in 25 years? Last time I looked at this people
               | were going on about logstash and elasticsearch. Nobody
               | could show me how to to the above command without
               | touching the mouse.
               | 
               | You can run ElasticSearch queries via the API, and can
               | still do it today. I don't know about, but Loki is a
               | statically compiled binary with only optional external
               | dependencies. You'd still be able to run it in 25 year
               | just fine.
        
         | jldugger wrote:
         | > maybe I'm fundamentally missing something about how I'm
         | "supposed to work", but honestly all I have ever wanted to do,
         | when looking at logs, is see the log from one process, from
         | beginning to end, as a text file.
         | 
         | This is still a valid use case but pretend for a minute you
         | have thousands or millions of log lines to inspect. Even after
         | filtering for ERROR level only, you still have too many "those
         | are normal" errors, devs swear (but do not fix). And maybe the
         | data you need to diagnose isn't even in ERROR!
         | 
         | The solution? Use log queries to compare a normal and abnormal
         | process or cluster, group them by some kind of fingerprint,
         | then apply some Laplace smoothing or other bayesian techniques
         | to score fingerprints by strength of association with abnormal.
         | This lets me rapidly identify problems at scale that would
         | otherwise take hours of pouring through logs to exclude stuff
         | by hand.
         | 
         | This works any time you can divide logs into "good" and "bad."
         | Example scenarios:
         | 
         | - canary analysis, comparing canary and baseline
         | 
         | - single faulty pod in a deploy, comparing the bad container to
         | the n good ones
         | 
         | - one AZ or region in a multi-region deploy
         | 
         | - now versus yesterday, or versus an hour ago, etc
         | 
         | - Android versus iPhone
        
         | Jenk wrote:
         | > I'm not really a cloud expert so maybe I'm fundamentally
         | missing something about how I'm "supposed to work", but
         | honestly all I have ever wanted to do, when looking at logs, is
         | see the log from one process, from beginning to end, as a text
         | file.
         | 
         | That's the rub that I think you are missing. In distributed
         | and/or cloud environments it is quite unusual for there to be a
         | single end-to-end process, and thus we need new ways to trace
         | across a system.
         | 
         | In harmony with tracing, we also need the aggregated view
         | _across_ the estate to understand where system hotspots, levels
         | of throughput, redundant infrastructure, error rates, etc.
        
       | nicoritschel wrote:
       | While this is a step in the right direction, just let me write
       | something closer to SQL. Influx did this correctly.
        
         | willseth wrote:
         | That's what LogQL is, which is already in Loki. This is a new
         | feature.
        
           | hamandcheese wrote:
           | LogQL is nothing like SQL. Try aggregating and quickly you'll
           | be asking yourself wtf an instant query is and how is that
           | different.
        
         | remram wrote:
         | Influx used SQL, then deprecated it and made everyone use Flux,
         | then deprecated that and moved back to SQL. They are definitely
         | not "doing it correctly" when it comes to query language.
        
       | wsatb wrote:
       | I'll preface this with the fact I haven't look at Loki in a bit,
       | so maybe this has changed. But I found the documentation needing
       | a lot of work and the configuration for promtail to be obtuse and
       | not very user friendly. I haven't used it for those reasons, not
       | because of the query language.
        
         | John23832 wrote:
         | Having used Loki/promagent etc, it was sort of a
         | pain/nonintuitive to set up.
        
         | corytheboyd wrote:
         | They gotta sell their managed cloud service somehow, I have
         | always assumed that this is part of the sales strategy
        
         | liampulles wrote:
         | Our team uses loki and I have to say I think their collected
         | helm charts are pretty easy to use - my problem is more that it
         | seems to be quite slow to run on-prem. Very often my loki query
         | times out and I have to do more work filtering down the log
         | lines or selecting a narrower time range.
         | 
         | I'm kind of amazed the UI doesn't select small time ranges
         | iteratively to build up the response, especially since I
         | believe this is what the CLI does. Perhaps this is also part of
         | their cloud offering provides and it is part of their marketing
         | strategy. Not a good one because if we came down to the
         | decision I would start by looking for something else from being
         | p'ed off by Loki.
         | 
         | But I guess it still works pretty well considering it is free.
        
           | skrtskrt wrote:
           | Loki UI in Grafana Explore seems to only select 1000 lines by
           | default for me?
           | 
           | Also Loki on the backend splits/parallelizes requests if it
           | can.
           | 
           | The Grafana backend Mimir / Loki / Tempo products all appear
           | to be architected pretty similar, and I'm more experienced
           | operating Mimir, but the answer to read load often has to
           | just do with right-sizing the deployment scale, and using
           | caches aggressively.
        
         | blue_cookeh wrote:
         | It's also incredibly easy to shoot yourself in the foot and
         | rack up huge cloud bills - something we recently hit:
         | https://github.com/grafana/loki/issues/8756
        
       | damm wrote:
       | This is hard coded to searching for service_name in the query
       | which doesn't return any data for me.
       | 
       | I will stop wasting my time here and try the Metric Explore panel
        
       | dangoodmanUT wrote:
       | LogQL is honestly fine, it's not SQL, but it's fine
        
       | pachico wrote:
       | I am a happy customer of Grafana Cloud, yet, I can't use their
       | backends for logs and metrics as they are terribly expensive and
       | slow.
       | 
       | Somehow, VictoriaMetrics manages to provide much better results.
        
       | hamandcheese wrote:
       | LogQL so far just does not click for me. I get that it's trying
       | to be like Prometheus, but logs are not the same as time series -
       | we have each and every log! So why am I forced to query it like a
       | time series data source?
       | 
       | I want to query my logs like a SQL table, not a time series
       | database.
        
       | saintfiends wrote:
       | Loki OSS is just a sales pitch for their managed service. It
       | doesn't work well without dedicating significant time tweaking
       | and configuring it. Documentation is confusing at best if you
       | want to do anything serious. You have to also be ready to handle
       | support calls if you open it up for others to use, because it
       | WILL have issues fairly regularly if you have a good volume of
       | logs and query range is more than a day or two.
       | 
       | Unless you have the bank to go with their managed service, don't
       | bother.
        
       | Spiwux wrote:
       | After having used Datadog for several years, going back to
       | Grafana / Loki / Prometheus felt like regressing by two decades.
       | As much as I appreciate free solutions, I feel like Grafana has
       | really fallen behind when it comes to developer experience
        
         | nthngtshr wrote:
         | Could you provide more details? Although I've never had the
         | opportunity to use Datadog at any of my previous positions, I
         | am quite familiar with Grafana and I'm generally pretty happy
         | with it.
         | 
         | What's the TL;DR for why Datadog is better?
        
         | zikohh wrote:
         | Grafana cloud is better for querying logs. Grafana cloud is
         | probably a bit better for querying metrics. Grafana cloud is
         | terrible at finding traces or even loading them. Datadog is
         | lightyears ahead. For alerting I feel datadog has better
         | features but is overwhelming with all the different options.
         | 
         | grafana is very quirky for searching for traces. And has a huge
         | learning curve.
        
       ___________________________________________________________________
       (page generated 2024-04-09 23:02 UTC)