[HN Gopher] Lnav - An advanced log file viewer for the small-scale
___________________________________________________________________
Lnav - An advanced log file viewer for the small-scale
Author : thunderbong
Score : 155 points
Date : 2023-01-04 09:01 UTC (1 days ago)
(HTM) web link (lnav.org)
(TXT) w3m dump (lnav.org)
| LinuxBender wrote:
| The package is available in Alpine so I tried it out. It seems to
| work best if the terminal is using en_US.UTF-8. My default is C
| but I have a function to set UTF-8. I was curious if it could
| handle stdin and it does. It saves the output in ~/.config/lnav/
| something to probably add to my clean up script. It is convenient
| that it can prepend timestamps to logs. I could see this being
| nice for people that do QA on build logs all day or that have to
| debug access logs often. If used with sensitive data one should
| probably add a shell function and log rotate scripts to clean up
| ~/.config/lnav/ using the shred command.
| pinkorchid wrote:
| This will depend on the specific threat model for the sensitive
| data you are dealing with, but generally files in an SSD can't
| be destroyed securely (and shred specifically does not help due
| to wear leveling).
|
| Instead, you may want to encrypt your disk so that as soon as
| the key is gone it all becomes unreadable. For a bigger threat,
| maybe you need to follow the NIST destroy guidelines [1] to
| "Disintegrate, Pulverize, Melt, and Incinerate" the media.
|
| 1:
| https://nvlpubs.nist.gov/nistpubs/specialpublications/nist.s...
| c0wb0yc0d3r wrote:
| I was under the impression that the shred command was pretty
| hard on SSDs. Did that change?
| LinuxBender wrote:
| It would be as hard as writing the file {n} number of times
| again. So if one created a 1MB log file and then did a 3 pass
| wipe that would write 3MB. Number of passes would really just
| depend on the system being worked on. If PCI I would expect 7
| passes and I am sure that would just be factored into the
| cost of the servers for that environment.
|
| Shred also isn't perfect as it has no concept of the file
| systems journal and does not clean that data but I think it
| is still good practice for sensitive data, in addition to
| filesystem or file level encryption on systems with highly
| sensitive data.
|
| With sensitive data one must look at the value of the data vs
| the value of the SSD. If I go out of my way to extend the
| life of my SSD, do I risk losing $5billion? That is how I
| factor in whether or not extra wear-and-tear on a SSD makes
| sense.
| electroly wrote:
| Shred doesn't work _at all_ on SSDs because wear leveling
| will spray the writes all over the drive, rather than
| overwriting the blocks you intended to. Your random bytes
| will end up in new blocks, not overwriting the original
| blocks. Trying to shred individual files is a totally
| pointless exercise on SSDs. It isn 't doing what you want
| it to do.
| LinuxBender wrote:
| I've also seen people say that and to instead use
| "blkdiscard -sz" but that also does not write to specific
| blocks. I still can not find a definitive source saying
| that shred is not able to write to specific blocks on a
| SSD and it is not clear to me that wear leveling actually
| prevents shred from writing to specific blocks. But I
| agree that I have seen people say this repeatedly on
| StackExchange, ServerFault and Reddit.
|
| Either way one should also use filesystem and file level
| encryption for sensitive files and encrypt sensitive
| attributes inside databases. Swap must also be encrypted
| as it may contain sensitive data. Datacenter bad-drive
| mishaps do happen _as in not shredding the physical
| drive_ as a few government and military agencies have
| recently been embarrassed by.
|
| I do agree with pinkorchid that ultimately drives should
| be destroyed physically.
| j0057 wrote:
| It's also explained in the shred manual itself, that
| seems like a decently authoritative source to me: https:/
| /www.gnu.org/software/coreutils/manual/html_node/shre...
| -- halfway the page all the caveats are listed up to &
| including forensic analysis of magnetic traces of the
| deleted data.
| electroly wrote:
| Wear leveling prevents _anyone_ from writing to specific
| blocks. It decouples the OS 's block numbering from the
| physical blocks. That indirection allows it to distribute
| writes evenly across the drive even if the OS is writing
| to consecutive blocks (or the same block).
|
| This behavior is fundamental; it's what wear leveling
| _is_ and why it exists. Wear leveling exists to prevent
| you from writing to the same block repeatedly, which is
| exactly what you 're trying to do with shred.
|
| I'm not sure what other source you're looking for, but I
| hope you find it. Perhaps, at least, the number of people
| "on StackExchange, ServerFault and Reddit" and now also
| HackerNews telling you the same thing is sufficient
| evidence to stop spreading the idea that shred might work
| on SSDs. Do what you want on your own systems, of course,
| but this is dangerously misleading guidance to be
| offering on an open forum.
| LinuxBender wrote:
| So my concern with following this logic is that such
| tools that are writing to specific block locations would
| then be wiping unrelated inodes depending on the low
| level logic. I tried emailing Colin but his last known
| email is invalid now. He may be working for the NSA so I
| probably won't be able to reach him. Some people there
| are responsive, some are not and a few are on here.
|
| So even if wear leveling prevents overwriting a file then
| such tools should in theory be a risk of data corruption.
| If this is the case then the tool should be updated to
| detect if the target is an SSD and abort with a scary
| message. Perhaps another route is to reach out to the
| coreutils team.
| sgc wrote:
| Does that mean that partitions on an SSD therefore do not
| affect the long term wear on the SSD?
| tstack wrote:
| The data consumed from stdin is stored in a file to support
| scrollback.
|
| > something to probably add to my clean up script.
|
| These capture files are deleted if they are older than a day
| the next time lnav is run.
|
| (I realize they should've been stored in ~/.local/state instead
| of ~/.config ...)
| hericium wrote:
| Do we need lnav thread every 3 weeks?
| mhitza wrote:
| Yes /s
|
| Not everyone checks out daily HN to see all the reposts.
| Pulling numbers out of the air, I'd say that 5-10% of daily
| content on HN is a repost, should get used to it. Just hide it
| an carry on is what I would suggest.
| qbasic_forever wrote:
| Keep a log of how often it occurs and we can review it... in
| lnav
| xdfgh1112 wrote:
| First I'd heard of it. I have wanted a tool like this for a
| while.
| asah wrote:
| Same, and I'm working on an IoT project that precisely needs
| this (superhigh$ per device, don't need scale or efficiency)
| joshka wrote:
| lnav is pretty great. I wish it had a library that could be
| pulled out into an app instead of a terminal. There are a few
| things that make more sense with a GUI like standard keyboard
| shortcuts, filtering, saving exports, etc.
| hericium wrote:
| Not saying there's something wrong with lnav - just that
| spamming HN every 3 weeks isn't great.
|
| There were submissions with links to lnav.org main site:
|
| - 21 days ago
|
| - 22 days ago
|
| - 46 days ago
|
| - 68 days ago
|
| It's a spam campaign from my pov.
| __float wrote:
| Do you browse new? None of these have had more than a
| couple points since
| https://news.ycombinator.com/item?id=9294622 in 2015.
|
| It seems very reasonable that most visitors simply never
| saw any of those submissions.
| chalst wrote:
| Five submissions in 70 days is rather less than a spam
| campaign in my view.
| Thaxll wrote:
| I mean if the goal is to read logs the default color scheme looks
| pretty bad, why there are so many colors in the example?
|
| For example the http 403/404 is red but it does not stand out.
| Also why a 404 would be red in the first place.
| tstack wrote:
| > why there are so many colors in the example?
|
| Identifiers, like process-names/pids/IPs, are semantically
| highlighted by default to make it easier to visually match up
| values that are the same. And, it's this way because I like it
| this way, so that's the default.
|
| > For example the http 403/404 is red but it does not stand
| out.
|
| To me, the red stands out. But, I would also have a much wider
| window, so the red would stand out even more with the rest of
| the text.
|
| > Also why a 404 would be red in the first place.
|
| Because 4XX are error codes.
|
| Of course, lnav is pretty customizable at this point, so you
| can adjust it to your liking. The theme can be changed to
| something less colorful and the log format definitions can be
| patched to change their behavior.
| mihaigalos wrote:
| Nice work! I've written something similar based on regexes:
|
| https://github.com/mihaigalos/pipeview
| _bax wrote:
| I'm using every day on multiple erver and application. It a MUST
| haspre wrote:
| Works fine on my pi.. thx
|
| With caddy server
| calmoo wrote:
| I used this extensively when working in a cloud support role,
| incredible tool.
| lathiat wrote:
| Does anyone have a trick for connecting to remote hosts as a non-
| root user but having it elevate to root to read the logs?
| prmoustache wrote:
| For systemd/journald managed log, best is to add non privileged
| user to the systemd-journal group.
|
| For logs managed by syslog daemons, most implementations allows
| you to set the owner and group of the log files. You could
| decide on a specific log group and add the desired dedicated
| user to this group.
|
| In the end it is usually better to ship the logs to a dedicated
| machine/space/database.
| yellow_lead wrote:
| It's cool you can SSH to demo it. More CLI tools should have
| that.
| tstack wrote:
| The SSH demo was inspired by git.charm.sh
| zulu-inuoe wrote:
| Say I'm building a logging library, is there any sort of
| generally agreed-upon standard per what a log file should look
| like? I know there's several formats that lnav supports, but I'm
| not familiar with them.
|
| I've always had trouble before figuring out what a good
| compromise between log format and flexibility looks like.
| Especially wrt newlines in the log message itself (eg if I want
| to log a stack trace)
|
| Any thoughts for a format that would generally work ootb in lnav
| or other log viewers?
| 10987654321 wrote:
| Not sure about lnav but most log aggregation systems support
| json and logfmt-formatted logs, and there are many standard
| logging libraries that supports emitting those formats
|
| Of those json is better if you want to be able to do more
| advanced stuff (nest dictionaries, use lists, ...) and logfmt
| is better if you want to have human-readable logs without
| external tools as well, an example line can look like
| msg="Request finished" tag=request_finish status=200
| user=brandur@mutelight.org user_id=1234 app=mutelight
| app_id=1234
|
| Some more info here https://www.brandur.org/logfmt
| GordonS wrote:
| You can use logfmt with Serilog on dotnet too:
|
| https://github.com/serilog-contrib/Serilog.Logfmt
| ithkuil wrote:
| Has anybody written an actual spec for logfmt? I noticed that
| different implementations handle escaping of quoted strings
| subtly differently
| jteppinette wrote:
| Agree with logfmt. I wrote the logfmter python library:
| https://github.com/jteppinette/python-logfmter. You can
| quickly have all of your logs (including 3rd party) converted
| to this style.
| tstack wrote:
| lnav does support JSON-lines and logfmt logs. For JSON-lines,
| it will pretty-print the log messages to make them human
| readable.
|
| For logfmt, I seem to remember the spec not being very clear
| on quoting semantics (maybe I'm wrong). Anyhow, I would
| suggest using JSON since it has pretty broad support at this
| point.
| zulu-inuoe wrote:
| Thank you! Now when you say json logging, are there any
| common patterns you can guide me to look at? Just looking
| around for JSON logging gives me results like [1] which talk
| about JSONL (object per line, presumably JSON serializer
| makes sure to not emit literal newlines).
|
| But some other places describe this a bit more liberally. And
| [2] notes that I should include a time stamp in each log
| entry which makes sense.
|
| [1]: https://stackoverflow.com/questions/10699953/format-for-
| writ... [2]:
| https://www.papertrail.com/solution/tips/8-essential-tips-
| fo...
| capableweb wrote:
| There is a bunch of them (log formats). I don't think any is
| like ISO/ANSI standarized, but many tools work with many of the
| different formats.
|
| Probably the most common one (or at least was most common,
| maybe not anymore) would be "NCSA Common log format" (or just
| CLF) which looks something like this:
| 127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700]
| "GET /apache_pb.gif HTTP/1.0" 200 2326
|
| https://en.wikipedia.org/wiki/Common_Log_Format
|
| Many tools for getting analytics out of server-side logs can
| work with CLF and the various variations. But probably today
| there are more "modern" formats as well.
| shae wrote:
| This tool saved me weeks of work once. You gotta try it.
| hoyd wrote:
| Related to this, is there a tool like this, but that can show
| some kind of simple analytics, for example page views over a
| period of time from an access log produced by python html server.
| guessmyname wrote:
| > _[...] a tool like this, but that can show some kind of
| simple analytics_
|
| Yes, you can use - https://goaccess.io/
| kiliancs wrote:
| You can run queries on the log files and write them to files. I
| have used these files to feed pre-built spreadsheets with
| charts, etc.
| tstack wrote:
| As mentioned in a sibling comment, https://goaccess.io is
| probably best for simple analytics on log files.
|
| If you want to write more complicated queries, lnav exposes log
| data through SQLite vtables[1]. So, you can do a SQL query and
| get a simple bar chart visualization.
|
| [1] - https://docs.lnav.org/en/latest/sqlext.html#sqlite-
| interface
| tobylvp wrote:
| I mentioned this below, but you could try LogViewPlus. It has a
| built in SQL based reporting feature you can use.
|
| You can see it in action here:
| https://www.logviewplus.com/docs/create_a_custom_report.html
| tobylvp wrote:
| For Windows users, there is LogViewPlus. (I am the author).
| https://www.logviewplus.com/
|
| It has a similar feature set (tail, syntax highlighting, SQL
| reporting) with a focus on accessing files remotely.
| tstack wrote:
| I'm the author of lnav and had never heard of logviewplus
| before, thanks for mentioning it. It looks like we've
| independently arrived at a very similar feature set, that's
| pretty neat. I will now shamelessly steal some of your ideas :)
| tobylvp wrote:
| It looks like you have done a great job with lnav - it's
| impressive. I know how much work has gone into LogViewPlus
| and you must have made a similar commitment. I thought I was
| the only one crazy enough to want to solve this problem. :-)
|
| I would be happy to chat to a like mind if you are
| interested. You can contact me here:
| https://www.logviewplus.com/contact.aspx
|
| Steal away! :-)
| winter_blue wrote:
| Is LogViewPlus heavily tied to Windows? How hard would a port
| to Linux be?
|
| I love that so many Windows tools get a nice-looking GUI,
| versus a heavy CUI lean to *nix tools.
| tobylvp wrote:
| As things currently stand - hard. LogViewPlus is written in
| .Net using WinForms, so it is heavily tied to Windows.
| However, Microsoft is evolving. The bulk of the code can now
| be made to be cross-platform easily, but the front-end
| remains a problem.
|
| I would love to do a Linux port as soon as there is a viable
| cross-platform GUI technology with strong 3rd party vendor
| support. The controls in LogViewPlus are very rich.
| kcbanner wrote:
| I've been using https://glogg.bonnefon.org/. The mark / matches
| feature is really handy. However there are a few bugs with
| highlighting and it hasn't been updated in a while. Will have
| to check this out!
| tobylvp wrote:
| Glogg is a great tool. We take a bit of a different approach
| in that we parse the log file. This enables some more in-
| depth features such as SQL reporting. We also support remote
| file access (for example, SFTP).
|
| Let me know if you have any questions or feedback. You can
| reach me on the contact page of the site.
___________________________________________________________________
(page generated 2023-01-05 23:02 UTC)