[HN Gopher] Show HN: Piping logs, visualizing in a web app - jus...
___________________________________________________________________
Show HN: Piping logs, visualizing in a web app - just suffix "| npx
logscreen"
Author : soorajsanker
Score : 115 points
Date : 2024-03-06 06:42 UTC (16 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| lionkor wrote:
| Neat, thats a pretty good use of a web app.
|
| How is it handled when I want to view logs that are too large to
| fit in memory?
| number6 wrote:
| Install more memory
| codetrotter wrote:
| Download more RAM
| soorajsanker wrote:
| Depends on the size, for me it worked well with decent size
| JSON log lines, but if it is too much that doesn't fit on the
| browser memory, can think of index storage of sqlite in the
| future. Search and filtering can be then loaded to the server +
| sqlite setup.
| wiseowise wrote:
| > This tool has been generated entirely with ChatGPT and hence it
| is also a demonstration of how useful products can be developed
| with collaborative reiteration with GPTs.
|
| All you need to know.
| jasonjmcghee wrote:
| I was going to ask about the strange combination of
| package.json and inlined cdn js dependencies (especially
| strange because it's a local server)
|
| Suppose that answers it.
| soorajsanker wrote:
| Haha yeah, I wanted to frame this as a single file tool, and
| hence all the quick wins. But later moved the react app into
| a separate file because I was not getting the syntax
| highlights and all.
|
| Definitely this can be made better with proper packaging and
| tooling! As I mentioned, more than the idea, implementation
| was an experiment on the ability of collaborative iterations
| with GPTs- it was fun!
| jasonjmcghee wrote:
| Oh- how much manual intervention was there? It was not
| entirely generated?
| bilekas wrote:
| Yeah, there are better ways to do this, I wonder if it's purely
| an excersize in cgpt useages.
| soorajsanker wrote:
| True, This was also a personal experiment in understanding
| the level of collaboration that someone can do with GPTs
| during the product iterations.
| fragmede wrote:
| > All you need to know.
|
| For those of us not in the know, what does that mean?
| KronisLV wrote:
| > For those of us not in the know, what does that mean?
|
| I reckon that some people are either critical of LLMs due to
| the data that they're trained on (concerns about the legality
| of using the outputs, or the ethics of taking open source
| projects with a variety of licenses and how transformative or
| not the output might be), or question the quality of the code
| that they might output. Oh, also there's hype cycles and
| right now people at large might overestimate the capabilities
| of what LLMs can or cannot do well, which can be tiring, as
| seeing constant posts about crypto was (where we had a
| solution in search of a problem).
|
| In my eyes, they can be used in conjunction with something
| like IntelliSense and IDEs to solve select problems (ones
| with solutions that already exist out there) more quickly, as
| well as save time on some boilerplate, as long as you still
| validate the outputs and actually check if everything works.
| Nowadays I'm using GitHub Copilot and the user experience is
| mostly okay, I'd say it actually lets me write code faster,
| at the expense of occasionally getting things pretty wrong,
| but is still a net positive, other considerations aside.
| fragmede wrote:
| In that case, I'd quote the guidelines to GGP poster:
|
| > Please don't fulminate. Please don't sneer, including at
| the rest of the community.
|
| https://news.ycombinator.com/newsguidelines.html
| CaptainFever wrote:
| If it works, I don't see anything wrong with it. It looks good.
| Hating on projects that disclose their GPT usage will only
| incentivize them to not disclose in the future.
| h1fra wrote:
| As much as I love to hate on chatgpt engineer, the code is
| definitely okay; not much difference from a project you would
| write quickly in a single weekend.
| oneeyedpigeon wrote:
| The first thing that struck me when I looked at some of the
| code was: <div className="flex items-start
| border-b border-gray-300 py-2 hover:bg-blue-100 cursor-
| pointer transition-all"
|
| I'm not saying that humans never come up with this kind of
| crap, but it feels like it's going to be a LOT more common
| the more we resort to AI. I wonder if "... and make the code
| easy-to-read so that it's maintainable in future" is ever
| included in prompts for LLM-generated code.
| Kiro wrote:
| That's just Tailwind and exactly how my HTML looks as well.
| janpmz wrote:
| I'm a clean code guy and tailwind made my code look
| terrible. But I gave it a chance because I heared so many
| positive things about it. Now I'm happy with it. I don't
| have to jump around from HTML to CSS and back to HTML.
| agos wrote:
| that's the horror of Tailwind for you, which is different
| from the horror of AI
| addandsubtract wrote:
| Software development is horror all the way down. The
| sooner you embrace the horror, the easier life gets.
| chrissoundz wrote:
| That is just tailwind CSS, one the most common CSS libs out
| there I would think. Your comment is actually a lot more
| telling of something as opposed to this AI written project
| though.
| karmakaze wrote:
| > Features Web-Based Log Viewer: Get a
| cleaner and more organized view of command outputs.
| Real-Time Updates: Logs are displayed in real-time as the
| command executes. Interactive Interface: Search,
| filter, and navigate through logs easily. // TODO
|
| ChatGPT has its limits // TODO
| asplake wrote:
| Very tempted! Installation instructions?
| soorajsanker wrote:
| Simply suffix with any unix commands that emits stdout, ` | npx
| logscreen` (need npx though)
| tchetwin wrote:
| `npx` is a tool that ships alongside `npm` (alongside `node`)
| that either finds a local installation or performs a just-in-
| time download, and then runs the default "bin" entrypoint for
| the package.
|
| Newer versions of `npx` (last few years) will prompt the user
| before installing a package. Older versions did not. Note that
| installing the package can run arbitrary scripts as a side
| effect with at least the level of permission of the current
| user, so there is implicit trust required of the author, and
| the authors of all transitive dependencies.
| asplake wrote:
| Thanks. Unfamiliar with the Node ecosystem, that escaped me
| jealousgelatin wrote:
| Looks neat. Don't listen to the haters
| mg wrote:
| Is there a benefit over command | vim -
|
| ?
|
| I love doing that, because then I get all the power of Vim to
| look at the output. Search, regex search, sorting, bookmarks,
| counting, macros ...
|
| I started to write output that is indented, so I can use Vim's
| folding to comfortably look at complex data.
|
| For example if you have entries like this:
| 2023-04-12 17:22 Sighted and UFO Location: 27.1742424
| 12.137234 Temperature: 24degC Observer:
| Joe Actually it turned out to be a dove.
|
| And you press zM in Vim, they all turn into:
| 2023-04-12 17:22 Sighted and UFO + 4 lines
|
| So you have a nice overview and can open individual entries by
| putting the curser on them and hit zo.
|
| And while folded, you still have all the power of Vim to search
| within them. Regex search. Count by any measure you like etc etc.
| Actually, you can do anything, as Vim allows you to execute
| arbitrary vim commands, vim macros or shell commands on your
| buffer.
|
| Wanna know how many doves typically get mistaken for UFOs around
| New York grouped by month? Write a script for it and it will be
| forever at your fingertips.
| CGamesPlay wrote:
| I think this is more designed for streamed output rather than
| viewing files. Vim doesn't seem to support that, in fact it
| just says "Vim: Reading from stdin..." forever.
| mg wrote:
| Hmm.. I guess you could write a little script (2 lines?),
| let's call it "vimlive" which reads from stdin, streams to a
| temp file and opens that temp file in vim.
|
| Then with: command | vimlive
|
| You get vim with the current state of the data and you can
| update it with :e
| darrenf wrote:
| You can have vim constantly refresh a buffer if it's
| modified externally, like this turns your buffer into a
| kinda-sorta-version of `tail -f`: :set
| autoread | au CursorHold * checktime | call feedkeys("G")
|
| So if it's already a file, just open it with vim; if it's a
| STDOUT/STDERR stream, redirect/`tee` to a file and open
| that.
| pletnes wrote:
| Also it's not as snappy when you cross the 1GB limit,
| although it does work, mostly.
| kevincox wrote:
| With a quick look at the code I'm pretty sure this WebUI
| also wouldn't be snappy along before that.
| adql wrote:
| try vim filename
|
| lmao
| fragmede wrote:
| This is pretty neat! I did _brew upgrade | npx logscreen_ as the
| first thing I could think of that would generate logs to play
| with this with.
| gatolgaj wrote:
| This is very neat. It would be super cool if we could extent it
| to multiple processes. From the Looks of it , it should be very
| much possible.
| piterrro wrote:
| Congrats on launching! Its interesting to see that ChatGPT is
| able to build something I've been working on (https://logdy.dev)
| for last two months. I guess new era is around the corner then.
| rnts08 wrote:
| Tbf your tool looks 110% more useful and well built than a
| simple single screen showing the logs with an in-browser
| search. Keep at it!
| piterrro wrote:
| Thank you! I have a full roadmap of features to deliver :-)
| mittermayr wrote:
| Your landing page looks fantastic, just wanted to throw that
| out (before ChatGPT starts making that, too)
| piterrro wrote:
| Thanks!
| marmaduke wrote:
| I was looking for something like this for a while, and it could
| be just the landing page, but I much prefer a tool that's been
| built slowly with thought and consideration (which is evident
| from your landing page).
|
| How does your tool handle semi-permanent usage or very long
| logs? For instance, I would run this to view logs from an ssh
| server that's been up for 6 months, when systemd has rotated
| the log file etc.
| emadda wrote:
| Why would you watch logs for more than 6 months?
|
| Do you filter them? What does the system you are watching do?
| piterrro wrote:
| Thanks for your kind words. I have a feature onnthe roadmap
| which is very close to delivery that will allow you to load a
| big log file and browse it through the UI. As long as you can
| load the content from disk or remotely, Logdy will be able to
| handle that. The aim isnto provide DataDog like experience
| but in a portable and temporary way, that allows you to set
| it up quickly, check what you need, tear down and move on.
| All without much hassle or costs for sending the logs to a
| remote service.
| neomantra wrote:
| This is a great tool too and echoes task teleportation, which
| is popping up more and more.
|
| There's a new era, but one in which you will be even more
| powerful. LLMs will improve in pumping out code embodiments...
| but it was your human creativity borne out of operations
| frustrations, coupled with your care to do something about it,
| that manifested your product.
| piterrro wrote:
| Thank you for this comment, you got a point and this is
| indeed a good thing that's happening!
| corytheboyd wrote:
| Please keep working on this! I started a similar tool because I
| desperately needed it, but haven't found the time to maintain
| it. https://github.com/corytheboyd-smartsheet/json-log-explorer
|
| Having something like the log explorer in DataDog for local
| development is so insanely useful
| piterrro wrote:
| That was the idea exactly: I wanted to have DD log explorer
| locally that works insanely fast to be coupled within
| development process. As much as I love terminal, its just so
| limiting when it comes to browsing and filtering through more
| content.
| corytheboyd wrote:
| Absolutely agreed, and glad someone else is out there
| fighting the good fight :)
| tstack wrote:
| > As much as I love terminal, its just so limiting when it
| comes to browsing and filtering through more content.
|
| What terminal-based tools are you thinking of when you say
| this? Just tail/grep/less?
| piterrro wrote:
| that and more magic of sed, awk, jq and similar. I know
| you can do real magic in there but still, I believe it's
| easier to learn GUI than vim shortcuts or sophisticated
| jq/awk formulas. Still, I admire terminal for being
| around for so long and follow it as an example when
| building Logdy.
| edudobay wrote:
| Nice, will give it a try!
|
| There is a similar tool that I've been using for years:
| https://github.com/mthenw/frontail/
| kevincox wrote:
| I'm not super familiar with express but this seems to be
| listening on external interfaces as well:
|
| https://github.com/soorajshankar/logScreen/blob/f8b29aaef428...
|
| So this will be sharing your logs with the world if you run this
| outside of a trusted network. (And personally I avoid trusting
| any networks)
| soorajsanker wrote:
| Are you talking about the difference between 0.0.0.0 and
| 127.0.0.1 /localhost (loopback address)? If so It should be a
| tiny thing to change to implement I can add that on the next
| update- I just went with the express JS defaults to keep things
| simple:)
|
| Or I can even make a flag to configure this :thinking
| ggpsv wrote:
| `app.listen` uses the same signature as Node's
| `server.listen`. You can specify the `host` [0] or else it
| defaults to `0.0.0.0`. This means that when you run this
| service, it will be accessible to anyone that can reach the
| server.
|
| Typically, services allow you set the host address by
| configuration. This way you can set it to `127.0.0.1` so that
| it is only available locally.
|
| [0]: https://nodejs.org/api/net.html#serverlistenport-host-
| backlo...
| gothink wrote:
| If you do make it configurable, it's still a good idea to
| make the default `127.0.0.1` for the reasons @ggpsv
| mentioned. Since the link automatically opens at
| `http://localhost:PORT` anyway, it's an easy security win.
| Calzifer wrote:
| It does, but express doesn't tell it directly in the docs.
|
| Instead http://expressjs.com/en/4x/api.html#app.listen sends
| you to https://nodejs.org/api/http.html#http_server_listen
| which also won't give the answer but sends you to
| https://nodejs.org/api/net.html#serverlisten which at some
| points finally documents that default is 0.0.0.0.
|
| Welcome to the JavaScript ecosystem :/
| throwitaway1123 wrote:
| I wouldn't exactly call this a failure of the JS ecosystem.
| The Express documentation you've linked to mentions that the
| host is the (optional) second argument to the listen
| function:
|
| > app.listen([port[, host[, backlog]]][, callback])
|
| You literally just have to run:
| app.listen(3000, '127.0.0.1')
|
| If you don't trust the network you're on you should just be
| explicit about which IP address you'd like to listen on.
| stevage wrote:
| That's a sensible default. It would be super frustrating if
| by default the web server you built couldn't be accessed by
| anyone else in your network.
| neomantra wrote:
| This is handy stuff, thanks for sharing. We are working on
| transporting tasks back and forth from TUI<>Web<>App<>VR ; this
| is a great and straightforward example of that.
|
| I use VSCode CoPilot constantly as simple augmentation and
| typing-saving -- not for wholesale creation of code. I've seen my
| pre-novice daughter use ChatGPT and the results were mixed (good
| in some ways, horrible in others).
|
| Can you elaborate more on your ChatGPT/LLM working style for
| this?
| soorajsanker wrote:
| Sure thing! The collaboration with ChatGPT was like having a
| friendly coding buddy. I threw in ideas, got feedback, and we
| built this tool together. We chatted about names, features, and
| even how to post it. It's like having an AI pal who's great at
| brainstorming and coding. Fun times!
|
| ^ asked Chat GPT to generate this from my earlier prompts :)
|
| Some other notes, I am a developer who love to make usable
| products, and these years I learned the usability is most
| important especially when you are developing in the early
| stages of a product. So I fed some ideas, asked suggestions,
| gave some tech stacks (for example, I asked chatGPT to not go
| with webpack, instead simply load React from CDN for quick
| iteration. and suggested memoisation on React to render things
| faster). So process wise similar to any other product
| development, but it's like having someone listen to your
| feedback and instructions and do the work- I would like to call
| collaboration!
| neomantra wrote:
| Thanks for those details!
| corytheboyd wrote:
| > Interactive Interface: Search, filter, and navigate through
| logs easily. // TODO
|
| This is exactly why you would show logs in the browser though--
| this is not a nice to have, it is THE core feature. The browser
| is perfect for building the complex UI necessary for filtering
| and making sense of log streams.
| reidjs wrote:
| That's probably why it's on their TODO list
| soorajsanker wrote:
| Ahh actually you can do search through the logs in real-time, I
| forgot to split that and remove TODO from the readme. Complex
| filtering based on the timestamp is yet to implement.
| simonw wrote:
| Found the implementation: https://github.com/soorajshankar/lo
| gScreen/blob/f8b29aaef428...
| simonw wrote:
| "This tool has been generated entirely with ChatGPT and hence it
| is also a demonstration of how useful products can be developed
| with collaborative reiteration with GPTs."
|
| Any chance you could share a link to the ChatGPT conversation, or
| a copy of the prompt you used?
|
| I build little tools like this with ChatGPT all the time, it's
| one of my favourite use-cases for it. I often have an idea for a
| tool that wouldn't justify an hour of development work but is
| absolutely worth spending 5 minutes iterating on with ChatGPT.
| samsquire wrote:
| Good idea. Thank you for sharing.
|
| One thing I've always wanted was simple filters on log lines in a
| GUI that let me collapse tags in the log line. I created a proof
| of concept that showed logs as a tree.
|
| Or I can say I want logs from the point where key=value and until
| method x.
|
| A bit like a debugger but for logs. Show logs that are between
| these facts.
| wfn wrote:
| > showed logs as a tree.
|
| _Yes_ , that is exactly what I want as well - a tree-like
| structure where you can trace events across multiple layers of
| abstraction as well as across systems (so both horizontal and
| vertical relations of events, so to speak).
|
| Some infra systems output syslog loglines and there's no
| changing that, and I still want to capture and relate these log
| messages to other events (incl. high-level business
| transactions / processes).
|
| Any way you could share your PoC? :) very interested.
| samsquire wrote:
| Thanks for your lovely reply.
|
| I don't have the code anymore might have been called
| "logtree", it was a HTML file and very small, it would parse
| dot "." separated lines such as (user.connected) and arrange
| them in a tree with indentation, if you clicked the dot
| syntax it would highlight in yellow other instances of that
| bucket of entries later in the log file.
|
| I think you want elegant navigation between linear sequences
| (next, previous) of log lines that are apart in time, but
| related.
|
| You might want to see all events of the internals of two
| different components in the tree structure.
|
| Need to process log lines into a graph and then provide quick
| navigation for them.
|
| It is indeed horizontal and vertical.
|
| This would be very useful to anti-abuse measures if you could
| trace to the cause of something and cancel all thing caused
| by it.
| psim1 wrote:
| This is a solution looking for a problem. The browser is a
| horrible place for viewing text logs. Companies like Splunk and
| Datadog put countless hours into trying to make it work and it's
| never not-painful to view logs in a browser.
| soorajsanker wrote:
| What are the main specific issues that you felt difficult in
| viewing logs on a browser? imagine there is better
| visualisation, parsing, highlighting, filtering, collapse etc.?
| I feel like the terminal is where I spend more time looking for
| the relevant parts from the logs.
| dloss wrote:
| Some CLI viewers for logs are listed on
| https://github.com/dloss/klp, as alternatives to my own little
| tool "klp".
| wetmore wrote:
| What's better? I ask out of ignorance, but personally I think
| Datadog is amazing for viewing logs.
| liveoneggs wrote:
| Did you know AWStats (first released 2003) could accept streaming
| logs and produce nice dashboards? https://awstats.sourceforge.io/
___________________________________________________________________
(page generated 2024-03-06 23:02 UTC)