[HN Gopher] Show HN: Log collector that runs on a $4 VPS
       ___________________________________________________________________
        
       Show HN: Log collector that runs on a $4 VPS
        
       Hey guys, I'm building erlog to try and solve problems with
       logging. While trying to add logs to my application, I couldn't
       find any lightweight log platform which was easy to set up without
       adding tons of dependencies to my code, or configuring 10,000
       files.  ErLog is just a simple go web server which batch inserts
       json logs into an sqlite3 server. Through tuning sqlite3 and
       batching inserts, I find I can get around 8k log insertions/sec
       which is fast enough for small projects.  This is just an MVP, and
       I plan to add more features once I talk to users. If anyone has any
       problems with logging, feel free to leave a comment and I'd love to
       help you out.
        
       Author : Nevin1901
       Score  : 15 points
       Date   : 2023-02-11 20:44 UTC (2 hours ago)
        
 (HTM) web link (github.com)
 (TXT) w3m dump (github.com)
        
       | withinboredom wrote:
       | Neat! Have you considered using query params instead of bodies,
       | then just piping the access logs to a spool (no program actually
       | on the server, just return an empty file). Then your program can
       | just read from the spool and dump them into sqlite.
       | 
       | That should tremendously improve throughput, at the expense of
       | some latency.
        
       | unxdfa wrote:
       | I see your idea but you could drop the JSON and use rsyslogd +
       | logrotate + grep? You can grep 10 gig files on a $5 VPS easily
       | and quickly! I can't speak for a $4 one ;)
        
       | folmar wrote:
       | Sorry, but I don't see the selling point yet. Rsyslog has
       | omlibdbi module that send your data to sqlite. It can consume
       | pretty much any standard protocol on input, is already available
       | and battle proven.
        
       | keroro wrote:
       | If anyones looking for similar services Im using vector.dev to
       | move logs around & it works great & has a ton of
       | sources/destinations pre-configured.
        
       | Dachande663 wrote:
       | I've found the hard part is not so much the collection of logs
       | (especially at this scale), but the eventual querying. If you've
       | got an unknown set of fields been logged, queries very quickly
       | devolve into lots of slow table scans or needing materialised
       | views that start hampering your ingest rate.
       | 
       | I settled on a happy/ok midpoint recently whereby I dump logs in
       | a redis queue using filebeat as it's very simple. Then have a
       | really simple queue consumer that dumps the logs into clickhouse
       | using a schema Uber detailed (split keys and values), so queries
       | can be pretty quick even over arbitrary fields. 30,00 logs an
       | hour and I can normally search for anything in under a second.
        
       | harisamin wrote:
       | Ah cool! Somewhat related I built a json log query tool recently
       | using rust and SQLite. Didn't build the server part of it
       | 
       | https://github.com/hamin/jlq
        
       | Hamuko wrote:
       | I feel like if you're going to use "$4 VPS" as a quantifier, you
       | could at least specify which $4 VPS is being used.
        
         | teruakohatu wrote:
         | DO's 512mb basic VPS starts at $4, so I am guessing it is that.
        
       ___________________________________________________________________
       (page generated 2023-02-11 23:00 UTC)