Post AaKTC4v7OOnKTOxqds by bahmanm@mastodon.social
(DIR) More posts by bahmanm@mastodon.social
(DIR) Post #AaKFx8vyt6mfAruqZc by galdor@emacs.ch
2023-10-01T12:11:22Z
0 likes, 0 repeats
Now that I've seen the price of Elastic Cloud (and no I don't want to deal with Java software on my servers), I really want to build a sane Open Source log storage platform. Store raw data on a S3 or on a NAS, treat indices are disposable, and split in shards to scale. I need more time.
(DIR) Post #AaKJGMFjtRN9qW0mGW by bahmanm@mastodon.social
2023-10-01T12:48:11Z
0 likes, 0 repeats
@galdor Have you checked sumo logic? It's quite robust and has got a decent UI/UX. Not sure if pricing matches your budgets though.
(DIR) Post #AaKKnVjTljFCl0IK6i by galdor@emacs.ch
2023-10-01T13:05:40Z
0 likes, 0 repeats
@bahmanm I don't want yet another Datadog clone, I want a simple and practical Open Source system to store logs and search through them.One of those numerous cases where if you want something, you're going to have to make it yourself.
(DIR) Post #AaKPSAP4NI0EINOEWO by louis@emacs.ch
2023-10-01T13:57:47Z
0 likes, 0 repeats
@galdor @bahmanm Fully agree, the whole log storage solution spectrum is highly overcommercialized and overpriced. Their UIs are mostly bloated and slow. And I don't want to learn yet another "for-purposes-of-lock-in-invented" *QL language to query the logs.I moved our log storage needs to plain old Postgres and some shell scripts which deprecate old log entries (I don't need long-term storage). It might not be the most efficient solution but it is easy to manage and can be self-run on some cheap VMs.Datadog et al. are clearly more suitable for "enterprise" clients who have excess money to burn.
(DIR) Post #AaKPhJxb7mYkHNlZCq by louis@emacs.ch
2023-10-01T14:00:32Z
0 likes, 0 repeats
@galdor Btw. MySQL/MariaDB have a "CSV" and "ARCHIVE" and MariaDB also supports native S3 storage for table data. Might be a solution for you?
(DIR) Post #AaKQFfpQKFd5wqK7BQ by galdor@emacs.ch
2023-10-01T14:06:46Z
0 likes, 0 repeats
@louis @bahmanm I learned years ago that a company cannot make decent money on a product with "just" log storage/search; pricing is dominated by storage cost, it's just not interesting. Companies doing just that end up sold and integrated to some kind of platform (e.g. Logmatic bought by Datadog). This is also why Elastic is so much more than just logs.Relational databases work for tiny volumes, but it's not the right tool for the job (good luck storing gigabytes of logs a day with months of history, and yes this is a small number). And you still have to write a decent web interface.
(DIR) Post #AaKRRQR5TqMLdd6kds by bahmanm@mastodon.social
2023-10-01T14:20:06Z
0 likes, 0 repeats
@galdor Not sure I agree w/ a custom solution for this unless there's a pressing need. Software is a liability which needs constant maintenance and updates - esp if the only reason for not waning the self-hosted solution is JVM 😂 —@louis I like your approach. In fact, I did it back in 2018-9. But even that low-tech solution turned out to be too much burden when things got complicated (eg multiple datacentres). In the end I settled for the self-hosted ELK.
(DIR) Post #AaKRSzuB5mGTvwSqx6 by bahmanm@mastodon.social
2023-10-01T14:20:21Z
0 likes, 0 repeats
@galdor @louis Once has to pick their battles carefully, IMO.
(DIR) Post #AaKSLgj66FH4NYMNyC by galdor@emacs.ch
2023-10-01T14:30:17Z
0 likes, 0 repeats
@bahmanm @louis Fortunately there are people who simply write better software instead of just using what exists. If they did not do so, they would be no progress.
(DIR) Post #AaKTC4v7OOnKTOxqds by bahmanm@mastodon.social
2023-10-01T14:39:43Z
0 likes, 0 repeats
@galdor @louis I think you're missing my point. What I was trying to get at, was not doubting yours and others' competencies. Rather I was getting at the point that releasing software is a demading bond and commitment.I write something **iff** I feel I'm not stretched too thin and I can nurture and grow it along w/ the rest of my liabilities for the years to come.Now you probably are feeling like that. In that case, more power to you 💪
(DIR) Post #AaKTvPvg80UQf9qUHQ by galdor@emacs.ch
2023-10-01T14:47:56Z
0 likes, 0 repeats
@bahmanm @louis Ah I understand your point of view better. I sort projects into different categories:1. What I do professionally (i.e. for money), where it is all about being pragmatic.2. What I do to fix my own problems, where I do indeed try to be sure I can finish what I build because I want to fix the problem.3. What I write to learn or just for fun, hoping that it could be of use for 1. or 2. if it gets successful.Here I was thinking of 3.
(DIR) Post #AaKdSzX3g4DoE8Pj5U by bahmanm@mastodon.social
2023-10-01T16:34:48Z
0 likes, 0 repeats
@galdor @louis Oh, now I see where you were coming from 🙂Your seriously serious tone (😆) and the fact that I had to deal with this professionally only a few years ago, made me thinke it was a category 1 😁Go for it then!