[HN Gopher] Show HN: Open-source Kibana alternative for logs and...
___________________________________________________________________
Show HN: Open-source Kibana alternative for logs and traces in
ClickHouse
Hi HN, Mike and Warren here! We're excited to share some (early)
work towards our next major version of HyperDX. HyperDX makes it
easy to visualize/search logs & traces on top of Clickhouse (so
incident & bug investigations hopefully go by a little easier). For
example, if a team is thinking of migrating to Clickhouse for their
observability data warehouse [1][2][3] usually due to cost or data
privacy reasons, they can easily throw HyperDX on top to do the UI
layer for analysis and dashboarding in a dev-friendly way (aka not
needing to type paragraphs of SQL to find some logs) Over the past
year we've seen a ton of excitement in companies adopting
Clickhouse-based observability stacks - but one of the biggest
challenges we've seen is that the UI layer on top of Clickhouse is
either clunky to use for observability use cases (ex. BI tools), or
too tied to a specific ingestion architecture to scale to every use
case (we used to be in this category!). For companies that needed
more flexibility in how their data is ingested and stored (usually
due to running at a large scale), there's really no good options
for a developer experience (DX) focused observability layer on top
of Clickhouse (Shopify spent 3 years building it in-house!) Our
current release works completely in the browser - and it does this
by building on top of Clickhouse's HTTP interface, which our React
app can directly talk to. This means you can actually try HyperDX
in your browser on your own Clickhouse with no installation! This
was fortunately easy for us to accomplish due to being full stack
Typescript, making it incredibly easy to shift between server and
client code. On top of this we've been spending time baking in
performance optimizations to ensure that HyperDX can continue to
leverage Clickhouse efficiently at larger data volumes. We do a few
tricks like only fetching columns that are needed for the current
search, and re-querying to expand the entire row if needed to fully
leverage Clickhouse's columnar nature (40% faster, ymmv!) - or
rewriting queries to use materialized columns to speed up Map
column access when available (10x faster!). On the DX side: we
support querying using both Lucene (ex. `fullText property:value`)
and SQL syntax. We've found the former to be our favorite for how
concise it is. Similarly for charts, our chart builder has been
upgraded to accept SQL expressions as well, so you can leverage the
full power of SQL, while avoiding typing paragraphs of boilerplate
SQL for time series data. We also make it easy to switch between
UTC/local timestamps! Lastly, we've added high cardinality outlier
analysis by charting the delta between outlier and inlier events (a
la bubble up) - which we've found really helpful in narrowing down
causes of regressions/anomalies in our traces. We have a lot more
planned for the full release - but wanted to get this out early to
hear your feedback and opinions! In Browser Live Demo:
https://play.hyperdx.io/search Github Repo:
https://github.com/hyperdxio/hyperdx/tree/v2 Landing Page:
https://hyperdx.io/v2 [1]: https://www.uber.com/blog/logging/ [2]:
https://blog.cloudflare.com/log-analytics-using-clickhouse/ [3]:
https://www.youtube.com/watch?v=LDj3_jMsCXg&list=PLvQF73bM4-...
Author : mikeshi42
Score : 43 points
Date : 2024-11-12 18:04 UTC (4 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| maxthegeek1 wrote:
| We use HyperDX for our observability! We had been using google's
| observability suite before, because we're using GKE anyways, but
| HyperDX's search over traces is just waaaay better and I can't go
| back.
| miah_ wrote:
| Whats hilarious is that Kibana started out as a Open Source.
|
| Hard to trust anything released as OSS these days that hits this
| site run by a for profit company.. Its all destined to have a rug
| pull after some VC funding. Considering HyperDX is a for profit
| company, I'm sure we won't have to wait long!
| bboreham wrote:
| Kibana is once again Open Source, as of 2 months ago.
| https://github.com/elastic/kibana/blob/main/LICENSE.txt
| mathfailure wrote:
| What do we say in such cases? It was good while it lasted!
|
| Once that happens - eventually some new kids would appear on
| the block.
|
| Such is the life.
| DAlperin wrote:
| Super neat! Does the v2 branding mean that the more "fully
| featured" observability product is going away? Or is it all going
| to be rebuilt on top of clickhouse?
| akdor1154 wrote:
| Very interested - I'm currently toying with Grafana set up in the
| same way, i wonder how this compares?
| est wrote:
| Hmm, this is not the "Kibana" alternative I imagined.
|
| Kibana was supposed to be an easy UI. You go to Discover, and the
| data automatically shows in chronological order, I can explore it
| with different options.
|
| Kibana is very suitable for non-tech or less-tech people. I hope
| your product find a clear target audience. With too much ES query
| JSON or SQL it would scare people off.
___________________________________________________________________
(page generated 2024-11-12 23:00 UTC)