[HN Gopher] Show HN: We built a ClickHouse-based logging service
___________________________________________________________________
Show HN: We built a ClickHouse-based logging service
Hey hn! I'm one of the co-founders of highlight.io, an open source
monitoring tool. Today we're sharing a ClickHouse-based logging
solution we've been working on. We wanted to showcase how we built
it and share how you could try it out to give feedback. Since we
started working on highlight.io, we've been hyper-focused on
"cohesion", or ensuring that when you install your monitoring
stack, all of the resources in that stack (user interactions,
requests, traces, logs, etc.) are connected in a consumable way.
We've written up more about our philosophy on this here [1]. We
started building towards this by connecting your client-side app
and your server-side exceptions with session replay and exception
monitoring; i.e. if an error happened in a server-side app, we
would make it easy (with session replay) to trace all the steps
that a user took leading up to it. Especially for larger companies
using highlight.io, the request to tie in logs came up repeatedly,
and we wanted to build this with the same philosophy in mind. Now,
you'll see client-side and server-side logs all in one place,
brought together in the context of a user session, as well as logs
in the context of an error. Like the rest of our stack, this
project is written in Go and Typescript, and for log
ingestion/querying, we're using ClickHouse [2]. Before deciding on
ClickHouse, we were planning to use OpenSearch (an aws fork of
elasticsearch [3]) for this part of our product, but as our traffic
has increased, we encountered quite a few pains with write
throughput for OpenSearch. After evaluating a few options, we
eventually landed with ClickHouse (their cloud offering was icing
on the cake), which has also proven to be much more cost-effective
so far. Building with ClickHouse from scratch has been an exciting
journey. Eric (the mastermind behind this project) wrote a blog
post [4] on a handful of ClickHouse learnings we've gathered since
starting the project. For those wanting to try out the product
locally, you can run the following commands [5]: git clone
--recurse-submodules https://github.com/highlight/highlight cd
highlight/docker; ./run-hobby.sh; To send logs to highlight, you
can use your own OpenTelemetry implementation [6] or use our SDKs
[7] which provide lightweight wrappers over OTEL. Like the rest of
highlight.io, we plan to make money from this with our hosted cloud
offering. For those interested in trying out the cloud-hosted
version, you can get setup at app.highlight.io. To open the floor
for feedback, we would love to get some thoughts on what we've
built so far. Beyond that, what are parts of a logging product you
wish you had with your current setup? And are there any notable
pain-points of using a hosted monitoring product? (We're toying
with the idea of an enterprise deployment). Excited to hear from
everyone. [1]: https://highlight.io/docs/general/company/product-
philosphy [2]: https://clickhouse.com [3]:
https://news.ycombinator.com/item?id=26780848 [4]:
https://www.highlight.io/blog/how-we-built-logging-with-clic...
[5]: https://www.highlight.io/docs/getting-started/self-
host/self... [6]: https://www.highlight.io/docs/getting-
started/backend-loggin... [7]:
https://www.highlight.io/docs/getting-started/overview#for-y...
Author : vadman97
Score : 135 points
Date : 2023-04-20 17:26 UTC (1 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| pech0rin wrote:
| "Deploy a hobby instance in one line on Linux with Docker
| (recommended 16 CPU cores, 32GB RAM, 256GB disk"
|
| Cant say I would call these specs "hobby" at all
| [deleted]
| vadman97 wrote:
| Apologies for the inconsistency here. Our Docker resource
| requirement recommendations were out of date after some recent
| improvements (https://github.com/highlight/highlight/pull/5074
| and https://github.com/highlight/highlight/pull/4993).
|
| Just updated this: 8GB of RAM, 4 CPUs, and 64 GB of disk space.
| that_guy_iain wrote:
| 8GB of ram? For a hobby? Very resource intensive, eh?
| redskyluan wrote:
| Totally agree with that. 8G might be too much to make the
| open source product popular.
|
| I would say 4G may more sense to me, I know how much
| engineering effort it requires though. LOL
| vadman97 wrote:
| We have had folks in our Discord successfully run highlight
| on a Raspberry Pi with 4GB RAM, so our recommendation is
| definitely on the safe side. We're running multiple infra
| services in the docker stack (postgres, opensearch,
| clickhouse, influxdb, kafka, redis) that we would look to
| consolidate in the future to help with running on leaner
| instances.
| JP44 wrote:
| I agree with your sentiment here. imo they chose the wrong
| words to describe what they meant e.g. ~0.5-1GB memory
| usage is more like a hobby-setup (assuming you run it on
| the same hw as the services you monitor).
|
| However, after scrolling through the GitHub page, I feel
| like this is not a service which aims at people (might be
| completely mistaken) who either have a small set of
| services to monitor (and/or understand the logs and/or the
| interest to do it) or have their homelab at a relatively
| low financial priority (1x 8GB 2400 CL17 is EUR15 here).
|
| 8GB for a single service in a (home) environment is, imo,
| still a lot, but I think it is a sort-of reasonable figure
| for what it does and need to do to make that happen
| JimmyAustin wrote:
| Just about anything with only one instance is essentially
| hobby.
| mrits wrote:
| Hard disagree. A hobby instance should be able to be spun up
| in the cloud on a free or near free tier.
| KronisLV wrote:
| > Just about anything with only one instance is essentially
| hobby.
|
| Not high availability? Sure.
|
| However, I've seen software out there that ran as a monolith
| with a single deployment unit, facilitated lots of business
| processes and entire teams of people for continued
| development. Not everything necessarily needs high uptime,
| either. Some software can also serve particular time zones
| and have ample windows for scheduled maintenance, upgrades
| and so on.
|
| There's probably at least a few classifications between hobby
| projects on one end and HA distributed systems on the other.
|
| >> Cant say I would call these specs "hobby" at all
|
| With this, however, I'm inclined to agree. In my eyes,
| "hobby" would imply something more along the lines of: "Just
| give this half a CPU core and about 512 MB of RAM, maybe up
| to a GB of storage depending on what you'll use it for, it'll
| probably work well enough for a few users."
|
| Some software that mostly fits that definition, in my
| experience: Nextcloud, Apache2/Nginx/Caddy, Grav, Mattermost,
| Gitea, Heimdall, YOURLS, PrivateBin, phpBB, Uptime Kuma,
| Zabbix, PostgreSQL, MySQL/MariaDB, Redis, RabbitMQ, Docker
| Swarm and plenty others.
|
| Some software that needs more resources: SonarQube, PeerTube
| (for encoding), OpenProject (Ruby app), Sonatype Nexus
| (bloated Java app, but lots of functionality), Matomo (issues
| with displaying historic data with low resources), BackupPC
| (compresson of backups), K3s and other Kubernetes cluster
| distros and plenty others, too.
|
| Not to say that it somehow makes the software worse, just
| that people have different expectations. Perhaps more
| realistic expectations on my part for hobby software should
| be: "You should be able to launch it with whatever spare
| resources your laptop has."
| goodpoint wrote:
| This is not logging. It's web session telemetry.
| semi-extrinsic wrote:
| As a non-web developer I was also confused about the generic
| terms like "monitoring" etc. and had to scroll a bit before I
| realized it's just for web stuff.
| vadman97 wrote:
| The new feature we're announcing is the server-side + browser-
| side logs ingest, query interface, and alerting engine. Our web
| session recording product is something we've had around but now
| closely integrates with logs to help you debug.
| podoman wrote:
| Why isn't it logging? You can send us raw logs if you like as
| well: https://www.highlight.io/docs/getting-started/backend-
| loggin...
| iudqnolq wrote:
| I suggest adding a pricing tier between $0 and $50/month. I
| nearly bounced until I saw you also offer very reasonable usage-
| based billing. You could just calculate what $5/month of usage-
| based billing could buy and list it as a tier, but having the
| lowest tier be $50 sends an incorrect signal that this isn't for
| hobbyists.
| podoman wrote:
| That's good feedback. We'll update the pricing page accordingly
| when the time comes. Question: do you worry about usage based
| billing with respect to surprise charges? Or do you expect a
| way to limit this?
| pdimitar wrote:
| Both. IMO you should have a plan where the user pays e.g. $7
| and when the resources for it are drained, you start refusing
| requests until the throttle period expires.
|
| It's extremely useful to prototype and experiment with a
| project and have it have a total budget that will not be
| surpassed.
|
| One more idea: pay as you go. I pay $10, that turns out to be
| not enough, I pay another $20 and get immediately unblocked.
| iudqnolq wrote:
| No, I'm perfectly happy with usage-based billing. But when I
| see the cheapest option costing $50 my heuristic is that
| there probably isn't an affordable other option. I was on
| mobile, so the usage-based section was below the fold.
|
| Limiting would be ideal, but all I was suggesting is that you
| indicate above the fold that you have cheaper options
| codegeek wrote:
| How do you compare against something like Signoz (YC Backed). You
| should probably add them to your competitors list since they are
| also Open Source APM.
| vadman97 wrote:
| Highlight is a full-stack observability platform, so our
| recordings start from a user's frontend session and associate
| server-side errors and logs to provide debugging context. We've
| been building quite a lot to bolster the backend observability
| use case, including releasing the new logging product. We can't
| offer feature parity for backend traces and metrics yet, but
| we're planning to get there by the end of the year.
| 0x706B wrote:
| Is this like schema-less? How do you do indexing for individual
| log fields and stuff?
| vadman97 wrote:
| You get structured attribute search without a schema with
| Highlight. More in our docs:
| https://www.highlight.io/docs/general/product-features/loggi...
| metahunter wrote:
| I am curious about the motivation you choose Clickhouse over
| Apache Pinot, and Apache Druid? It could be helpful for other
| folks when choosing the OLAP db from one of them.
| tnolet wrote:
| Slight hijack. I / we went through a very similar tech
| selection process for timeseries metrics (not logging) ~1.5
| years ago. We looked at Druid, ElasticSearch, TimeScale and a
| bunch of others.
|
| Main takeaways were: the SQL flavor and its aggregations in CH
| are amazing. Running on a single node for dev laptops is
| trivial. It's crazy fast with almost zero tuning.
|
| It does not surprise me at all the CH is powering new products
| and startups.
|
| Note: hosted CH did not exist yet. We are using Altinity to run
| our cluster.
| preseinger wrote:
| if you can afford SQL then you're not really doing timeseries
| in any meaningful sense
| metahunter wrote:
| Interesting, just found another post from yesterday about the
| comparison: https://news.ycombinator.com/item?id=35642522,
| though the comparison is coming from Pinot team.
| jiripospisil wrote:
| Not having to deal with a JVM is a major plus tbh.
| douglasisshiny wrote:
| I've seen so many variations of this comment on HN and I'm
| still not sure why not having to deal with the JVM is a major
| plus.
| drowsspa wrote:
| Having to worry about GC in a database is a pretty bad
| experience. It also tends to require way more resources
| than necessary, and just a pretty complex configuration
| preseinger wrote:
| gc isn't the issue, the jvm is the issue
| vetrom wrote:
| JVM runtimes have a relatively high startup cost, are not
| often good 'citizens' in an instance running multiple types
| of software, and the build processes for a lot of JVM
| deliverables is an ungodly mess.
|
| Many of those bells and whistles are near-necessary in the
| enterprise world, but you have the accumulated mass of 'red
| zones' and developmental landmines in that ecosystem that
| can quickly turn you off it as a whole if you want to
| understand the whole system.
| douglasisshiny wrote:
| I still don't understand some of this -- I developed in
| Java for 5+ years.
|
| >JVM runtimes have a relatively high startup cost I think
| many people are okay with that when developing server
| software that's going to run weeks at a time. It can get
| a bit annoying with trying to rapidly iterate. And I
| think things are changing pretty quickly with AOT builds
| and general improvements.
|
| >and the build processes for a lot of JVM deliverables is
| an ungodly mess.
|
| I recall using "mvn package." That's it. This was on two
| different systems that served a good bit of traffic and
| weren't simple trivial projects.
| drowsspa wrote:
| I take it you haven't experienced the hell that is to
| deal with Hadoop JARs. It's absolutely ridiculous.
| jeroenhd wrote:
| I don't know if it's a standard Java thing or just an
| IntelliJ thing, but there's a setting that will hot-patch
| a running JVM when you change code. Things can get messy
| if you (or your dependencies) make assumptions about the
| ClassLoader being used, but other than that it works
| great.
|
| Still not as good as C#'s debugger in Visual Studio (hit
| a breakpoint, edit the code, drag the execution back
| before the problem, resume and run the patched version)
| but nothing I've seen really is.
|
| Setting up Gradle projects is a bit more involved
| depending on your setup, but in the end it's still a
| single command to build an executable JAR.
| douglasisshiny wrote:
| Yeah, it's been a second since I've used IntelliJ/Spring,
| but I recall that being the case as well.
|
| Gradle is something I've never messed with, but that
| makes sense.
| preseinger wrote:
| basically, the jvm is technically sophisticated but
| operationally complicated
|
| it sucks to use
|
| many people believe otherwise, but those people have rich
| jvm experience, which is not easy to get
| wpietri wrote:
| I'm perfectly fine with JVMs, but at a guess, some of it is
| the usual snobbery for anything strange. But some of it is
| due to associating JVMs with enterprise nightmares. And
| some is that JVM tuning is a bit of a dark art. I've made
| some very good money going in and turning JVM knobs that
| others were afraid to touch. (The secret, by the way, is to
| hack together some decent load simulation and then measure
| not just median numbers but things like 99th percentile
| latency.)
| vadman97 wrote:
| For us, a significant reason was the ClickHouse cloud-hosted
| offering, rather than having to manage a cluster ourselves.
| Their use of S3 as the backing storage medium means that large-
| scale data retention is quite affordable.
|
| A good comparison we've referenced:
| https://leventov.medium.com/comparison-of-the-open-source-ol...
| grumblestumble wrote:
| For reference, Apache Druid has an equivalent in Imply
| Polaris, and Apache Pinot has an equivalent in Startree. I
| can't speak for Startree, but Polaris similarly uses S3 for
| backing.
| [deleted]
| dimitrios1 wrote:
| When I was highly engaged with Imply (Druid) a few years ago,
| S3 was also used as a backing storage. Is this not the case
| anymore?
| metahunter wrote:
| I think both Pinot and Druid nowadays offer cloud-hosted
| solutions. Maybe you started early that only ClickHouse had
| that offering. Is cloud hosting the only reason you guys
| choose Clickhouse? I am also wondering is it possible to let
| users choose the data source?
| drowsspa wrote:
| Druid has like 9 different node types and inherits the whole
| Hadoop configuration mess and complexity
| grumblestumble wrote:
| 3, and there's absolutely no need for hadoop, particularly
| with MSQ
| tnolet wrote:
| Anecdata: tried out Druid and Clickhouse for my SaaS.
| Couldn't get Druid working. CH ran in 2 minutes.
| berkle4455 wrote:
| Clickhouse is fast and doesn't have absurd architectural
| complexity.
| podoman wrote:
| +1
| bbu wrote:
| How does this compare to Loki?
| podoman wrote:
| In isolation (just our logging product), Loki is comparable. It
| would be interesting to do a benchmark on Loki and see what a
| comparison looks like. Beyond logging, we do session replay and
| error monitoring and tie all of these things together.
| mekster wrote:
| I think what matters is the interface. A clean interface to
| read structured logs in table view than throw raw text at the
| user like many products do and nice way to filter them.
|
| Loki + Grafana isn't really good for log viewing at all. I
| use Metabase to read logs sent to ClickHouse which gives far
| nicer interface.
| openizr wrote:
| [dead]
| liketochill wrote:
| Did you consider Victoria metrics at all? It is based on
| clickhouse design. I am considering it for time series type
| historian so different Application but I think it could work
| where clickhouse works too.
| podoman wrote:
| Have not seen Victoria! Seems like they don't have a cloud
| offering though?
| vadman97 wrote:
| Haven't heard of it unfortunately. What kind of time series
| data are you thinking of storing? If you're building an
| application on top of it, you're better off using an OLAP DB
| like ClickHouse, InfluxDB, etc. to have access to lower-level
| constructs.
| mekster wrote:
| Does it work well to store logs? I thought it's great for
| metrics.
| glenjamin wrote:
| Do you have any plans in ingest trace/span data in future?
| podoman wrote:
| Yes, we do. It's on our public roadmap:
| https://github.com/orgs/highlight/projects/11/views/1 (marked
| as "Future Work")
|
| We'll likely get to it by Q3 of this year. We hope that the
| design choices we're making (Clickhouse, OTEL, etc..) will set
| us up for supporting tracing easily when we get to it.
| mdaniel wrote:
| Wow, I can't wait to try this out, it could be a Sentry killer
| and doubly so given the friendly license
|
| Also, thank you for introducing me to air
| (https://github.com/highlight/highlight/blob/sdk/highlight-go...)
| as that also looks super handy
|
| _p.s._ for Show HN historians, here is the prior thread:
| https://news.ycombinator.com/item?id=34897645
| podoman wrote:
| Thanks, appreciate the kind note on the license. And yeah, air
| is really nice; we've been using it since the beginning. Gives
| us a "JS hot-reloading" like experience with a server-side app.
| tnolet wrote:
| Isn't sentry also open source and also using clickhouse at
| least in some parts?
| mdaniel wrote:
| yes to the second https://github.com/getsentry/self-
| hosted/blob/23.4.0/docker-... and only after the embargo is
| over to the first: https://github.com/getsentry/self-
| hosted/blob/23.4.0/LICENSE...
|
| I also miss the "good old days" when running sentry was like
| 3 containers, not the 32 of modern Sentry
| tnolet wrote:
| Yeah, fair point. Sentry has added quite some features and
| increased complexity. Probably for good reasons like
| serving large customers. But still
| mekster wrote:
| Why do you want to replace Sentry? It's mature and works well.
|
| Not sure how the license makes a difference in terms of being a
| value to your day to day development.
|
| But I do love projects that can be self hosted, so Sentry is
| very nice on that part, especially when it doesn't limit its
| capability on the self hosted version.
|
| Also this one doesn't even have PHP sdk and the GitHub issue
| doesn't have much of a request for it, which sounds like it has
| still some way to go to be mature given the lack of interest in
| the project at this point.
|
| Couldn't they just reuse other project's sdk or help with
| OpenTelemetry? Developing yet another batch of sdk for a dozen
| language seems like wasted time.
|
| But I do hope we get a nice open source (self hostable) tool
| that can do what it claims (error reporting, tracing, logging,
| metrics) because there's no such thing that does all well at
| the moment that's mature.
| podoman wrote:
| > But I do hope we get a nice open source (self hostable)
| tool that can do what it claims (error reporting, tracing,
| logging, metrics) because there's no such thing that does all
| well at the moment that's mature.
|
| Keep an eye out for Highlight then!
|
| > Couldn't they just reuse other project's sdk or help with
| OpenTelemetry? Developing yet another batch of sdk for a
| dozen language seems like wasted time.
|
| We use OpenTelemetry for all of our SDKs. All we do is thinly
| wrap the SDK so developers don't have to deal with
| OpenTelemetry internal if they don't want to.
|
| There's a doc on it here:
| https://www.highlight.io/docs/general/company/open-
| source/co...
| mekster wrote:
| Thank you for the response and sorry for not doing the
| homework there.
|
| Looking forward to seeing the project take off as we need
| to glue stuff together from various projects to achieve
| your goal now.
| lysecret wrote:
| I started using BigQuery with stream inserts for logging. I have
| around 300k inserts a day und it costs a whole cent a day for
| writes and even less for reads (I don't read much though).
| vadman97 wrote:
| What does you ingest setup look like with BigQuery? Are you
| using something like fluentd to pipe logs over?
| captaintobs wrote:
| Why is this linking to a random markdown page?
| dang wrote:
| Changed from
| https://github.com/highlight/highlight/blob/main/docs-conten...
| now. Thanks!
| vadman97 wrote:
| Apologies for the confusion. We meant to link to our main
| readme:
| https://github.com/highlight/highlight/blob/main/README.md
| francoismassot wrote:
| How does it compare to OpenReplay ?
|
| > Before deciding on ClickHouse, we were planning to use
| OpenSearch
|
| You should have tried Quickwit :)
|
| Anyway, sounds like a great project, best of luck!
|
| [1] https://github.com/openreplay/openreplay
|
| [2] https://github.com/quickwit-oss/quickwit
| vadman97 wrote:
| Interesting, thanks for sharing. Do you implement your own
| storage with Quickwit or can it be backed by a cloud storage
| solution like S3?
|
| Our session replay is similar to OpenReplay but we've focused a
| lot of effort on making a cohesive backend debugging
| experience. Highlight sessions give you backend error
| monitoring and logging out of the box to make it easy to get to
| the root cause of a bug.
| 7sidedmarble wrote:
| I would absolutely try this out but elixir support is a must. Is
| there anything in progress?
| podoman wrote:
| Just added an issue:
| https://github.com/highlight/highlight/issues/5082
|
| We don't have pointed plans for supporting Elixir, but
| hopefully this opens the floor for anyone interested.
| preseinger wrote:
| you can't keep a raw stream of logs in an indexed database like
| clickhouse usefully
|
| volume for any nontrivial organization is too large
| mekster wrote:
| What's the suggestion to do it efficiently?
|
| And what kind of volume is it that ClickHouse can't handle when
| Uber can?
|
| https://eng.uber.com/logging/
| anileated wrote:
| Intuitively it seems like logging and error reporting software
| would be prime targets for supply chain attacks, as they have
| access to a treasure trove of exploitable data. A strategic
| modification in some tiny transitive dependency or two and your
| logging server could log not just for you. Presumably, gathered
| data could be used to deanonymize users or exploit your software.
|
| How do people go about vetting and/or self-hosting those, if you
| operate [software for] a business with actual customers? Is
| stripping sensitive data at the client enough? Do you lock down
| outgoing connections through external networking configuration if
| you self-host? Am I being too paranoid?
| d4mi3n wrote:
| This is an interesting topic I've explored a bit. The tldr; is
| that if your business can't afford or recover from a vendor
| having an incident, you shouldn't be using a vendor. This is
| often why highly regulated industries are slower moving and
| more expensive to operate: You need to either spend a lot of
| time vetting your vendors with policy (via contracts,
| compliance requirements, certification requirements) and
| practice (sign NDAs and review source code for components of
| interest, run joint penetration tests, fund bug bounty
| programs).
|
| That said, there are mitigations you can take. There are end-
| to-end encrypted log solutions out there. Honeycomb.io used to
| have (they might still?) an interesting offering I used at one
| employer to encrypt sensitive fields in logs leaving our
| infrastructure. They had the UI set up to talk to our
| encryption service and decode things on the fly in the user's
| browser-side UI so that they (Honeycomb) never had direct,
| unfettered access to sensitive data.
|
| There are other approaches you can take, but things get tricky
| when you either need to audit your vendor's access to your data
| or assume that your vendor can't secure your data to your
| satisfaction. Better to do it yourself at that point if you
| have the resourcing to do so.
___________________________________________________________________
(page generated 2023-04-21 23:00 UTC)