[HN Gopher] Scaling request logging with ClickHouse, Kafka, and ...
___________________________________________________________________
Scaling request logging with ClickHouse, Kafka, and Vector
Author : mjwhansen
Score : 91 points
Date : 2025-10-08 09:56 UTC (5 days ago)
(HTM) web link (www.geocod.io)
(TXT) w3m dump (www.geocod.io)
| rozenmd wrote:
| Great write-up!
|
| I had a similar project back in August when I realised my DB's
| performance (Postgres) was blocking me from implementing features
| users commonly ask for (querying out to 30 days of historical
| uptime data).
|
| I was already blown away at the performance (200ms to query what
| Postgres was doing in 500-600ms), but then I realized I hadn't
| put an index on the Clickhouse table. Now the query returns in
| 50-70ms, and that includes network time.
| nasretdinov wrote:
| BTW you could've used e.g. kittenhouse
| (https://github.com/YuriyNasretdinov/kittenhouse, my fork) or
| just a simpler buffer table, with 2 layers and a larger
| aggregation period than in the example.
|
| Alternatively, you could've used async insert functionality built
| into ClickHouse:
| https://clickhouse.com/docs/optimize/asynchronous-inserts . All
| of these solutions are operationally simpler than Kafka + Vector,
| although obviously it's all tradeoffs.
| devmor wrote:
| There were a lot of simpler options that came to mind while
| reading through this, frankly.
|
| But I imagine the writeup eschews myriad future concerns and
| does not entirely illustrate the pressure and stress of trying
| to solve such a high-scale problem.
|
| Ultimately, going with a somewhat more complex solution that
| involves additional architecture but has been tried and tested
| by a 3rd party that you trust can sometimes be the more fitting
| end result. Assurance often weighs more than simplicity, I
| think.
| nasretdinov wrote:
| While kittenhouse is, unfortunately, abandonware (even though
| you can still use it and it works), you can't say the same
| about e.g. async inserts in ClickHouse: it's a very simple
| and robust solution to tackle exactly the problem the PHP
| (and some other languages') backends often face when trying
| to use ClickHouse
| frenchmajesty wrote:
| Thanks for sharing I enjoyed reading this.
| tlaverdure wrote:
| Thanks for sharing. I really enjoyed the breakdown, and great to
| see small tech companies helping each other out!
| mperham wrote:
| Seems weird not to use Redis as the buffering layer + minutely
| cron job. Seems a lot simpler than installing Kafka + Vector.
| SteveNuts wrote:
| Vector is very simple to operate and (mostly) stateless, and
| can handle buffering if you choose.
|
| Kafka and Redis is a "pick your poison" IMO, scaling and
| operating those have their own headaches.
| albertgoeswoof wrote:
| Currently at the millions stage with https://mailpace.com relying
| mostly on Postgres
|
| Tbh this terrifies me! We don't just have to log the requests but
| also store the full emails for a few days, and they can be up to
| 50 mib in total size.
|
| But it will be exciting when we get there!
| fnord77 wrote:
| How does Clickhouse compare to Druid, Pinot or Star Tree?
| jamesblonde wrote:
| Here's a good performance study by OneHouse comparing
| Clickhouse, StarRocks, Trino:
|
| https://www.onehouse.ai/blog/apache-spark-vs-clickhouse-vs-p...
|
| Druid is real-time analytics, similar to Clickhouse. StarRocks
| is best at Joins - Clickhouse is not good for joins.
| manish_gill wrote:
| > Clickhouse is not good for joins
|
| This is less and less true as time goes on tbh. 25.9
| introduced Join Reordering as well -
| https://clickhouse.com/blog/clickhouse-release-25-09
| saisrirampur wrote:
| Sai from ClickHouse here. Very compelling story! Really love your
| emphasis on using the right tool for the right job - power of row
| vs column stores.
|
| We recently added a MySQL/MariaDB CDC connector in ClickPipes on
| ClickHouse Cloud. This would have simplified your migration from
| MariaDB.
|
| https://clickhouse.com/docs/integrations/clickpipes/mysql
| https://clickhouse.com/docs/integrations/clickpipes/mysql/so...
___________________________________________________________________
(page generated 2025-10-13 23:00 UTC)