[HN Gopher] Mach: A Pluggable Metrics Storage Engine for the Age...
___________________________________________________________________
Mach: A Pluggable Metrics Storage Engine for the Age of
Observability [pdf]
Author : greghn
Score : 15 points
Date : 2022-11-27 18:53 UTC (4 hours ago)
(HTM) web link (www.cidrdb.org)
(TXT) w3m dump (www.cidrdb.org)
| catiopatio wrote:
| I know that naming is hard, and almost every short name has
| already been taken, but "Mach" is _really_ already taken:
|
| https://en.m.wikipedia.org/wiki/Mach_(kernel)
|
| Mach is still quite relevant on Apple's platforms, very visible
| in public APIs, and you'd be ill-advised to try to claim symbols
| with a "mach_" prefix.
| valyala wrote:
| VictoriaMetrics core developer here.
|
| Intetesting paper! It mentions VictoriaMetrics, but doesn't
| compare its write and read performance and scalability to Mach.
| It would be great performing such a comparison, since
| VictoriaMetrics should scale better than both Prometheus and
| ImfluxDB (the systems used in performance comparisons) with the
| number of available CPU cores according to [1].
|
| It looks like Mach maintains an in-memory append-only buffer per
| each active time series with 4KiB size. This means that its
| memory usage can be estimated as 4KiB * active_series. For
| example, it needs at least 4GiB of RAM for a million of active
| time series. This is 4x bigger than VictoriaMetrics' memory
| usage.
|
| A 4KiB in-memory buffer may contain up to 4KiB/16=256
| (timestamp,value) samples. If a time series receives every new
| sample every 10 seconds, then 256 samples may cover up to 42
| minutes worth of data. If per-series in-memory buffers are
| persisted when they become full, then Mach may lose up to 42
| minutes of data on unclean shutdown such as out of memory crash
| or power off. This looks like not very good tradeoff for higher
| data ingestion speed.
|
| It is also unclear how efficiently does Mach compress data stored
| on disk. Production stats from VictoriaMetrics users show that
| the needed disk space is the most expensive part of their
| monitoring stack. That's why on-disk data compression efficiency
| is very important here. VictoriaMetrics provides better on-disk
| metrics compression than other monitoring solutions at the cost
| of slighlty higher CPU usage [2].
|
| [1] https://valyala.medium.com/measuring-vertical-scalability-
| fo...
|
| [2] https://faun.pub/victoriametrics-achieving-better-
| compressio...
| [deleted]
| candiddevmike wrote:
| I just want one binary to store metrics, traces, and logs
| awestroke wrote:
| That would make way too much sense
| greghn wrote:
| According to the paper, that's one of the goals of the project:
|
| _" In the future, we plan to investigate several natural
| extensions to Mach. Specifically, we believe that metrics
| workloads exhibit similar characteristics to other
| observability data types: logs, events, and traces [20, 21].
| For example, integrating log data will require the
| investigation of fundamental design considerations (e.g.,
| trading off compression speed with size, enabling search over
| compressed data), whereas supporting traces might necessitate
| the development of entirely new data models (e.g., graph-based
| approaches to facilitate root cause analysis). Our ultimate
| goal is to establish Mach as a unifying storage engine for
| observability workloads."_
| ignoramous wrote:
| We (ab)use our logging pipeline for metrics too (can afford to
| do so because of our rather meagre scale, which also means we
| don't really need tracing much).
|
| Whenever metrics are needed, we pull (hive-compatible time-
| partitioned) logs (transformed to parquet or csv) from the blob
| store into DuckDB and on-demand query from there. No online
| dashboards, but pretty good for the cost.
|
| https://duckdb.org/docs/extensions/httpfs
| Terretta wrote:
| Paper from CIDR'22, January 9-12, 2022, Chaminade, USA
|
| _"This paper describes Mach, a pluggable storage engine we are
| building specifically to handle high-volume metrics data. Similar
| to many popular libraries (e.g., Berkeley DB, LevelDB, RocksDB,
| WiredTiger), Mach provides a simple API to store and retrieve
| data. Mach's lean, loosely coordinated architecture aggressively
| leverages the characteristics of metrics data and observability
| work- loads, yielding an order-of-magnitude improvement over
| existing approaches--especially those marketed as 'time series
| database systems' (TSDBs). In fact, our preliminary results show
| that Mach can achieve nearly 10x higher write throughput and 3x
| higher read throughput compared to several widely used
| alternatives."_
___________________________________________________________________
(page generated 2022-11-27 23:01 UTC)