[HN Gopher] Vector: A high-performance observability data pipeline
___________________________________________________________________
Vector: A high-performance observability data pipeline
Author : tosh
Score : 45 points
Date : 2024-03-17 19:14 UTC (3 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| nextworddev wrote:
| Is this a Splunk alternative?
| ikut3hva wrote:
| No, it is an opensource of timber.io but now Timber was
| acquired by Datadog ;)
| atombender wrote:
| Not directly. Vector is a tool to build pipelines that receive,
| transform, and send data. But it doesn't index data to make it
| searchable. You can use it to ingest into Splunk, however.
|
| Vector could be used to build something Splunk-like. For
| example, you can use it to ship logs into Kafka, then let it
| ingest that data into ClickHouse, and then use a frontend like
| Grafana to search logs using ClickHouse.
| makmanalp wrote:
| It's more like a fluentd alternative, with a lot of
| improvements: much less sluggish throughput if you're doing
| anything nontrivial, nicer overall architecture and language,
| more 'batteries included' integrations ecosystem.
|
| We're in the process of switching to this at work for some very
| high volume logs and I'm quite hopeful - other teams saw pretty
| decent improvements.
| nextworddev wrote:
| It'd be awesome if a simple UI + hosting is provided if it's
| in your roadmap, I'd be interested in paying for that (as a
| startup segment)
| loloquwowndueo wrote:
| I don't think hosted (saas?) Vector makes much sense. It's
| meant to sit next to your deployed workloads ingesting data
| and sending it elsewhere for storage/analysis whatever (and
| it does make sense for that target to be hosted - think
| datadog, betterstack, etc). So you'd deploy it as a sidecar
| in a pod, or as another container in your deployment or as
| a plain old executable in your server/vm or whatever.
|
| Also a UI is relatively pointless since you only mess with
| the config occasionally and otherwise just leave Vector
| running doing its thing. Want to see how it's doing? Have
| your Prometheus scrape Vector for its own metrics and set
| up alerts/analysis using Prometheus itself or Grafana.
| _pob wrote:
| No offense meant to the contributors or authors, but I don't know
| if I trust Datadog (the company) to steward what looks to be an
| OTEL competitor.
| lokar wrote:
| Same here. At work we were gearing up to use this, but after
| the acquisition changed focus to otelc.
| FridgeSeal wrote:
| IIRC this work predates acquisition by datadog, and it's
| continued well since then.
| FridgeSeal wrote:
| I've used this before, to great success. Nice and straightforward
| to configure, the vrl language is just powerful enough for its
| needs, the cli's handy "check" feature helps you catch a bunch of
| config issues. Performance wise it's never missed a beat and it's
| resource efficient, strongly recommend.
| esafak wrote:
| Better than otel?
| souvlakee wrote:
| Much more flexible.
| jauntywundrkind wrote:
| Otel support in Vector is an _often_ requested feature.
| Across multiple threads. There seems to be good noises &
| some occasional we'll get to it, but so far there's just otel
| log ingest support, which has been there for a while now.
| https://github.com/vectordotdev/vector/issues/17307
|
| I'm excited for these front-end telemetry routers to keep
| going. Really hoping Vector can co-evolve with and grow with
| the rest of the telemetry ecosystem. Otel itself has really
| started in on the next front with OpAMP, Open Agent
| Management Protocol, to allow online reconfiguration. I'd
| love to know more about Vector's online management... quick
| scan seems to say it's rewriting your JSON config & doing a
| SIGHUP. https://opentelemetry.io/docs/specs/opamp/
|
| Vectors configurability & fast-and-slim promise looks
| amazing. Everyone would be so much better off if it can grow
| to interop well with otel. Really hoping here.
| FridgeSeal wrote:
| I'm personally still waiting for Otel stuff to just...evolve
| a bit more? There's some sharp edges, and a bunch of "bits"
| in that ecosystem that aren't clear how we're supposed to
| hold them and things don't _quite_ work well enough yet.
|
| Don't get me wrong, I want to use OTEL, but it's a struggle.
| In the meantime, I've still got normal apps and libraries
| outputting normal logs and normal prom metrics, so I've got
| to stick with that.
| tedk-42 wrote:
| Same here.
|
| We had to push metrics we scrape via Prometheus into DataDog
| (coincidence that they acquired this) and do a custom transform
| to map to a set of custom metrics.
|
| Very straightforward in how it runs and the helm chart had all
| the right things in there
___________________________________________________________________
(page generated 2024-03-17 23:00 UTC)