[HN Gopher] Show HN: OpenObserve - Elasticsearch/Datadog alterna...
___________________________________________________________________
Show HN: OpenObserve - Elasticsearch/Datadog alternative
Hello folks, We are launching OpenObserve. An open source
Elasticsearch/Splunk/Datadog alternative written in rust and vue
that is super easy to get started with and has 140x lower storage
cost compared to elasticsearch. It offers logs, metrics, traces,
dashboards, alerts, functions (run aws lambda like functions during
ingestion and query to enrich, redact, transform, normalize and
whatever else you want to do. Think redacting email IDs from logs,
adding geolocation based on IP address, etc). You can do all of
this from the UI, no messing up with configuration files.
OpenObserve can use local disk for storage in single node mode or
s3/gcs/minio/azure blob or any s3 compatible store in HA mode. We
found that setting up observability often involved setting up 4
different tools (grafana for dashboarding, elasticsearch/loki/etc
for logs, jaeger for tracing, thanos, cortex etc for metrics) and
its not simple to do these things. Here is a blog on why we built
OpenObserve - https://openobserve.ai/blog/launching-openobserve.
We are in early days and would love to get feedback and
suggestions.
Author : prabhatsharma
Score : 120 points
Date : 2023-06-11 10:35 UTC (12 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| jakozaur wrote:
| I wonder why you want to write your own database called
| zincsearch. This gets very hard at scale and with all edge cases.
|
| Wouldn't be easier to use Cickhouse?
| prabhatsharma wrote:
| ZincSearch was an old product that we built. We don't use that
| in OpenObserve in any way. OpenObserve is built on top of
| awesome Apache arrow datafusion.
| makkesk8 wrote:
| Looked at this a couple of days ago and it looks impressive!
| However, it does saddens me to see the elasticsearch api
| compatibility being behind a license, I had hoped this was also
| going to be open after discovering the functionality existed in
| the first place.
|
| I guess having the cake and eating it too is actually not
| feasible in this case.
|
| With that said I will indeed have a go at this and test and see
| if I can replace elasticsearch in our stack.
|
| Good job guys!
| wdb wrote:
| Do you support OTLP endpoint for this? E.g. send metrics, traces,
| and logs in OpenTelemetry protocol? It's a bit unclear on the
| site
| prabhatsharma wrote:
| Yes, OTLP is supported. You can use standard Opentelemetry SDKs
| with no changes required.
| prabhatsharma wrote:
| Check the docs -
| https://openobserve.ai/docs/ingestion/traces/opentelemetry/
| wdb wrote:
| That's traces? I was wondering if I could use
| https://github.com/open-telemetry/opentelemetry-
| js/tree/main...
| anadem wrote:
| May be intentional but fyi the Getting Started video from
| your docs won't run for me, gets "Video unavailable / This
| video is private". I think it would be useful introductory
| info, although I'm just a hobbyist not a very likely
| customer.
| MuffinFlavored wrote:
| Might I suggest a Helm chart so that people can just apply
| it/install it to their home test k8s cluster, etc.
| rad_gruchalski wrote:
| And instructions on how to build a Docker image from sources.
| Looks nice, would love to try this out.
| prabhatsharma wrote:
| Here you go - https://github.com/openobserve/openobserve/blob
| /main/CONTRIB...
| prabhatsharma wrote:
| Helm chart is available for HA mode installation that allows
| you to use minio/s3/gcs etc -
| https://openobserve.ai/docs/ha_deployment/ .
|
| For single node installations, you can use base k8s manifests
| that is available using instructions at -
| https://openobserve.ai/docs/quickstart/#self-hosted-installa...
| .
|
| Mike created a helm chart for s ingle node installation too -
| https://github.com/mshade/openobserve-chart/
| hamandcheese wrote:
| I poked around the user guide[0] and discovered this:
|
| > Currently OpenObserve support steams of type logs, in future
| streams types metrics & traces will be supported.
|
| - are metrics and traces queryable yet? I admit, I feel a little
| misled, if only logs are supported for now that should be made
| more clear.
|
| - do (or will) metric and trace queries use a similar SQL syntax
| as log search?
|
| Finally... is there a demo? I would love to be able to try out
| the product without actually putting in the effort to set it up.
|
| [0]: https://openobserve.ai/docs/user-guide/streams/
| prabhatsharma wrote:
| Here is a quick demo - https://www.youtube.com/watch?v=fZ-
| ErfMdF-o . Covers basics, logs, function, dashboards. Does not
| cover traces and metrics.
| prabhatsharma wrote:
| I should fix it. Metrics and traces are supported today. Logs
| and traces are advanced.
|
| Metrics is still in its infancy though.
| prabhatsharma wrote:
| Logs, Metrics and traces are all queryable.
| benpacker wrote:
| How does this compare to Parseable?
|
| https://github.com/parseablehq/parseable
|
| First guess is that the underlying storage / query layer is
| pretty similar (Parquet + Datafusion), but OpenObserve has more
| built in use cases?
|
| As an aside, it's awesome that Datafusion's existence and
| maturity makes launching a product with scalable analytical reads
| 10x easier than before and cool to see so many projects
| integrating it
| prabhatsharma wrote:
| Parquet and data fusion are awesome. Parseable is a good
| project. Just has logs though. No HA either. They will get
| there though - I don't know anyone there, but they certainly
| need to if not there already.
|
| Yes, OpenObserve has a lot more use cases than parseable and
| our focus on ease of use has allowed us to build far more
| features and join them in an elegant way making it easier to
| use it, than any other platform out there.
|
| Give it a shot and let us know what you think.
|
| We wish parseable team our best wishes.
| alexk307 wrote:
| What metrics and tests do you have to support your petabyte scale
| claim?
| prabhatsharma wrote:
| We have run internal tests to ingest 1.2 petabayte of data in
| 24 hours period. We are making further improvements in
| performance and we will publish details that will allow you to
| replicate it yourself.
| gettodachoppa wrote:
| Hah, funny to see this on HN, I just tried this 3 days ago when I
| saw it being discussed on /r/selfhosted.
|
| As someone running a homelab, and hadn't set up logging yet, it
| was a great find. I didn't have to learn and combine 3+ log
| technologies, it's just a single all-in-one monitoring server
| with web UI, dashboards, log filtering/search, etc. RAM usage of
| the Docker container was under 100MB.
| prabhatsharma wrote:
| Glad you liked it. Yeah, resource usage is so low that some
| folks are running it in their homelab even on a raspberry-pi.
| gitowiec wrote:
| Once I wanted to use Data dog with our internal project.
| Unfortunately I couldn't use the trial, it ended when I was ill
| at home. So I asked to prolong the trial. Ohhh that was a big
| mistake. They came upon me with zoom meetings, with two persons
| from their side. Asking do many questions about everything made
| me angry. After our internal meeting we agreed that we will use
| another service. So I emailed Data dog that we will pass on their
| offer. And then emails started. I was like stalked about "let's
| do the meeting, tell us more for what do you need 3 weeks more of
| the trial .." and so on. That was worst corporate behaviour I
| experienced
| prabhatsharma wrote:
| Yeah, Reddit is filled with horror stories from Datadog. One
| thread here -
| https://www.reddit.com/r/devops/comments/zz4naq/datadog_i_do...
| Szpadel wrote:
| so if i understand docs correctly i can integrate it with
| existing prometheus setup to get eg. long term metrics with S3?
|
| and this would allow me also to get all logs to the same place?
|
| i see that graphana support in enterprise only feature, does it
| mean that long term metrics mirrored from prometheus are no
| longer available there?
|
| is there any way to get just graphana plugin for self hosted
| deployment?
| prabhatsharma wrote:
| Yes, you understand it correctly that you can integrate
| existing prometheus with OpenObserve for long term storage of
| metrics with s3.
|
| And yes you can get your logs there too.
|
| You don't need grafana as OpenObserve has good dashboarding for
| metrics and logs built in. Still early days for OpenObserve -
| some features are missing and has some rough edges.
|
| I am not sure what you mean by:
|
| > does it mean that long term metrics mirrored from prometheus
| are no longer available there?
|
| > is there any way to get just graphana plugin for self hosted
| deployment?
|
| ping us on slack channel, If you are able to make friends with
| the team you can get anything :-) . Fair warning - We have
| found OpenObserve logs UI to be much smoother and advanced than
| grafana UI.
| Szpadel wrote:
| > I am not sure what you mean by:
|
| > > does it mean that long term metrics mirrored from
| prometheus are no longer available there?
|
| Ok maybe i didn't describe it clearly, what i mean is that
| long term metrics will be only available over OO, and when we
| use grafana and our team is already familiar with that then
| it's issue that for querying long term metrics we would need
| to use different tool/UI.
|
| > We have found OpenObserve logs UI to be much smoother and
| advanced than grafana UI.
|
| Grafana isn't very good with logs, but as far as I understand
| you are able to generate metrics from logs in OO (like
| errors/exceptions per hour or time since last error etc) and
| adding them to existing dashboard in grafana is quite
| important.
|
| PS: overall OO looks very promising and I'm currently
| investigating self hosted product to aggregate logs and I'm
| for sure will be evaluating OO :)
| adamm255 wrote:
| How long until you have SSO/SCIM/IDP support? Being able to
| provision and manage users and sign in via Okta will be really
| useful.
| prabhatsharma wrote:
| An ask, we are getting from a lot of users. We are trying to
| figure out the most sophisticated and easiest way. Goal is to
| be able to do it in next 2-3 months. We have looked at Dex, but
| have not finalized on anything yet.
| samspenc wrote:
| This looks great, I had some questions about the company itself
| if that's OK, I'm just curious to understand the background:
|
| At the bottom of your GitHub project home page, you say the best
| way to join the project is to join a WeChat group (in Chinese
| text), but likely only a very small minority of us outside China
| use WeChat, so that may be a stumbling block if you are trying to
| encourage people outside Asia to contribute to the project.
|
| Per https://openobserve.ai/about , the address at the bottom says
| San Francisco, California, but in the same page it says
| "headquartered in Bangalore, India". So where are you based out
| of?
|
| Also curious what the relationship is between OpenObserve the
| open-source project and Zinc Labs, which is referenced in the
| website (but not in the GitHub project).
| prabhatsharma wrote:
| You can join Slack if you are outside of China, or wechat, if
| you are in China. We had some good early users whoBoth the
| options are available.
|
| > headquartered in Bangalore, India This is embarrassing, just
| fixed it. We are a delaware based company, headquartered in San
| Francisco. Pure copy paste error.
|
| Zinc Labs (Legal name) is the company behind the product
| OpenObserve.
| lopkeny12ko wrote:
| How do you typo "San Francisco" into "Bangalore"? This seems
| extremely shady.
|
| Copy and paste from where? Did you even build this product or
| just hire cheap labor out of India to build it?
| aliencat wrote:
| It seems that the 140x lower storage cost comes from: 1. S3 (OO)
| vs EBS (ES): about 5x 2. No indexing: About 10x ? 3. No data
| duplication (due to using S3 I assume) in HA deployment: 3x
|
| Is my math right? Or do you use something different for
| compression?
|
| 2 Orders of magnitude of storage saving is pretty impressive.
| prabhatsharma wrote:
| You are right.
| [deleted]
| codegeek wrote:
| How are you different from something like Signoz ?
| prabhatsharma wrote:
| I think Signoz is a good product. The team really understands
| that it you need all 3 - logs, metrics and traces - to
| effectively observe and troubleshoot applications and has done
| a good implementation.
|
| Certain difference on how OpenObserve is different than Signoz:
|
| 1. Built in rust for high performance and memory safety. 2. Has
| its own data storage engine based on open specification of
| Apache Parquet. This allows the data to be used by other
| applications outside of OpenObserve if desired. 3. Use
| stateless nodes in architecture (You are going to love it from
| maintenance point of view). 4. Use of object storage for
| extreme scalability and low cost. 5. Requires far lower
| resources than Signoz. Can practically be run using a single
| binary/container. There are users ingesting terabytes of data
| on a single node. There are people running OpenObserve on
| raspberry-pi. 6. A lot easier to setup and maintain. 7. Ingest
| and query functions are awesome. Give them a shot.
| michaelmior wrote:
| > It is a drop-in replacement for Elasticsearch if you are just
| ingesting data using APIs and searching using kibana (Kibana is
| not supported nor required with OpenObserve. OpenObserve provides
| its own UI which does not require separate installation unlike
| kibana).
|
| This seems contradictory. If OpenObserve doesn't support Kibana,
| how can it be a drop-in replacement for Elasticsearch queried
| using Kibana? Even if the OpenObserve UI has similar features, I
| can't just replace Elasticsearch with OpenObserve and keep the
| rest of my workflow the same, which I would consider a
| precondition to call something a drop-in replacement.
| vlovich123 wrote:
| Said an alternate way, if Kibana drop in compat could be a
| thing, it would be easier for infra teams to do it as a drop in
| into existing stacks at bigger companies which makes it easier
| to steal market share of existing customers on the ES stack.
| There's probably a non trivial amount of existing dashboards
| built and UX familiarity that would be a friction point without
| seamless compat.
|
| Is Kibana support totally out of the question?
| prabhatsharma wrote:
| >Is Kibana support totally out of the question?
|
| Actually no.
|
| We already have a good amount of pieces in place to support
| kibana. 2-3 weeks of dev work and it will be ready. Just not
| the top priority right now though.
| vlovich123 wrote:
| Wow that's really cool. Great job!
| jerrygenser wrote:
| Intro video is private. I could not access it on YouTube.
|
| https://openobserve.ai/docs/getting-started/
| prabhatsharma wrote:
| Thanks for pointing out. Fixed it.
| sgarland wrote:
| > Use of SIMD instruction sets (AVX-512 and neon)
|
| Have you done any comparison testing with various generations of
| hardware? AFAIK the latest generations don't down clock their
| scalar turbo that much (or at all), but Skylake definitely did.
| prabhatsharma wrote:
| We have done minimal performance testing for now. A lot of work
| is remaining in this area.
| hamandcheese wrote:
| What differentiates you from the grafana stack? Grafana labs now
| have solutions for every pillar of observability (metrics, logs,
| traces), and that whole stack also touts cheap ingest (due to
| minimal upfront indexing and cheap blob storage) as one of its
| biggest selling points.
| prabhatsharma wrote:
| Grafana stack is cool. It solved the problem around
| observability for the first time in a neat and light way. Think
| about it however, for someone setting up their observability
| stack they will need to setup 4 components: Grafana
| (dashboarding), Loki (logs), Mimir (Metrics) and Tempo
| (Traces). This is a lot of work and a lot of moving parts. It
| is also a lot of work to maintain and upgrade. Loki, Mimir and
| Tempo have their own querying languages and their own quirks.
| Good amount of stuff to learn. It is really hard for a small
| team to learn and maintain all of this. Junior engineers really
| struggle with this and seniors resent the time they have to
| spend on this. Loki also has issues around high cardinality,
| which is a problem for many teams.
|
| We need something new and better.
|
| We made sure to build a solution that is easy to setup and
| maintain. One single binary/container for single node setup and
| one stack to setup/upgrade/maintain for HA setup.
|
| We also made sure that it is easy to use and learn. We allow
| standard SQL to be used for querying logs, metrics and traces -
| nothing extra to learn here. Metrics querying is supported
| using PromQL too. Drag and drop for creating panels and
| dashboards is there too :-).
|
| Dashboarding is supported within the same stack. No need to
| setup Grafana or something else separately.
|
| OpenObserve also offer functions that grafana stack does not
| offer. Give it a shot, you are going to love it.
|
| Data is stored in open parquet format, allowing its usage even
| outside of OpenObserve if so desired.
|
| Hope this helps.
| vlovich123 wrote:
| One of the most annoying things of grafana is that panels can
| only be edited via the UI and the "changes" summary when you
| commit is near unusable with a bunch of auto generated
| garbage.
|
| Having a simple language/YAML to define those panels and an
| easy way to preview them (eg paste in your file to preview
| the page, paste in a panel source to try it) in addition to
| editing via GUI and being able to copy changes back to
| revision control would be great
| prabhatsharma wrote:
| This is a good feedback. That would be true DaC (Dashboards
| as code). We already have plans to implement versioning of
| dashboards via git. Will see how to get DaC in place.
| smw wrote:
| https://grafana.github.io/grafonnet-lib/
| PlutoIsAPlanet wrote:
| Grafana log handling also isn't the best, especially when
| compared to New Relic etc.
| [deleted]
___________________________________________________________________
(page generated 2023-06-11 23:00 UTC)