[HN Gopher] Launch HN: OpenMeter (YC W23) - Real-Time, Open Sour...
___________________________________________________________________
Launch HN: OpenMeter (YC W23) - Real-Time, Open Source Usage
Metering
Hi HN! I'm Peter, the co-founder of OpenMeter
(http://openmeter.io). We are building an open-source project to
help engineers to meter and attribute AI and compute usage for
billing and analytics use cases. Our GitHub is at
https://github.com/openmeterio/openmeter, and there's a demo video
here: https://www.loom.com/share/bc1cfa1b7ed94e65bd3a82f9f0334d04.
Why? Companies are increasingly adopting usage-based pricing
models, requiring accurate metering. In addition, many SaaS
products are expected to offer AI capabilities. To effectively
cover costs and stay profitable, these companies must meter AI
usage and attribute it to their customers. When I worked at
Stripe, my job was to price and attribute database usage to product
teams. You can think about it like internal usage-based pricing to
keep teams accountable and the business in the margins. This was
when I realized that it's challenging to extract usage data from
various cloud infrastructure components (execution time, bytes
stored, query complexity, backup size, etc.), meter it accurately,
and handle failure scenarios like backfills and meter resets. I was
frustrated that no standard exists to meter cloud infrastructure,
and we had to do this on our own. Usage metering requires
accurately processing large volumes of events in real-time to power
billing use cases and modern data-intensive applications. Imagine
you want to meter and bill workload execution on a per-second
granularity or meter the number of API calls you make to a third
party and act instantly on events like a user hitting a billing
threshold. The real-time aspect requires instant aggregations and
queries; scalability means to able to ingest and process millions
of usage events per second; it must be accurate--billing requires
precise metering; and it must be fault tolerant, with built-in
idempotency, event backfills, and meter resets. This is
challenging to build out, and the obvious approaches don't work
well: writing to a database for each usage event is expensive;
monitoring systems are cheaper but inaccurate and lack idempotency
(distributed systems use at-least-once delivery); batch processing
in the data warehouse has unacceptable latency. Companies also
need to extract usage data from cloud infrastructure (Kubernetes,
AWS, etc.), vendors (OpenAI, Twilio, etc.), and hardware components
to attribute metered usage to their customers. Collecting usage in
many cases requires writing custom code like measuring execution
duration, listening to lifecycle events, scraping APIs
periodically, parsing log streams, and attributing usage of shared
and multi-tenant resources. OpenMeter leverages stream processing
to be able to update meters in real-time while processing large
volumes of events simultaneously. The core is written in Go and
uses the CloudEvents format to describe usage, Kafka to ingest
events, and ksqlDB to dedupe and aggregate meters. We are also
working on a Postgres sink for long-term storage. Check out our
GitHub to learn more: https://github.com/openmeterio/openmeter
Other companies in the usage-based billing space are focused on
payments and basically want to be Stripe replacements. With
OpenMeter, we're focusing instead on the engineering challenge of
collecting usage data from cloud infrastructure and balancing
tradeoffs between cost, scale, accuracy, and staleness. We're not
trying to be a payment platform--rather, we want to empower
engineers to provide fresh and accurate usage data to Product,
Sales, and Finance, helping them with billing, analytics, and
revenue use cases. We're building OpenMeter as an open-source
project (Apache 2.0), with the goal of making it the standard to
collect and share usage across many solutions and providers. In the
future, we'll offer a hosted / cloud version of OpenMeter with high
availability guarantees and easy integrations to payment, CRM, and
analytics solutions. What usage metering issues or experiences do
you have? We would love to hear your feedback on OpenMeter and to
learn from which sources you need to extract usage and how the
metered data is leveraged. Looking forward to your comments!
Author : hekike
Score : 158 points
Date : 2023-06-22 12:19 UTC (1 days ago)
| debarshri wrote:
| Slightly distracting question and because I am very curious. Why
| did you decide to opensource the project? What is the benefit
| that you get in your GTM by opensourcing your project. Don't you
| think it is a bit more of a headache to create and manage a
| community around the project. Document, manage external user
| changes etc.
| hekike wrote:
| Great question. It has two components for OpenMeter: 1. My
| personal frustration when I had to do usage-metering at Stripe
| and wanting to have a standard for it, like how we have it for
| metrics. I also find it silly that every company trying to
| figure this out on their own. 2. We want to build usage
| collectors for common cloud components and third-party APIs and
| I think it's going to be easier to do it together as a
| community instead of working in silos.
|
| But yes I agree, it's more challenging to build in open but I
| think it benefits the customer, the quality has to be there.
| [deleted]
| ygouzerh wrote:
| Notes: Something that I always says in my engineering team: I
| will encourage you to fully leverage SemVer for the versioning of
| the product and use the major for the major public releases /
| breaking changes.
|
| In this way, it will enable you to be able to fully use the minor
| number and the patch number when you will need them.
| mirchiseth wrote:
| At the risk of sounding old - this is called mediation part of
| the usage based billing in the Telecom/Utility billing, which
| happen to be the OG of usage based billing with millions of
| record. I have worked with Digital Route for this in the past.
| Definitely good to see newer architectures and thinking applied
| to this problem for newer use cases. Godspeed to OpenMeter.
| villgax wrote:
| Awesome work, looking try this out!
| mercurialsolo wrote:
| I would love to see a wrapper around all API calls which is
| metered by default. Showing usage and meters which you can set at
| a per-integration level, per-user, per-key, per-function level.
|
| Congrats on launching this, there's lots of ways to deal with
| this, but would love a metering standard much like we have one
| around tracing.
| hekike wrote:
| I like the idea of meter all primary business executions by
| default and pick what you want to use later. This is useful if
| you want to do cost analysis, iterate on pricing, etc.
|
| Would you track API calls by default like a usage-based third-
| party (OpenAI API, Twilio, etc.)? Or your own web server
| (Lambda, Next.js etc)? Or both.
| mercurialsolo wrote:
| We do both, services as well as a mix of serverless and on-
| demand servers we spin up. I would also like this to have the
| SaaS usage sprawl bundled into this, e.g. emails, notifs, and
| SaaS tools with expanding usage tiers,
| hekike wrote:
| Cool, thanks for sharing it. Would you use SaaS usage for
| cost analysis or to charge your own customers for shared
| resources?
| runlevel1 wrote:
| Metering is such a PIA to implement properly, and the cost of
| getting it wrong is super steep: Lost revenue or, worse,
| overbilling. Love the idea of this project, and I'm liking what I
| see so far!
| hekike wrote:
| Thank you. Do you have learnings or use-case you can share and
| you want to see OpenMeter to cover?
| that_guy_iain wrote:
| Your discord link is invalid. "undefined" I assume is from a JS
| error.
| hekike wrote:
| Thanks for letting us know! We fixed it, also here is the link
| to join: https://discord.com/invite/nYH3ZQ3Xzq See you on
| Discord ;)
| methodical wrote:
| This isn't a ding against the product or idea and is likely
| something you're looking to improve in the long term, but the
| graphics on the marketing site just feel like visual sugar and
| don't convey any information. This may be just me but I
| frequently just go through a marketing site's graphics as a way
| to grasp high level details of the company and product, but the
| graphics on your marketing site feel like random geometrical
| placeholder images with different layouts, as opposed to anything
| that actually complements the information written in text next to
| them. Again, this is a minor gripe, and something that probably
| was already going to be addressed long term. The product sounds
| interesting and I look forward to taking a deeper dive into the
| open source side of it later!
| hekike wrote:
| Thank you for the honest feedback on the landing, appreciate
| it. Yes, we should make it more informative. Maybe docs is more
| useful in the meantime: https://openmeter.io/docs
| lharries wrote:
| Congrats on the launch! If I understand it right you're
| interested in collecting this usage data both to bill external
| clients but also to much more granularly understand the infra
| costs of a particular feature/product/team? Very cool. Do you
| have any good examples of the internal usage one? How much do big
| companies actually care about this?
| hekike wrote:
| Internal usage is basically Cloud Unit Economics:
| https://www.finops.org/wg/introduction-cloud-unit-economics You
| treat infra teams as providers and price-in their services. For
| example you may price Compute team by "selling" vCPUs, DB team
| by selling GB stored per month and product team per business
| transactions. Each team builds on top of each other and buys a
| service from their downstream provider. Like product team buys
| DB storage, DB team buys vCPU to run DB etc. If you price in
| these internal resources (rate cards) and track usage you can
| attribute cost to each team based on their consumption. These
| internal bills together equals to the total company wide AWS
| bill. This not just creates accountability for teams but you
| can also do margin analysis and tell if certain team's product
| is profitable. Engineers can also plan future services and
| architecture by estimating cost like, new service will store
| 1TB/day which will cost X and then business/PMs can make
| decision based on it. It's a also a great way to optimize cloud
| cost beyond obvious isolated things like upgrading instance
| types as it tells you where optimization needs to happen, which
| provider need to lower their cost to increase margins.
| hekike wrote:
| You can find the architecture diagram here:
| https://openmeter.io/docs Decisions:
| https://github.com/openmeterio/openmeter/tree/main/docs/deci...
| [deleted]
| mrkurt wrote:
| We're neck deep in figuring out usage based billing for Fly.io.
| What we're doing is pretty extreme, and may not be representative
| of many other people, but it feels like no one's really solved
| this in a way that makes sense for us.
|
| We run millions of tiny VMs. Each gets billed on a number of
| dimensions: egress, runtime (per cpu / memory combo), storage /
| io. We also have other metered services: ssl certificates, IP
| addresses, etc.
|
| The thing is, we _already_ have metrics for everything we want to
| bill. They're in a time series DB (VictoriaMetrics, in this
| case). Sending a shit ton of events to yet-another-system is
| complicated and brittle.
|
| Your k8s pods example is a good source of my hives. Anything that
| runs on a timer to fire off events is going to get bogged down
| and lose data at our scale. And we're not very big!
|
| This is a somewhat solved problem for metrics stacks. It's
| relatively straightforward to scale metrics scrapes. And once we
| have that infra, it's pretty easy to just start pushing "events"
| to it when events become useful. We don't end up needing Kafka
| and its complexity.
|
| My dream for billing is: a query layer on top of a time series
| database that takes data I already have and turns it into
| invoices.
|
| One thing about your post that struck me - the last mile to
| billing and reporting is the thing we're most interesting in
| buying. It's less specialized. There also aren't any products out
| there that have really figured this out, I don't think (we've
| evaluated pretty much all of them).
|
| Usage tracking and reporting is a thing we're ok building,
| because it's core to our product.
| johndbritton wrote:
| A couple things to check out if you haven't already seen them:
|
| * Lago - https://www.getlago.com
|
| * Octane - https://www.getoctane.io
|
| * Metronome - https://metronome.com
| hekike wrote:
| Those are all solid products for billing. Tough none of them
| are both open-source and have the scalability of streaming
| aggregation. Even Lago is AGPL which can be challenging for
| some.
| strawberrysauce wrote:
| We at zenskar.com are building a billing platform that can plug
| into your metering store (in addition to the traditional api
| way) and generate invoices for your customers. We have a query
| layer that can be used to define aggregates that in turn can be
| used to create invoices. I will reach out to you on email to
| see if we can solve your problems.
| dangoodmanUT wrote:
| Yeah pulsing should always be a last resort, never works at
| scale
| kshitijgrover wrote:
| At Orb (https://www.withorb.com/) we've worked with a bunch of
| infra customers that are in a similar situation to what you're
| describing. Generally, usage tracking and usage reporting is
| not the hard part. We think that giving you a full SQL layer on
| top of raw events is very powerful, so we provide direct SQL
| access to your events. Naturally, we use a real-time time
| series database under the hood to power that.
|
| We're working hard to make it easy to send data to us (e.g.
| just drop it in S3). Granted, this isn't ideal per your note of
| already throwing it into VictoriaMetrics, but we're finding
| that often times companies don't have the optimized in-house
| infra to run the sort of usage metric queries to go from raw
| events -> invoice, especially when you get to >100k
| events/second scale. It often makes sense for us to control the
| storage (and sharding) format -- and we can bring those costs
| down over time.
|
| Given the usage events and the SQL queries over them, we then
| focus a bunch on making the core billing interactions work
| well, do invoice delivery for you (example: https://invoices.wi
| thorb.com/view?token=IlJwOVRSdWhEQ3NiRUVH...), and make it easy
| for you to do financial reporting & rev-rec (honestly, this is
| its own rabbithole -- but we find it's extremely outside the
| core competency of eng/product teams)
| hekike wrote:
| I think it's substantial risk for the core business. To be
| able to glue usage to invoice, Orb has to become the source
| of truth and billing engine top of Stripe, while Stripe is
| still the payment gateway. This creates a fragmented world
| where the customer has to use both systems simultaneously;
| this is challenging especially that Stripe ships features
| fast and frequently, leaving the customer exposed and vendor
| locked.
| tonyhb wrote:
| Hi Kurt! We do the same metrics type thing at Inngest, and we
| share the exact same thoughts as you. We 'solved' it by, well,
| using Inngest to charge for Inngest.
|
| We track everything as metrics, then hook up functions to read
| from the TSDB each billing period. How it works:
|
| - One Inngest function runs each billing period (scheduled),
| and "fans out" by creating an event for each user we want to
| charge.
|
| - A billing function reacts to these individual user events,
| queries metrics, queries our plans, and then generates a stripe
| invoice
|
| - This is then charged by stripe.
|
| - We have other functions that respond to other billing events
| and do dunning like flows.
|
| We only have to focus on the small amount of billing code
| required. We don't have to handle scheduling, state, retries,
| or the event flows -- that's built into Inngest.
|
| I'd really like to make this more robust. We had to put in the
| work on emails, billing, and dunning, even if it is simple
| function code. It would be nice to have someone "productize"
| this. I feel your pain.
|
| In general, it definitely makes sense to read from our own
| metrics stores to charge for our products. Very much agree with
| everything you've said. There's not the most joy in billing.
| hekike wrote:
| Very cool, thanks for sharing. I like this reporting
| approach, once you have the usage data aggregated somewhere a
| state machine is a great way to report usage to Stripe. Do
| you trigger the Inngest billing fan out end of the month CRON
| style or when Stripe changes billing period and triggers a
| webhook to Inngest?
| hekike wrote:
| That's a great insight, thanks for sharing. Would it be the
| ideal solution code / config driven, something like: SELECT
| user, SUM(runtime) FROM usage => invoice_template(Z) Wdyt?
|
| Btw how do you handle idempotency with time series? I think
| replays and least-once delivery in distributed systems can be
| challenging around usage collection if you don't have a key to
| deduplicate by.
| mrkurt wrote:
| I may not be interpreting your question right, but most of
| our metrics are counters. It doesn't matter how often they
| get scraped, or if they get scraped twice, because we only
| care about the increase. Almost everything we track is
| aggregated with `increase()`: https://prometheus.io/docs/prom
| etheus/latest/querying/functi...
|
| I'm fond of SQL, but I'd be ok with some other way to express
| "aggregate my usage into an invoice".
| andrastoth wrote:
| Have you encountered any challenges related to metric
| aggregation? To the best of my understanding, such issues
| often stem from inherent inaccuracies due to sampling
| (https://www.youtube.com/watch?v=67Ulrq6DxwA&t=175s). I'm
| curious if using a smaller window for metric aggregation
| and routing these events to a stream aggregator could
| provide a possible solution.
| mrkurt wrote:
| Not really, but we might've just implicitly designed
| around it.
|
| Continuously aggregating smaller windows and sending them
| off is roughly how we send usage to Stripe right now
| (which, let's be clear, is not working for us AT ALL).
| We've resigned ourselves to having to do the same thing
| if we want to use a billing SaaS, but I don't love it.
| jmacd wrote:
| We've built https://GitHub.com/tierrun/tier with a local
| sidecar that can talk directly to stripe to help simplify
| situations like this.
|
| A pure SaaS model for billing doesn't do much more for
| you at then end of the day other than to apply billing
| ratings and generate an invoice (which Stripe can and is
| doing for you already).
|
| I'd love to learn more about what you'd like to do with
| Stripe and how you are handling entitlements, plan
| versioning, etc
| ylere wrote:
| Fixing the missing links in the billing chain is what we have
| been working on at Wingback (https://wingback.com). Agree that
| the actual metering part is not that hard to get (and having a
| standard here would be great!), but using it to correctly
| enforce entitlements in your product and accurately bill your
| customers who might be on all kind of different plans, models
| prices and discounts over a large number of metered (and
| umetered) features is.
| michaelt wrote:
| _> Your k8s pods example is a good source of my hives. Anything
| that runs on a timer to fire off events is going to get bogged
| down and lose data_
|
| I'd have said if you want to gracefully deal with a system that
| loses events, a timer-based system can ensure the errors from
| lost events will always be in your customer's favour.
|
| The last thing you'd want is to start the taximeter with a
| start-pod event, then lose the stop-pod event and end up
| billing the customer for nothing, forever.
| [deleted]
| mlhpdx wrote:
| I'm building a service billed for via usage. As others have
| pointed out, I have to solve metering for a variety of reasons
| (beyond billing) and won't take an external dependency for it.
| It's a nonstarter, metering is just too central.
|
| Also, any invoice system that's based on query-time aggregation
| is probably too expensive and definitely a risk. Fees are
| accumulated in real time, incrementally - not when I generate an
| invoice.
|
| But probably the biggest miss I encounter is around auditing.
| Customers dispute bills, and always will to some degree. The
| process of resolving those disputes needs to be automatic, clear
| and understandable, and based on something that the customer can
| cross check (ideally).
| eob wrote:
| +1 to this from the Steamship perspective.
|
| We're a hosting platform for AI Agents and in practice that means
| keeping track of a stream of usage events across agents,
| microservices (equivalent of Fly's VMs), and third-party services
| (OpenAI, Eleven Labs, D-ID, Pinecone, Replicated, etc) that roll
| up into an itemized consumption bill.
|
| It might be this is just a super-niche billing situation, but we
| also haven't yet found an open-source solution that fits. Fully
| agree that a query layer atop a time-series database is the style
| of product that we're craving.
|
| Metronome comes very close, but since it's not open source we're
| hesitant to make the jump before it's a perfect fit.
|
| Congrats on the launch! It looks very cool!
| nylonstrung wrote:
| What does this do that cannot be accomplished with the other YC
| open source billing tools- Lago and lotus
| hekike wrote:
| It depends on your use-case. On a higher-level OpenMeter
| focuses on the data challenge to collecting usage data from
| infrastructure components on scale, and providing aggregated
| meters real-time. The other platforms you mentioned focuses
| more on billing with some metering component. For example Lago
| looks like writes all events to a relational database first
| than aggregates usage at query time when they do billing.
| Metering with a database has some limitation when it comes to
| scale or when you need to monitor meteres real-time and trigger
| actions based on them. I can see for example some combining
| OpenMeter with other billing platforms to have a scalable
| usage-based billing.
| ygouzerh wrote:
| Your UI is very nice, are you using a special CSS framework ?
| andrastoth wrote:
| It's built with Tailwind CSS. We'll open-source the website as
| well in 1-2 weeks.
| jeffchao wrote:
| For these things I look for flexibility, i.e., from the user
| perspective, control. That's the nice thing about OSS vs closed
| source here.
|
| I don't necessarily want to send data to another SaaS vendor's
| data lake/warehouse/tsdb/etc. Egress costs, privacy, and storage
| costs. Also if I already have a data platform, I don't want 2.
___________________________________________________________________
(page generated 2023-06-23 23:02 UTC)