[HN Gopher] Scaling to Count Billions
___________________________________________________________________
Scaling to Count Billions
Author : kiyanwang
Score : 40 points
Date : 2024-04-29 06:34 UTC (1 days ago)
(HTM) web link (www.canva.dev)
(TXT) w3m dump (www.canva.dev)
| tibbar wrote:
| The most interesting thing here is probably that they sometimes
| change their mind as to how to aggregate billing events, and used
| to have to manually fix the database if they changed their mind.
| The system they've arrived at seems to mostly handle
| recalculations, but they're making a pretty big assumption that
| you can always aggregate data on daily intervals. The day will
| come when they need to aggregate data over periods that are _not_
| a single day, and then they 'll have to figure out what happens
| if they've already billed for some prior usage events when they
| go to re-run their automated pipeline...
| dax77 wrote:
| Yeah, usage rating and billing systems get wildly complex.
| Especially when you start considering edge cases- beyond leap
| years and handling time zones, things like a billable event
| spanning days (run a few minutes just before midnight), etc.
| kwillets wrote:
| I'm guessing the next stage will be to keep the raw data and
| aggregate dynamically. We went through a similar progression with
| ad audiences -- maintaining pre-aggregated data sketches and
| summaries was such a PITA that we likely saved money by moving to
| a larger raw-facts database.
|
| > OLAP databases are not good at serving large volumes of
| requests with low latency within milliseconds.
|
| Just use Vertica lol.
| lima wrote:
| Use Clickhouse - it's specifically built for this use case (low
| latency OLAP).
| kwillets wrote:
| Yes, I almost added that -- there are several alternatives,
| but it takes some testing and tuning to be certain.
| nyrikki wrote:
| The same mistake we as an industry are doomed to repeat.
|
| RDMS are like Leatherman, while they can do lots of things, the
| only thing they do well is being a set of needle nose pliers.
|
| Relational databases are great for lots of people doing lots of
| transactions on small sets of data.
|
| Everything else is a compromise.
|
| It is one of the areas that is completely rational to use when
| small, but is worth putting in boundaries from the beginning.
|
| I have never regretted a persistence layer abstraction, but have
| been held hostage by not.
| willsmith72 wrote:
| To verify that you first have define "small". I've seen a large
| postgres db handling what I could call a large amount of data,
| and in a separate situation seen it scaled horizontally through
| sharding to an even larger scale.
|
| Premature abstractions cause pain
| j-cheong wrote:
| Kind of unrelated but I'm curious if there are any really robust
| usage-based billing solutions out there. Curious how they're
| architected to solve usage-based billing across their
| customers/various use cases.
|
| I'm always concerned about automating the billing process and
| risking accuracy/trust.
| dax77 wrote:
| Zuora is a well-known Enterprise-scale commercial option that
| displaced many others as "SaaS" took off several years ago (and
| its associated accounting standards).
|
| Depending on complexity, Netsuite can address some moderate
| scale use cases. Stripe, Chargebee, etc address more of the
| SMB-scale needs.
___________________________________________________________________
(page generated 2024-04-30 23:01 UTC)