[HN Gopher] Moving from relational data to events
___________________________________________________________________
Moving from relational data to events
Author : alexzeitler
Score : 199 points
Date : 2023-12-16 14:27 UTC (8 hours ago)
(HTM) web link (event-driven.io)
(TXT) w3m dump (event-driven.io)
| alecco wrote:
| This article is not good. Event Sourcing and the Relational Model
| are orthogonal.
|
| SQL:2011 added a lot of temporal features.
|
| Datomic is based on Datalog which even though is not relational
| it's kind of the same, and has temporal support. [2]
|
| [1] https://en.wikipedia.org/wiki/SQL:2011
|
| [2] https://vvvvalvalval.github.io/posts/2018-11-12-datomic-
| even...
|
| (BTW 24 points at the top of HN and no comments? Hmm)
| ithkuil wrote:
| Datalog is relational in the original sense of relational
| algebra.
| alecco wrote:
| Sure. But the point of the article is for Document or Object-
| Oriented. And the specific points would apply to Datalog,
| too.
|
| My point was you can do the equivalent of Event Sourcing with
| SQL and even better with Datomic.
|
| > In Event Sourcing Church, we're not doing that anymore.
| We're not losing business data; we keep them. We keep them as
| events.
| cmrdporcupine wrote:
| Datalog is definitely relational. More so than SQL.
|
| In terms of temporal data handling & relational & Datalog, it's
| worth looking at Differential Datalog:
| https://github.com/vmware/differential-datalog
| refset wrote:
| > Datomic [...] has temporal support
|
| Note it only supports "transaction time" (or "system time" per
| SQL:2011) but not "valid time" (~"event time") which is needed
| for a bitemporal data model. [1]
|
| [1] https://vvvvalvalval.github.io/posts/2017-07-08-Datomic-
| this...
| hot_gril wrote:
| You don't need specific temporal support for an event-oriented
| DB, nor would I want it. I usually design a vanilla relational
| schema around events regardless of which DBMS I'm using. E.g.
| instead of an "order" table with multiple states updated in-
| place, I'd have "order_placed" and "order_filled" where each
| row is an event, insert-only.
| ekorz wrote:
| I've never before seen a post hit #1 here after 30 minutes with
| zero comments. Is that normal?
| alexzeitler wrote:
| Funny, I had the same thought
| rockwotj wrote:
| Someone got around the upvote ring detection? I know there are
| lots of folks that claim to know how to get around it, but
| agreed it seems sketchy
| jddj wrote:
| To give the benefit of the doubt, maybe it was an intriguing
| title?
|
| I didn't personally think that the style of the writing or the
| content itself was particularly good, but event sourcing has an
| almost narcotic appeal.
|
| ES (nosql did too) has this wild ability to slip past
| scepticism. It _feels_ right, I think because it milks some
| dopamine by feeling both simpler in a rewarding abstract sense
| and sufficiently (for Devs) complex in a day to day operations
| sense.
| Tao3300 wrote:
| I've got nothing smart to say about databases on a Saturday
| alecco wrote:
| And at 1 hour it's still at the top in spite of all the
| negative comments. I'm assuming it got flagged by enough of us
| and yet it prevails. Odd.
| candiddevmike wrote:
| Where's the hitchhikers guide to moving back to relational data
| after our whiz bang dev made us event driven and left for a new
| opportunity?
| tarruda wrote:
| This reminds me of https://youtu.be/b2F-DItXtZs
| jtwebman wrote:
| He tried at the beginning to say only when the DB gets too big
| should you even think about this but there are things you can
| do there as well.
| Nextgrid wrote:
| The way I see it, either your business domain requires
| querying over a large amount of data, or it doesn't.
|
| If an application allows someone to be able to enter let's
| say an order number from anywhere in the world from the last
| 10 years and be able to find the order, there is no magic -
| some server out there is going to have to scan a huge amount
| of data to find a match.
|
| Tricks such as indexes, partitioned tables, etc can be
| employed, but those tricks have nothing to do with event-
| sourcing and are independent of it.
| mrkeen wrote:
| > Tricks such as indexes, partitioned tables, etc can be
| employed, but those tricks have nothing to do with event-
| sourcing and are independent of it.
|
| You might want to use different tricks in different
| situations. Different situations means different services,
| and different tricks means different storage/query
| technologies.
|
| So how do you get your data into three systems - and more
| crucially - keep them in sync? Webhooks? Triggers? Some
| bidirectional sync magic app that claims to beat CAP?
|
| Just use event-sourcing (append-only, disallow
| modification) and the multiple systems will stay in sync as
| long as they know how to process one more message.
| Nextgrid wrote:
| Agreed with your points but this article seems to present
| event-sourcing as a replacement for your database(s) and
| even makes claims about saving storage space, thus at
| least hinting at _not_ using databases anymore.
| mrkeen wrote:
| It's a replacement for your _source-of-truth_ , not your
| database(s). Although you're right about the article not
| explicitly mentioning slurping the events back into a DB.
| I suspect the reason is that there are plenty of articles
| which explain how to event-source from greenfield, but
| this is the first one I've seen which focuses on existing
| brownfield relational data - see the title.
|
| > makes claims about saving storage space
|
| I don't think that was the right reading about saving
| storage space:
|
| > We've been trying to optimise the storage size; we've
| made some sins of overriding and losing our precious
| business data
|
| I think it's his strawman RDBMS developer who optimised
| for saving storage space, and lost business data as a
| result. The suggested approach is:
|
| > We can optimise for information quality instead of its
| size.
| seanhunter wrote:
| There is no "too big" in databases and in particular size of
| data is not the criterion for deciding on using something
| like event sourcing. It's a really niche paradigm that is
| only ever going to be useful in quite unusual circumstances.
| Most of the time people don't need it, and most of the time
| when people do it, they find their immutable event source is
| very inconvenient for lots of the normal things you want to
| do with your data, so they end up doing something like
| CQRS[1] (ie having a database as well). This is one of those
| Martin Fowler[2] type things that looks good on a whiteboard
| but most people would be better off avoiding most of the
| time.
|
| [1] https://en.wikipedia.org/wiki/Command_Query_Responsibilit
| y_S...
|
| [2] This Martin Fowler, https://en.wikipedia.org/wiki/Martin_
| Fowler_(software_engine... not this Martin Fowler
| https://metro.co.uk/2023/11/06/eastenders-star-reveals-
| why-m...
| blowski wrote:
| So long, and thanks for all the eventual consistency issues
| corethree wrote:
| Actually event driven stuff is highly consistent. Event
| insertion is fast as hell and all of it is timestamped and
| tagged with a unique Id so there's no consistency issues.
|
| The problem is availability. You have to aggregate those
| events to get anything meaningful about it and aggregation
| can be really really slow.
| blowski wrote:
| Exactly. Because the aggregation is so slow, you cache the
| aggregates, but then the caching layer becomes slow to
| invalidate, and you get eventual consistency problems.
| corethree wrote:
| I wouldn't call that a consistency issue. That's just
| lag. The aggregation is valid for a specific time point.
| Caches aren't the source of truth. The source of truth
| remains consistent here.
| jabradoodle wrote:
| It's hardly a cache if it's the only way to query the
| data, it's a materialized view
| corethree wrote:
| Then it's an outdated materialized view. If you don't
| refresh your view from a browser is it a consistency
| issue? No. Not in in the way the term is usually used.
| blowski wrote:
| The event store and the cache are both part of the same
| system, and it's this whole system that is "eventually
| consistent".
|
| Say I create widgets on screen 1 and they are persisted
| with event sourcing into Postgres. I see the list of
| created widgets on screen 2, loaded from a materialised
| view in Postgres (or Elasticsearch). The "lag" between it
| being created on screen 1 and appearing on screen 2 is
| the "eventual consistency" issue I'm referring to here,
| whereas I think you're referring to the consistency only
| of the persistence on screen 1.
|
| I'm sure we both agree there's no getting away from CAP
| theorem. Event sourcing accepts less consistency, and
| every part of the system needs to deal with that.
| corethree wrote:
| >The event store and the cache are both part of the same
| system, and it's this whole system that is "eventually
| consistent".
|
| Then every single system of the face of the earth at a
| high enough level has a consistency issue. Just go to the
| level of the full client and backend system using web
| browsers. You don't refresh the browser you of course
| will eventually have a "consistency" issue.
|
| Usually when they refer to this stuff it's referring to
| the source of truth: The database. When you shard the
| database into two synchronizing copies, you increase the
| availability with a second copy but that leads to the
| potential for both copies to be inconsistent.
| mrkeen wrote:
| Why not get rid of git too while you're at it?
|
| Just store the current state of the code base. You can't have
| merge conflicts without conflicting commits.
| tkiolp4 wrote:
| Exactly. These kind of devs (like OP) cannot be quiet and need
| to constantly introduce something "cool" so that they can get a
| good salary raise. The moment they cannot introduce more BS,
| they move on to another company. The poor other "average" devs
| need to maintain all the crap.
| mhd wrote:
| Next to the 97 part video tutorial on how to get back from your
| semi-hydrated micro-frontend to something sane for your 12
| person company ;)
| revskill wrote:
| THe problem is everyone's interpretation and implementation
| of micro-frontend is different.
| mrkeen wrote:
| Do it like an accountant would:
|
| Throw away all invoices and receipts, and just represent each
| customer's balance as a single number. When it changes, get
| your crayon, cross it out, and write the new number.
| colonwqbang wrote:
| The concept sounds interesting, but the article doesn't do a
| great job of explaining how it works. How do I efficiently
| reconstruct the current state from the event stream? How would
| the event stream be modelled in the database?
| alexzeitler wrote:
| There are several talks by the author:
|
| https://www.youtube.com/watch?v=gG6DGmYKk4I
|
| https://www.youtube.com/watch?v=jnDchr5eabI
|
| https://www.youtube.com/watch?v=ArcypYS5XBQ
|
| https://www.youtube.com/watch?v=uODSwR2CIV4
|
| He also maintains samples on GitHub:
|
| https://github.com/oskardudycz/EventSourcing.NetCore
|
| https://github.com/oskardudycz/EventSourcing.NodeJS
|
| https://github.com/oskardudycz/EventSourcing.JVM
| corethree wrote:
| Two ways to do it.
|
| 1. Use a database designed for this stuff. Google big query,
| Amazon redshift, clickhouse..etc. all current data is
| essentially a type of aggregation. Or in other words it's
| equivalent to a group-by query on an event database.
|
| It makes sense right? With events I can technically rebuild the
| current state or the past state of the data through some
| aggregation query.
|
| 2. Rename your relational storage and call it a caching layer
| that lives next to the event system. It's functionally the same
| thing but won't trigger any red flags in people who are
| obsessed with making everything event driven.
|
| The architecture he describes exists. It's just massively
| complicated so services that utilize it usually do very
| targeted things. Think Google analytics, data dog, splunk...
| etc. Etc.
| mrkeen wrote:
| > How do I efficiently reconstruct the current state from the
| event stream?
|
| There isn't one 'current state'. That thinking comes from
| centralising everything in one DB.
|
| You create different states in different systems according to
| different requirements. If you're building a shopping system,
| with Purchases and Customers, one service could read events and
| produce a relational table for finance purposes. Another
| service could read events and produce a key-value store of
| customer data. A third service could power an OpenSearch
| service for searching over products.
|
| > How would the event stream be modelled in the database?
|
| It's a list. If you're using something fit-for-purpose like
| Kafka, then it's multiple lists (topics, partitions, etc.).
| tiku wrote:
| It would make more sense to use this for certain streams that
| change a lot and the data is interesting enough to see what
| happened along the line. But that could be solved within the
| relational model..
| 2023throwawayy wrote:
| > Once you distinguish all events you're fine with and want to
| migrate your relational data, don't try to cheat; don't put your
| events as small and granular. Relational data is flattened; if
| you try to retrofit what happened from the final state, you will
| likely fail or not be precise at best.
|
| .. what?
| mrkeen wrote:
| I think I follow.
|
| The article is aimed at people who already have relational
| data, and want to build an event-driven system (whose events
| will eventually end up as relational data again downstream.)
|
| Your system A might look like: | Name |
| Balance | | Michael | $3.03 |
|
| You might design your system B to have the events
| AmountCredited{name, amount} and AmountDebited{name, amount}.
|
| You don't know how system A ended up at its current state.
| That's what's meant by "flattened". When you want to "migrate
| the relational data", i.e. convert system A's relations into
| events, it's tempting to use the obvious
| AmountCredited{"Michael", $3.03} because you know it will
| result in Michael having the correct balance in the final
| system.
|
| But it's not good to reuse AmountCredited, _because no such
| event actually happened_, which is why it could be called
| "cheating". If future-you looks at historical data, $3.03 won't
| correspond to any real transaction that happened. Instead you
| should instead make a special event like AmountImported{name,
| amount}.
| lelanthran wrote:
| For this example the convention in accounting is to use
| 'balance brought forward'.
|
| It's a real transaction that happened, and everyone knows
| what it means i.e. the previous ledger with this account was
| closed off and the new one has the balance that was there
| when th book was closed.
|
| Using 'imported' describes what you did, but not what the
| intention was.
| bananaowl wrote:
| The architecture from hell. Hard to debug. Why didn't my service
| pick up that event? Error handling. What happens with the state
| if a service throws an exception? Resource hog. How do I
| map/reduce all these events into a state?
|
| I do like events. They go into my elk stack where I can look at
| pretty graphs which gives me a story of how my system behaves
| over time.
| mrkeen wrote:
| > Why didn't my service pick up that event?
|
| _What event_? If you 're not event-sourcing, you can't even
| ask that question. Instead, some user interacted with the
| system and got a 500. Maybe you got a stack trace in the logs,
| but the user's data is gone (or maybe half of it was stored).
|
| Persistent events mean you get to fix the problem and try again
| - no data lost.
|
| > What happens with the state if a service throws an exception?
|
| Whatever you programmed it to do. Same as any other service.
|
| > How do I map/reduce all these events into a state?
|
| There's no _one_ way to store them like there is in an RDBMS.
| UserService listens to User events. TransactionService listens
| to Transaction events. SearchService listens to both. All three
| have different schemas. Those schemas can be thrown away and
| redesigned at any time without data loss.
|
| Do a bad job of map/reducing them into a state today. Do it
| better tomorrow.
| xwowsersx wrote:
| I was looking forward to reading this based solely on the title,
| but I find the writing style and tone to be quite unbearable. The
| forced attempt at being relatable and light-hearted comes across
| as patronizing and distracts from the intended message or points
| being conveyed.
| hot_gril wrote:
| I don't take offense to the tone, it's just too much text and
| too little substance.
| move-on-by wrote:
| I was a team once that strongly considered event sourcing. To me,
| it seemed like a solution looking for a problem. It could have
| worked for us, but we ended up passing on it as the benefits were
| not immediately clear and the risk of doing something new and the
| lessons learned that would come with it just didn't seem in the
| best interest of the project/company. Maybe that makes us tools
| for passing up a learning opportunity, but I don't regret getting
| into that rabbit hole without a fox chasing us down it.
| Nextgrid wrote:
| A boring, conventional system that works is a threat to a
| bloated engineering team who don't have any work to do & polish
| their resumes with and might feel at threat of redundancy. That
| is the "problem" this solution solves.
| alecco wrote:
| Temporal databases make a lot of sense for financial data, for
| example.
|
| But in most cases you can just have a normal database and store
| the historic changes in auxiliary tables. So the main database
| is kind of a materialized view.
| GartzenDeHaes wrote:
| The traditional way to handle this is to use a GAAP style
| transaction/journal and a roll-up summary table. The current
| state can be reconstructed from the read only transaction
| table and you don't need any complex event processing system.
| theteapot wrote:
| > The current state can be reconstructed from the read only
| transaction table ..
|
| Isn't that called event-sourcing?
| devjab wrote:
| Almost every piece of data we store in SQL would be better on a
| document database, but since nobody is familiar with those we
| keep on trucking. I don't mind too much, I don't even think we
| made the wrong choice, but it does cause us some issues with
| how we have to handle data model changes.
|
| I think most data storage didn't really keep pace with how a
| lot of software is being build now though, and things like
| events and queues are what we build on top of what we have
| because we need it. For the most part a lot of the data
| relations happen outside of our databases today through various
| services, because that's just how the modern IT landscape looks
| in many organisations. You'll have internal master data that
| supports different teams in the business and interacts with
| 300+ different IT systems and applications in order to
| streamline things. With micro services it's easy to keep the
| business logic and data models clean, but then you need to
| manage events, queues and data states as well as reliant
| storage. Which is just so complicated right now.
|
| I do like SQL but these days, the systems we're building could
| frankly be put in a SQLite and be perfectly fine, well almost.
| zabzonk wrote:
| all the comments are negative, but the post at this time 64
| upvotes - why? i've seen this so often on HN, but i really don't
| understand it.
| simonbarker87 wrote:
| Hate the article enough to leave a negative comment, want to
| see the fall out of it and have it not drop off the front page,
| so stick an upvote on it as well would be my guess. I have
| commented on this but haven't upvoted as it clearly a bonkers
| article.
| jupp0r wrote:
| Because you can't downvote submissions, you can only go in and
| write a negative comment. Makes complete sense to me to see the
| effects of that with this article.
| zabzonk wrote:
| yes, i'm not denying the comments, it's the upvotes that i
| don't understand.
| jupp0r wrote:
| There are probably some people who find the article
| interesting and well written or who upvote based solely on
| the headline without reading it.
| lolinder wrote:
| Certain topics get a lot of interest (positive and negative)
| based on the title alone.
|
| Event-driven isn't quite peak hype any more, but it still gets
| a lot of instinctive love from a certain group of people, and a
| lot of instinctive hate from another. So you get a whole bunch
| of upvotes (but they don't have anything substantial to say
| about it), then a whole bunch of negative reactions in the
| comments based on the title alone. And then in this case, you
| get a whole bunch of negative reactions from people who tried
| to read the article and couldn't get past the weird tone.
| RaftPeople wrote:
| I don't think it's just the tone. At the start of the
| article, it implies relation and CRUD approaches should and
| will be replaced by Event Sourcing approach, but he doesn't
| support that, or even provide a good sense of pros and cons.
| In one of his comments (comment section) he mentions that the
| article was supposed to be a "how" not "why", and links to a
| "why" article, but that "why" article also doesn't do a good
| job of why with pros and cons.
|
| In summary: not a good presentation of the pros and cons
| which allows a person to clearly identify under what
| conditions this approach might be a reasonable tool.
|
| Additional note: A typical competing model, IME, is a
| relational model that writes business events at the same time
| state updates occur.
| lolinder wrote:
| It's definitely a bad article. I was more commenting on why
| there isn't much substantial conversation here, even though
| there are lots of upvotes. Very few people even bothered to
| read it because the tone was so bizarre--I couldn't get
| past it, but it's not surprising to me that the content is
| low quality too.
| simonbarker87 wrote:
| What on earth is this article trying to accomplish? The tone is
| bizarre and the underlying concept sounds horrendous to work with
| if you truly want to replace your static data store with it. By
| all means add a formal event layer on top of your existing data
| store but to replace it sounds madness.
|
| If that's not what the article is proposing then for once I'm
| going to say it's not a failure of my intelligence, it's the
| articles fault here.
| AtlasBarfed wrote:
| Four years ago I heard "Kafka IS your database".
|
| I thought maybe these insane people (probably parroting some
| tech company enterprise penetration propaganda e.g. Confluent)
| would have a better story, but... no.
|
| Anyway, yeah, sure, keep logs. But a lot of that article about
| commands and events is something that only exists if you had
| one ubiquitous language, system, and OS. You know, the almost
| literal "seamless" where there aren't any seams.
|
| Sure that will probably plug into some enterprise bus and
| enterprise integration and enterprise ... anyway.
|
| Competent developers will understand what events to preserve
| and log and possibly allow retry/repeats.
|
| Anyone who has looked at a Splunk bill will realize that just
| storing all the logs everywhere is very expensive, which is
| another way of saying "wasteful". But any generic enterprisey
| event system will basically start and end with splunk-level log
| aggregation and kinda-analysis.
| dgellow wrote:
| Dealt with something like this when I joined a previous job
| years ago, Kafka was the main data store, microservices had
| their state in memory they would build at startup by
| processing their whole history of events (that was
| ridiculously slow for some of them). Then GDPR came in place
| and the whole "keep everything in Kafka forever" had to face
| the reality of "not allowed to keep PII for longer than 30
| days" :)
|
| (Before someone suggests it, no messages weren't encrypted,
| just throwing the key away wasn't an option)
| jabradoodle wrote:
| Would be expensive but you can do a copy and replace to
| keep the data you need, on a new topic.
|
| If you don't need the data, then you don't need it.
| andsbf wrote:
| > ... they would build at startup by processing their whole
| history of events (that was ridiculously slow for some of
| them)
|
| There is a basic technique to solve this, you snapshot
| every "nth" event
| corethree wrote:
| Eh. The model he describes is actually standard for analytics.
|
| And because it's standard there are literally databases
| designed and optimized to do what he says. It's not madness
| when it already exists and is really common.
|
| Think, redshift, snowflake, biq query, clickhouse..
|
| Additionally their already exists user interfaces and web
| services that already do what he says.
|
| Datadog, splunk, Google analytics... Anything related to logs,
| analytics and aggregation of those analytics. What he proposes
| actually already exists.
|
| That being said I don't agree with the articles point to
| replace everything with this model. Usually these types of
| services target very specific use cases.
|
| I think your reaction is a bit extreme here. I don't agree with
| his proposed model but I see where he's coming from and it's
| not that the model won't work... It's been proven to work from
| all the examples I gave above.
|
| The problem with it is that it's just slower and much more
| complicated. But his proposal does increase the capabilities of
| your data.
|
| You can increase speed by having a pre-caching layer for your
| aggregations. Basically what was originally your static store
| is now a caching layer where the developer or user pre
| specifies an aggregation that the system should count live as
| the events come in as well as throwing the events into the
| event db. If when querying for that aggregation you get a
| "cache miss" then it hits the event layer and has to do the
| aggregation job live.
|
| So essentially if you build it like this you have all the
| capabilities and speed of your original static data store but
| now you have the ability to re aggregate events differently so
| you have MORE ways to deal with your data. It can work and it
| will have more features its just really really really
| complicated to make an entire system centered around events.
| Additionally theres also a boatload of extra data to deal with
| which is another engineering problem.
|
| That's why when people do build these systems it's usually
| centered around some business requirement that absolutely needs
| this ability to dynamically query and aggregate events. Logs
| and analytics being the two big ones. Or some service to data
| scientists as well.
|
| The theory behind it is attractive. All static data can be
| represented as a series of events. In fact static data is
| simply the result of a certain of aggregation query on an event
| database. It's attractive to use smaller primitives in
| programming and build higher level abstractions through
| composition so this style of event driven services seems more
| fundamental and proper. But of course like I said there's
| practical issues with it when you look past the theory such
| that this model is usually only applied to the specific use
| cases I mentioned above.
|
| So there is a failure here. Not of your intelligence. Failure
| of your experience.
|
| And as I side note I agree with you on the tone of the article.
| He's trying to be witty but he's trying too hard.
| hot_gril wrote:
| Relational and event-driven aren't exclusive concepts, that's
| the problem with the article. Also, it'd help to have a real
| example of the solution it proposes, since we all know the
| "old" way it describes is in Postgres/MySQL/whatever.
| alephnan wrote:
| > What on earth is this article trying to accomplish
|
| Most of these articles are for the author to promote themselves
| mrkeen wrote:
| > What on earth is this article trying to accomplish?
|
| Most articles explain building event-driven systems from a
| greenfield point of view. This article is for when you want to
| build an event-driven system but you already have brownfield
| relational data.
| freecodyx wrote:
| Sounds like a junior work
| nivertech wrote:
| It's top-down vs bottom-up, or custom vs generic.
|
| Top-down vs bottom-up:
|
| Top-down: starting from the business domain, and then mapping an
| implementation onto available technologies, tools, and vendors.
|
| Bottom-up: starting from the available technologies, tools, and
| vendors, and thinking how to bolt up a working solution out of
| them.
|
| Custom vs generic:
|
| Custom: DDD, CQRS/ES, Sagas, TBUI (Task-based/driven UI),
| GraphQL, Algebraic Data Types, etc.
|
| Generic: RDBMS, CRUD, REST, ACID transactions, CDC, generic admin
| UIs, nocode/lowcode, limited/generic types, etc.
| asah wrote:
| 2c: if you need PostgreSQL elsewhere in your app anyway, then
| store your event data in PostgreSQL + FOSS reporting tools
| (apache superset, metabase, etc) until you hit ~2TB. After that,
| decide if you need 2TB online or just need daily/hourly summaries
| - if so, stick with PostgreSQL forever[1]. I have one client with
| 10TB+ and 1500 events per sec @ 600 bytes/rec (80GB/day before
| indexing), 2 days of detail online and the rest summarized and
| details moved to S3 where they can still query via Athena SQL[2].
| They're paying <$2K for everything, including a reporting portal
| for their clients. AWS RDS multi-AZ with auto-failover
| (db.m7g.2xlarge) serving both inserts and reporting queries at
| <2% load. One engineer spends <5 hours per MONTH maintaining
| everything, in part because the business team builds their own
| charts/graphs.
|
| Sure, with proprietary tools you get a dozen charts "out of the
| box" but with pgsql, your data is one place, there's one system
| to learn, one system to keep online/replicate/backup/restore, one
| system to secure, one system to scale, one vendor (vendor-
| equivalent) to manage and millions of engineers who know the
| system. Building a dozen charts takes an hour in systems like
| preset or metabase, and non-technical people can do it.
|
| Note: I'm biased, but over 2 decades I've seen databases and
| reporting systems come & go, and good ol' PostgreSQL just gets
| better every year.
|
| https://instances.vantage.sh/aws/rds/db.m7g.2xlarge?region=u...
|
| [1] if you really need, there's PostgreSQL-compatible systems for
| additional scaling: Aurora for another 3-5x scaling, TimescaleDB
| for 10x, CitusDB for 10x+. With each, there's tradeoffs for being
| slightly-non-standard and thus I don't recommend using them until
| you really need.
|
| [2] customer reporting dashboards require sub-second response,
| which is provided by PostgreSQL queries to indexed summary
| tables; Athena delivers in 1-2 sec via parallel scans.
| datadeft wrote:
| When a comment on HN has more merit than the article.
|
| The only problem with postgres is that inserting has some
| interesting scaling problems. Putting a queue between the event
| sources and the db is usually recommended.
| withinboredom wrote:
| > Putting a queue between the event sources and the db is
| usually recommended.
|
| Emphasis on _usually_. If your db is at 2% CPU utilization
| with/without queues ... you probably don't need a queue.
| asah wrote:
| see my reply above - one classic case is db system
| maintenance.
| refset wrote:
| "normalize until it hurts, denormalize until it works" is
| evergreen advice for scaling both reads and writes.
| Synchronously enforcing referential integrity and other forms
| of normalized constraints is what gets expensive.
|
| Pat Helland has some really good writing on this stuff, e.g.
| https://pathelland.substack.com/p/i-am-so-glad-im-
| uncoordina...
| ldng wrote:
| And in between, at the coma, revise seriously your indexing
| policy and don't hesitate to remove unused and underused
| index (even on foreign key if you don't need them that
| much). People too often underestimate the impact of
| rebuilding and index on large inserts.
| asah wrote:
| thanks! forgot to mention queuing (e.g. SQS), which is SUPER
| valuable, for example when you want to do large scale
| maintenance on the database (major version upgrade where the
| on-disk format can change)
| varelaz wrote:
| SQS has eventual consistency, you can get the same message
| twice on 2 different intances for example (which was often
| the case for my projects). I would rather suggest Amazon
| MQ.
| layer8 wrote:
| > Putting a queue between the event sources and the db is
| usually recommended.
|
| That depends on the nature of the events and whether you can
| live with the database being out-of-date while the events are
| still in the queue.
| btown wrote:
| Along these lines, if you need the ability to "time travel" and
| "recover overwritten state" and "reinterpret the events of the
| past" sometimes all you need is audit logs that maintain
| snapshots of pre-save data, and a script that identifies and
| collects instances of a specific sequence of events, which a
| human can review and bulk-apply as necessary to backfill the
| effects of new logic.
|
| https://django-simple-history.readthedocs.io/en/latest/ and
| similar tools are a semi-reliable and simple solution for
| building audit tables, or you can add Postgres triggers if you
| need to audit direct database access.
|
| I love event sourcing in theory. In practice, there's so much
| boilerplate necessary to add a new CRUD workflow, or to quickly
| and reliably roll out the types of interventions and hotfixes
| that early-to-mid-stage startups need to do all the time for
| unforeseen circumstances. Unless you're doing something like
| implementing payment processing rails, event sourcing may not
| be the right choice.
|
| https://news.ycombinator.com/item?id=17817375 (2018) has some
| good conversations on the downsides to event sourcing as well.
| agentultra wrote:
| The problem with audit logs for deriving state is that
| migrations are a forgetful operation. They're great for
| telling you who changed what and when... but if the table
| lives long enough the audit logs will hold references to
| columns that no longer exist or data that is gone with no way
| to trust that it can be recovered properly.
|
| The author has another post on that site on when to avoid
| event sourcing.
| macNchz wrote:
| > I love event sourcing in theory. In practice, there's so
| much boilerplate necessary to add a new CRUD workflow, or to
| quickly and reliably roll out the types of interventions and
| hotfixes that early-to-mid-stage startups need to do all the
| time for unforeseen circumstances.
|
| The biggest pain point I had working on an event sourced
| system at an early stage startup was around schema changes,
| where changes (especially quick or unplanned) introduced
| something pathological in event data for some period of time
| that we didn't realize was an issue until it conflicted with
| some change much later on.
|
| Often the discovery of these issues would come at a bad time
| and block something important, e.g. because of small
| differences in dev environment seed data and real production
| data, leading to temptation and/or pressure to "just do a
| quick tiny mutation" to old events to address the problematic
| events from the past mistake, which realistically often just
| caused a different issue, because reasoning about all of the
| potential impacts of the change was hard.
|
| These days in early-stage land I am only interested in event
| sourcing for contained areas of a product where it can
| provide real value, rather than as a cornerstone of an entire
| application.
| gukoff wrote:
| From your experience, where does it bring value?
| lacrimacida wrote:
| Im not the OP but I'd chime in: first and foremost it's
| on the resume.
|
| Of course im not bashing event sourcing altogether but
| right now resume boosting seems to be the driving force
| to apply it everywhere.
|
| And this comes from my experience at my current
| workplace. Event sourcing has brought so much unnecesary
| complexity to a an older system and it's not even
| implemented properly. The engineers who took the
| initiative padded their resumes then went on to greener
| pastures while leaving a giant mess behind.
| gopher_space wrote:
| > I love event sourcing in theory.
|
| I've never heard this term before. From a quick scan it seems
| like a design tool for reasoning with unfamiliar schemas but
| that doesn't match the conversation. Would you mind
| explaining the context around its use or suggesting a link?
| dgellow wrote:
| Let's say I want to setup such a system, any idea where I could
| find a detailed write up?
| ryanjshaw wrote:
| This is an extremely well documented postgresql event
| sourcing reference implementation: https://github.com/eugene-
| khyst/postgresql-event-sourcing
| dfee wrote:
| Is the general idea to have a table with definition
| {id:uuid,created_at:timestamptz,data:jsonb}?
|
| It's difficult to get index functionality in JSONB, especially
| against diverse event structures and event definitions that
| evolve.
|
| I guess I should become more familiar with:
| https://www.postgresql.org/docs/current/datatype-json.html#J...
| ChicagoDave wrote:
| I'm on board with event-based architectures, but this article
| struggles to get its point across.
|
| I would focus on the difference between data relations and
| business behaviors. Once you start thinking in terms of behaviors
| and business activities, the move away from operational
| relational data stores becomes much more obvious.
| wrestlingmonkey wrote:
| Lot of negative comments so thought I should counter as I quite
| liked it. The article style isn't great but I get the impression
| it was only ever meant to be a toe dip into one aspect of
| thinking about events.
|
| I've worked at a couple of places that have had a lot of success
| with EDA, DDD and microservices. I'm sure these patterns are
| considered by many as buzzwords but they've been around for a
| long, long time now and they can be very effective. There's no
| such thing as a silver bullet but the problems these ideas look
| to solve I've experienced very often, so I don't buy into it
| being a solution in search of a problem.
| CodeCompost wrote:
| Didn't read the article but I'm in the process of eradicating
| Event Sourcing from a codebase and returning to the classical
| ACID database model. The boneheaded decisions made by our
| predecessors is staggering and choosing to use Event Sourcing for
| everything is the dumbest of them all.
| capableweb wrote:
| Ok? Not sure what you want to talk about here, you probably
| need to give us a bit more context, especially if you even
| acknowledge you haven't even opened up the article to talk
| about the submission itself...
|
| Sometimes, the situation when something gets created and
| designed, looks very different from the current situation
| you're in N years later. So what might have looked like a
| boneheaded decision, could have been the best decision _at that
| point_.
|
| But us engineers like to lament our predecessors' code, I'm
| guilty of this sometimes too. But I try to remember that I
| don't have the full context of how things were when the code
| was initially written.
| mrkeen wrote:
| > But I try to remember that I don't have the full context of
| how things were when the code was initially written.
|
| Ironically, that's what event-sourcing _is for_.
|
| If the facts were kept, then a better decision can be made
| today.
| lolinder wrote:
| I worked on a project where we decided that event sourcing was
| the way to go for a variety of legitimate business needs. We
| then _implemented it_ with ACID transactions--an application-
| level framework writes the event to the Postgres database and
| _in the same transaction_ updates all the computed views. At
| the scale we were working, this was totally fine performance-
| wise.
|
| Most people who have had bad experiences with event sourcing
| were actually having bad experiences with eventual consistency.
| All that event sourcing means is that you treat the events as
| the source of truth and everything else as computed from those
| events (and you could theoretically recompute it all at any
| time). Eventual consistency is an implementation detail and not
| a necessary one: you can implement event sourcing in a single
| Excel file if need be.
| hot_gril wrote:
| Just beware, if you're using Postgres or MySQL, it's not fully
| ACID (specifically "I") unless you run xacts in serializable
| mode.
| politician wrote:
| What are the major deficiencies of ES in that particular
| codebase? If you could drop some specifics in bullets that
| would be really helpful for me. I promise not to ambush you
| with apologetics.
| CodeCompost wrote:
| This system was developed from 2008-2013, a very different
| time when hardware was "cheap" and the Cloud was not not a
| thing.
|
| Event Sourcing dictates that Events are never deleted which
| means that the data volume keeps growing and growing. There
| is - in this system - no way to delete old events. When I
| brought this up, the response was "Just add another hard
| drive". In the modern Cloud era, adding a hard drive is
| extremely expensive.
|
| The system uses CQRS and all events generate reports that are
| stored in Elasticsearch. Data is never deleted, only an extra
| event gets added /saying/ it's deleted. The data is still
| cached in Elasticsearch. All of it, all the data back to
| 2013. Added an extra 32GB of RAM just to keep up with it is
| ludicrously expensive.
|
| We're in Europe. Guess what a system like this does to GDPR.
| Can you tell me which events I need to delete when somebody
| says they want to be forgotten? Yeah.
|
| I can't delete old data. It's impossible without collapsing
| the entire system like a house of cards.
|
| Finally, and this is the piece de resistance, the developers
| decided to develop a relational database structure ON TOP OF
| EVENT SOURCING. We're talking primary keys, foreign keys,
| cascading and non-cascading deletes. Importing an Excel sheet
| of 10000 rows takes TWO WEEKS because it generates hundreds
| of events per Excel cell that is being read. We brought it
| down to 10 minutes and there is plenty of room for
| improvement it's just that we have other priorities right
| now. Currently, simple flat "tables" with no foreign keys
| take several seconds to import (just like in a regular RDMS).
|
| Oh yeah note that this is a system that is used by 4-5 users
| at a time, not hundreds or thousands of users.
| revskill wrote:
| No, what you need is a command queue, command event is not domain
| event.
| ulrischa wrote:
| I made a php demo for a idea: a event based observer based
| modelling system. I.e. for game of life:
| https://github.com/ulrischa/OCell
| audnaun252 wrote:
| Modelling domain events is useful for describing the problem your
| trying to solve with the domain experts, and it should probably
| be left in the documentation when planning a solution.
|
| For actually implementing a system that provides an audit trail
| of long-lived state machines, you're probably better off using
| something like Temporal.io/durable functions which uses event
| sourcing internally for their persistence, and has a programming
| model which forces you to think about deduplication/idempotency
| by adding different constraints for the code that orchestrates
| the functionality (workflows), vs the code that actually
| interacts with the real world (activities)
| tiku wrote:
| Yeah, ehh I'm just going to stick to good old fashioned
| relational data.
| agentultra wrote:
| Good, do it until you can't. Don't use a hammer on a screw.
| kinddevil wrote:
| Question has been asked every time when someone wants to use
| event sourcing -- why do you want to redo the stuff that
| databases already did? "Ensure your design works if scale changes
| by 10X or 20X, not 100x".
| baq wrote:
| because the database is inconsistent and incorrect is usually
| the answer.
| onion-soup wrote:
| IT is doomed
| asimpletune wrote:
| Something that might be missing from these discussions is when
| event driven architecture is even appropriate. The short answer
| is if your customer did something and expects a response it's not
| even driven, that's just request/response.
|
| Event driven is when something happens out of band. E.g. you push
| your code to GH, which triggered a build. In this example, you
| reloading the page to see that your updated code is
| request/response, however that CI build that was enqueued is
| event driven.
|
| Hope that helps.
| jstummbillig wrote:
| I was not aware of event driven design until very recently, but
| coincidentally cooked something very much like it up, when
| thinking about what role data will play in a AI world.
|
| While it's clear how event driven design might have been a good
| idea in the past, given you can manage the complexities and
| actually make use of the data, the value that AI will be able to
| extract from it for any business will make it ubiquitous over the
| coming years.
___________________________________________________________________
(page generated 2023-12-16 23:00 UTC)