[HN Gopher] Databases are the endgame for data-oriented design
___________________________________________________________________
Databases are the endgame for data-oriented design
Author : cloutiertyler
Score : 152 points
Date : 2023-12-06 15:48 UTC (7 hours ago)
(HTM) web link (spacetimedb.com)
(TXT) w3m dump (spacetimedb.com)
| jitl wrote:
| If the site designers are on here by chance, take a look on
| iPhone. The blog post is quite squished there, pretty much
| unreadable.
|
| EDIT: nevermind, fixed after a refresh -\\_(tsu)_/- still maybe
| worth a look
| cloutiertyler wrote:
| I am on here. Will definitely take a look!
| vagrantJin wrote:
| another thing. firefox issue.
|
| firefox on android no longer supports mobile view?
| marshray wrote:
| While you're here... where's the scrollbar?
|
| I'm on desktop (Edge) and don't see a scrollbar.
|
| Very distracting.
| samsquire wrote:
| Thank you, I enjoyed this post.
|
| I am thinking that iteration is just traversal and traversal is
| just execution.
|
| Take iterators in C++ or standard library "algorithms" library in
| C++. Or iterators in Rust or in Java.
|
| You just want to traverse and collect values (and calls on
| functions in some interleaving), which is like joining tables as
| this article says.
|
| I'm thinking that the definition of the traversal (such as Kafka
| pipeline, clojure threading syntax, Clojure data driven design,
| Javascript lodash pipeline) can be mapped to tuples and then the
| computer can optimise arbitrary chains of traversals based on the
| number of potential tuples are available and what traversals are
| equivalent paths or routes to the same traversal.
|
| In other words, every program is a compiler pipeline or database
| query pipeline.
|
| Maybe Prolog and Datalog can help here. Optimisation of arbitrary
| traversals and determination of identical traversals.
|
| Or monads are just ordered traversals and OCaml compilers are
| just traversals of execution (function application) and
| relationship following, which are joins.
|
| The relational model is everywhere.
| BoiledCabbage wrote:
| > In other words, every program is a compiler pipeline or
| database query pipeline. > Or monads are just ordered
| traversals and OCaml compilers are just traversals of execution
| (function application) and relationship following, which are
| joins. > The relational model is everywhere.
|
| This is interesting - could you elaborate a bit more?
| fifilura wrote:
| My colleagues hate me, but I also found that SQL is The way to
| write business logic.
|
| Lots of caveats about difficulty to test and weird syntax.
|
| But it is just that SQL is the most terse and standard way so
| express logic.
|
| And that in itself is the most important factor to avoid bugs.
| Not what testing strategy you choose.
| Etheryte wrote:
| I feel like you've come to the right conclusion, but partly for
| the wrong reasons. Terseness in and of itself is not useful per
| se. Code golfing languages are the tersest there is, but we
| don't write code that way.
| ndriscoll wrote:
| Or they just used the wrong word. I'd agree with them, but
| clarify that SQL is not just terse, but concise.
|
| A join or a group by is going to be much clearer than writing
| the code the query plan is going to generate, creating
| temporary hash maps, doing nested loops, etc.
| randomdata wrote:
| _> And that in itself is the most important factor to avoid
| bugs._
|
| I'm not sure about that. SQL was the first language I learned
| and the language that has always been there throughout the
| decades, and is also where I make the most mistakes.
| hot_gril wrote:
| If you were to write the equivalent of a SQL query in some
| other language, you'll probably make a lot more mistakes.
| Especially if you're trying to achieve the same performance.
| And I mean a read-only query, not even something with multi-
| writes and locking.
| surprisetalk wrote:
| I completely agree!
|
| Transaction-safety is also sublime for business logic.
| buryat wrote:
| that's how you end up with multi-thousand LOC sql files that
| contain all the historical business logic and edge cases of a
| company and it just keeps being piled up on and its complexity
| keeps growing because not everything is expressible in SQL. So
| you end up having a big monster that someone needs to support
| and one that's not very testable easily
| nextworddev wrote:
| What you need is a semantic layer on top of SQL
| cma256 wrote:
| Glad that doesn't happen any where else!
| nightpool wrote:
| Most other programming languages are more composable than a
| SQL query. You can make functions, split logic into
| different classes, extract common logic, etc. Much harder
| with big sql queries
| ako wrote:
| Use views and stored procedures if you need
| composability.
| FridgeSeal wrote:
| Counterpoint: don't do this. The path to hell is paved
| with good intentions.
|
| Some co-workers and I inherited a code-base where the
| authors went down the views and stored procedures route.
| It was basicallly impossible to untangle; there was no
| knowing _what_ relied on a view or a proc, so you
| couldn't touch them at all, there were no docs, there
| were duplicates of everything (and again, no way to know
| what's used). A good number of them had all kinds of side
| effects and weird performance characteristics. They
| weren't version controlled.
|
| The issues just kept going.
| whstl wrote:
| If your colleagues are not applying basic engineering
| rules to views and stored procedures, it's not fair to
| say that this is a problem with views and sprocs.
|
| They can and should most definitely be testable,
| documentable, trackable and version controlled.
|
| Should one also judge all backend programming by PHP
| standards circa 1998?
| FridgeSeal wrote:
| I tend to judge things by the average-worst-case that
| they enable and how easily things go wrong, and IME
| stored procs specifically, go badly real quick.
| ako wrote:
| You're blaming the language for the bad coding practices
| of the developers. You can build maintainable database
| code, and modern SQL IDEs like Datagrip provide similar
| tools for SQL like you have for other languages, e.g.,
| refactoring, dependency/usage info, versioning, etc. The
| downsides you mention are not inherent to SQL.
| camgunz wrote:
| You can define functions in all major databases
| manicennui wrote:
| And how many non-technical users do you know who do this?
| ako wrote:
| And how many non-technical users do you know that write
| good javascript/java/.net/etc code?
| ako wrote:
| And in any other language this wouldn't result in multi-
| thousand lines of code with historical business logic and
| edge cases?
| manicennui wrote:
| The difference is that most people don't write their
| business logic in a single flat function in most
| programming languages, and there is a good chance that
| individual pieces would have tests.
| ako wrote:
| Neither do good SQL developers: you can easily structure
| database logic in multiple stored functions, stored
| procedures, packages, and multiple files. Unit test
| frameworks are also available.
| robertlagrant wrote:
| Testing code is pretty popular in most non-SQL
| environments.
| ndriscoll wrote:
| Testing SQL is easy. E.g. the JVM has H2, which works for
| simple stuff, or you can use testcontainers, or just spin up
| a container and run your tests against that. You just run
| your migrations, insert mock data, and run your test.
|
| In fact testability is one of the best parts. You can safely
| test read-only queries against a prod secondary database to
| see that it gives reasonable results on real data, and use
| the repl to explore parts of your query to really get a sense
| that things are working.
| whizzter wrote:
| SQL queries yes, what Bury above probably is talking about
| is obscure triggers/stored-procs,etc that encodes the
| business logic inside the database far away from any
| version control or sane ways to track it.
| crazygringo wrote:
| > _But it is just that SQL is the most terse and standard way
| so express logic._
|
| As someone who has written a _ton_ of complex SQL... I couldn
| 't disagree more.
|
| Trying to shoehorn things that can trivially and intuitively be
| expressed in a couple of for-loops with a couple of variables,
| into SQL expressions making use of joins and aggregate
| functions and GROUP BY's and (god forbid) correlated
| subqueries... having to replicate subqueries because their
| results in one part of the query can't be re-used in another
| part... teaching people arcane terminology distinctions like
| between WHERE and HAVING... not having basic functions for
| calculating percentiles or even a basic median... certain basic
| kinds of logic operations that simply can't be done... flipping
| a coin on whether the query engine will use the index and
| execute in 10 ms, or decide to follow a different execution
| plan and take 5 minutes to run...
|
| I have never encountered _more_ bugs in business logic than in
| dealing with SQL. It obfuscates what should be clear. SQL isn
| 't a solution for avoiding bugs, it's what _causes_ so many of
| them.
| deadbabe wrote:
| Are you using Postgres? It seems like you haven't discovered
| CTEs or lateral joins.
| crazygringo wrote:
| Oh, I've worked with all the major engines. To which we can
| add: trying to remember which seemingly basic features are
| supported by which engines, and which aren't (e.g.
| percentiles are great in Postgres, but you're SOL in
| MySQL). That there's virtually nothing "standard" about SQL
| syntax or functionality at all, except for the simplest of
| queries and the general basic concepts.
| paulddraper wrote:
| Well Java and JavaScript differ a lot too.
|
| Some similarities, but a lot of differences too.
| magicalhippo wrote:
| We're in the process of migrating our on-premise business
| application from SQLAnywhere to MSSQL.
|
| Imagine the fun we're having...
| eddd-ddde wrote:
| I'm really curious, in the practice, what causes a
| product to decide "we are moving from abcSQL TO xyzSQL" ?
| crazygringo wrote:
| In that direction, it's usually going to be about
| robustness -- more tooling, less edge cases, a wider
| developer base, built-in cloud support, and so forth. The
| more popular a database is, the easier just _everything_
| is.
|
| Occasionally it's going to be some very specific aspect
| of performance limits, or a feature like spatial indexes
| that work in a special way. And occasionally it's going
| to be about cost or better integration with a particular
| cloud or other specific piece of tooling.
| magicalhippo wrote:
| For us there's been two major motivations, and another
| few lesser.
|
| First and most prominent is that Sybase was bought by SAP
| and SAP has pivoted SQLAnywhere from a general RMDBS to
| focus on mobile applications (ie more embedded). Apart
| from product development changing focus we've struggled
| to buy licenses for our customers, so the writing on the
| wall was clear.
|
| The seconds is that we've got some big customers who run
| their own MSSQL servers, and they've been pushing harder
| and harder for us to use their servers so they can handle
| administration and access control directly.
|
| A third motivation is that while we've been primarily on-
| prem, a lot of customers these days don't want that, so
| we've started offering cloud hosted options. Since we're
| a Windows shop Azure made sense, and as such the hosted
| MSSQL also makes sense and is quite convenient.
|
| Fourth is that, while MSSQL is quite anemic in terms of
| SQL language functionality, it does support transactional
| schema updates, which makes schema upgrades much less
| error prone.
|
| Given that MSSQL syntax support is quite limited and we
| now have developed tooling and library support for
| multiple databases, we'll probably also add support for
| PostgreSQL later.
| stvltvs wrote:
| I agree that CTEs are a solution to some of the pain listed
| above, but CTEs aren't exotic. Lots of the major RDBMSs
| support them.
| fifilura wrote:
| CTEs are an absolute must to get the right "do-this-then-
| that" approach.
| fifilura wrote:
| I am happy for all the disagreeing posts and that HN is a
| place for it.
|
| It also helps me understand my colleagues.
|
| It may be how our brains are wired. For me whenever I see a
| for loop I see bugs.
|
| It is also about that you get the concurrency from
| map/reduce-type jobs for free without having to think about
| it.
|
| But yeah there are a couple of places you need to be careful.
| Avoiding duplicates and handling of NULL.
| chrsig wrote:
| The types of issues you're enumerating are really part of a
| learning curve that every language and environment will have.
|
| > not having basic functions for calculating percentiles or
| even a basic median
|
| I don't think this is really true anymore; windowing
| functions are pretty prolific, and I think every major
| database will have some percentile functions
| crazygringo wrote:
| > _The types of issues you 're enumerating are really part
| of a learning curve that every language and environment
| will have._
|
| They're not though. That's why excellent coders often
| despise having to learn and use SQL, and basically just
| refuse to. SQL is _uniquely_ terrible -- and I say this as
| someone with a career spanning from Win32 C++ to every
| major web technology.
|
| > _every major database will have some percentile
| functions_
|
| MySQL still doesn't. MariaDB only added it in 2018, and
| I've been writing queries for a lot longer than that. (And
| MySQL only added window functions in 2018 as well.)
|
| And remember that, for various reasons, you're often
| (usually?) stuck using a database engine in production
| that's several years old, since upgrading a database engine
| is not something to be taken lightly.
| chrsig wrote:
| > They're not though. That's why excellent coders often
| despise having to learn and use SQL, and basically just
| refuse to. SQL is uniquely terrible -- and I say this as
| someone with a career spanning from Win32 C++ to every
| major web technology.
|
| This is basically a strawman + appeal to authority
| argument. I've no desire to bicker. "Uniquely terrible"
| is quite subjective, we'll have to agree to disagree on
| if it applies to sql.
| crazygringo wrote:
| Excuse me, but I'm certainly not "bickering" and I resent
| the insinuation. And you don't seem to understand what a
| strawman argument is.
|
| I'm not going to write an article on this for you, but
| you can certainly see the sentiment frequently expressed
| on HN that the rise of NoSQL and of ORM's is a direct
| result of otherwise capable programmers not wanting to
| deal with SQL and its conceptual building blocks such as
| joins. Take that for what you will, but you might suspect
| there's a reason for it, and that it's not merely the
| learning curve for "every language".
| sgarland wrote:
| I take it as the same reason they don't want to use
| boring technology like HAProxy, or why they think
| infrastructure in general is beneath them: because they
| are easily influenced by breathless grifters on Medium
| promoting the New Thing.
|
| SQL isn't going away any time soon. At its heart, it's
| relational algebra. If you can learn DS&A, you can learn
| relational algebra.
| paulddraper wrote:
| How many application languages have percentile functions?
| hot_gril wrote:
| Same, every backend service I write now has the majority of the
| business logic in SQL and a little pre/post-processing in
| regular code. A well-designed schema will mean that your
| queries don't get messy. If some more complex thing starts
| feeling forced, I add a bit more non-SQL code to make it
| reasonable.
|
| When teammates look at my code, and logic is all right there
| instead of scattered around, and there's a schema file backing
| it all that makes it clear what all the relations are, they
| have an easy time making tweaks or adding on. Yes it's very
| testable too.
|
| This also kinda depends on having a multi-service architecture
| if your system is large. Separate database for each. That's a
| good thing anyway.
| whstl wrote:
| The fear of committing SQL to the codebase is one of the most
| baffling things in modern backends.
|
| To me the most infuriating thing is the "SQL query scattered
| around multiple files" pattern, where a backend engineer will
| decompose a perfectly fine SQL query into 3 or 4 files, with
| multiple functions, often with very artificial separations
| (for example: a function just for the "select ..." part,
| another for the joins).
|
| All that in the name of having small files, small functions,
| small lines. You take complexity away from the "micro" parts
| and embed it into the invisible parts of your program.
| hot_gril wrote:
| We're using some tool at work that recommends putting SQL
| into dedicated separate files, one file per query. And we
| have a code wrapper around each of those to make calls
| easier. It's so annoying.
| devoutsalsa wrote:
| Have you written an ORM in SQL to help make querying easier?
| vlovich123 wrote:
| I've heard that datalog (and similar languages in that vein) is
| far easier and terser to write. What do you think about that?
| bob1029 wrote:
| We've been doing this for a while and it has some pretty severe
| limitations at scale. The biggest issue is that humans are
| human and you cannot make everyone else on the team use your
| elegant SQL DSL business engine in exactly the same way you had
| envisioned. Eventually, it will grow into a bit of a monster.
|
| We are headed back in the other direction, but with some hedge.
| Code with a very simple & transparent CI/CD experience seems
| equivalent to some SQL configurable thing in our minds now, but
| with way more potential upside.
|
| In our latest code-based abstractions, nothing is stopping you
| from breaking out a SQL connection and running any arbitrary
| query. In fact, all of the data is still in SQL and it's still
| authoritative, you just now have access to way more powerful
| tooling to work with that data. Code+SQL together is the
| answer.
|
| For non-expert team members, modern C# is turning out to be way
| more intuitive than SQL the moment you encounter a 3+ table
| join scenario.
| paulddraper wrote:
| The relational model is one of those eternal diamonds that will
| be just as relevant in 2090 as in 1990.
| riku_iki wrote:
| I kinda think adding some logical/semantic level on top of it
| (like datalog) will make it even more robust.
| hitpointdrew wrote:
| Nope, nope and nope. Went to the github page for spacetimedb, it
| does everything that is terrible.
|
| >Instead of deploying a web or game server that sits in between
| your clients and your database, your clients connect directly to
| the database and execute your application logic inside the
| database itself. You can write all of your permission and
| authorization logic right inside your module just as you would in
| a normal server.
|
| Why? Databases should never, ever, ever, be used to perform
| logic, they are datastores, that is it. Your logic goes
| elsewhere. Stored procedures are the worst "feature" of any
| database, you are just asking for a hard time debugging,
| troubleshooting, and increasing the chance that you will fuck up
| the most valuable part of your system, your data.
|
| > This means that you can write your entire application in a
| single language, Rust, and deploy it as a single binary.
|
| It also means you have a single point of failure, no read-
| replicas or redundancy. Hate everything about this.
| riku_iki wrote:
| > Why?
|
| less moving parts, diversity and complexity in your stack
| (different languages, servers, build/deployment systems, etc).
|
| > you are just asking for a hard time debugging,
| troubleshooting
|
| what specifically hard about this in your opinion?..
| deadbabe wrote:
| Can you keep an open mind? We've used stored procedures for
| years. It has worked wonderfully for creating a single source
| and producer of truth for business data. Instead of potentially
| having business logic across multiple repos and deployments,
| everything exists in one place, with absolute unquestionable
| authority.
|
| It's not difficult to debug at all, you might just be
| unskilled.
| acdha wrote:
| > It's not difficult to debug at all, you might just be
| unskilled.
|
| I agree with your larger point but this seems too harsh: it's
| definitely harder to debug simply because, as with
| microservices, understanding how the app is functioning now
| requires you to understand different code in multiple
| languages and locations, you're highly likely to hit non-
| portable behavior across databases for authoring and
| debugging, and you're never going to get a debugger with the
| whole flow in context.
|
| That doesn't mean there aren't benefits as well and it could
| be especially useful as a way to force distinctions about
| contracts for common operations, but I wouldn't say it's
| right for all or even most projects. The sweet spot is going
| to vary widely.
| gocartStatue wrote:
| The tradeoff seems to be ,,ability to deploy working software
| without reliance on single central authority". You may get
| rid of several smaller bottlenecks this way, introducing an
| enormous, all-encompassing one. Or am I wrong?
| rgbrgb wrote:
| > Databases should never, ever, ever, be used to perform logic
|
| You're talking about a best practice like it's a fundamental
| law. It's not, it's just how we've mostly been doing things. A
| lot of interesting innovations in distributed systems /
| architecture (serverless, graphql, thin clients, thick clients,
| ORMs, RSC, WSGI, nodejs) have been made because the designers
| tried relaxing a constraint or taking a counterintuitive idea
| to a maximalist place.
|
| In fact, if you look harder, there are a fair number of
| existence proofs of successful systems built on stored
| procedures. There's even a "best practice" phrase recommending
| doing compute as close as possible to the data.
| wharvle wrote:
| > Why? Databases should never, ever, ever, be used to perform
| logic, they are datastores, that is it. Your logic goes
| elsewhere. Stored procedures are the worst "feature" of any
| database, you are just asking for a hard time debugging,
| troubleshooting, and increasing the chance that you will fuck
| up the most valuable part of your system, your data.
|
| You cannot sanely use a database with multiple heterogenous
| clients without putting logic at least about _what valid data
| should be_ in it. This is gonna include some "business logic"
| in practically any real-world system.
|
| Otherwise you have to elevate the same functionality to some
| gatekeeper-daemon that'll almost certainly perform far worse,
| lack features, and be an eternal source of dumb bugs,
| including, I can just about guarantee, data corruption bugs.
| cloutiertyler wrote:
| This is of course what SpacetimeDB does. The stored
| procedures fail any transactions that are not authorized to
| be carried out. e.g. if a player is not high enough level or
| something.
| da_chicken wrote:
| > Databases should never, ever, ever, be used to perform logic,
| they are datastores, that is it.
|
| I wouldn't go that far. Relational algebra is performing logic.
| Constraints and foreign keys are logic, as well.
|
| I'm not going to argue that you should go back in time 15-20
| years and start shredding XML strings in stored procedures
| again. But thinking of the database as one step above a flat
| file is similarly backward thinking.
|
| More than that, the concept of putting application logic
| adjacent to the data store is sound. That's exactly what a web
| API or a microservice is doing. They allow a uniform mechanism
| of requests and responses to a data store. At a concept level,
| that's exactly the same thing. The concept of keeping logic at
| or immediately adjacent to the data layer so that a whole range
| of disparate applications can request and maintain data from
| the source is what the design goal of "database side logic" is.
| wvenable wrote:
| > At a concept level, that's exactly the same thing.
|
| But at a practical level they very different. If you have a
| middleware layer as you are describing, it's written in a
| real programming language with all the adjacent tools (source
| control, debugging, etc).
|
| I'm not hard-core against stored procedures used lightly but
| they have a lot of downsides and they simply aren't needed.
| There's no performance advantage. There are complexity
| advantages and disadvantages that might be a wash.
| coldtea wrote:
| > _If you have a middleware layer as you are describing, it
| 's written in a real programming language with all the
| adjacent tools (source control, debugging, etc)._
|
| And nothing stops a DB internal language used to write
| store procedures and such be a "real language" -- with
| source control, debugging and everything. RDBs can do a
| whole lot more offering an even better environment for
| those than common RDBs do - Smalltalk or Symbolics level
| good.
|
| In fact in the case of this project, that language is Rust.
| da_chicken wrote:
| > If you have a middleware layer as you are describing,
| it's written in a real programming language with all the
| adjacent tools (source control, debugging, etc).
|
| I'm not sure what you're using, but you can absolutely use
| source control for stored procedures. There's any number of
| database change management tools available. You should
| already be using one for your schema. Stored procedure
| debugging tools also exist for most platforms.
|
| I'm not really interested in the "it's not a real
| programming language" topic. That's almost universally
| someone going on an ego trip about what they like. If your
| point is, "We get to use a single language that the whole
| team is experienced and familiar with," then sure that's
| valuable. But that's about the team's capabilities more
| than anything.
|
| > There's no performance advantage.
|
| No, that's an outrageous claim. I've seen and implemented
| some processes as stored procedures and in some of those
| cases it's worked much better simply because we don't have
| to pull the data pool out of the cloud and across the
| country to wherever the CPU is, manipulate it, and then
| push it all back up to the data store.
|
| Stored procedures are not some universal panacea that the
| 4GL crowd wanted them to be, but it's also not something
| that's universally worse.
| shepherdjerred wrote:
| > you are just asking for a hard time debugging,
| troubleshooting, and increasing the chance that you will fuck
| up the most valuable part of your system, your data.
|
| This sounds like a tooling problem. One could imagine a
| database that doesn't have these issues.
|
| > It also means you have a single point of failure, no read-
| replicas or redundancy.
|
| What? Why would any of this be the case?
| paulddraper wrote:
| > It also means you have a single point of failure, no read-
| replicas or redundancy. Hate everything about this.
|
| How does writing an application as a single language/binary
| prevent read-replicas or redundancy?
|
| The application could do that.
|
| Or you could do it at disk level (RAID).
| coldtea wrote:
| Stored procedures ensure all your clients get the same logic.
| They're only "the worst feature of any database" if the
| language you're writing them in is not suitable (which is the
| case with PL/SQL and co, which were tacked on to RDBs and reek
| of bad 80s syntax and facilities).
|
| If the language is nice and has well designed access to db
| facilities like records and such, it can be better than writing
| your code outside the database, especially coupled with a data-
| oriented design model/ECS (which can be extremely debuggable
| and offer great visibility).
|
| > _and increasing the chance that you will fuck up the most
| valuable part of your system, your data._
|
| Since you can write anything outside you can inside, no. I can
| send a "delete from/drop <table>" from any client at any
| moment, or make any mistake in updating. That's what backups
| are for, and databases make them even easier (not to mention
| transactions being very good and neglected part of business
| logic).
| Xeamek wrote:
| Nothing you mentioned is an inherent problem with the
| databases. In fact, databases have arguably more potential to
| solve these problems then any currently used system.
|
| Like debugging - if every memory access is a database access,
| then you have a builtin logging for _all_ your memory access
| that is both more performant and optimal then any normal normal
| debugger. You can take snapshots of your 'memory' pretty much
| at any time, the data layout is clearly defined. You can
| manually edit your memory any time you want, and serialization
| is already solved for you.
|
| The more I think about it, the more possibilities I come up
| with. Damn, this is actually genius!
| munificent wrote:
| As an ex-game developer and software architecture nerd, I'm very
| excited about data-oriented design and ECS. It really is a cool
| pattern, and it's a very common one in shipping games today. It's
| not just architecture astronaut stuff.
|
| At the same time, the level of hype about ECS today reminds me an
| _awful lot_ of the amount of hype surrounding OOP in the 90s. Can
| ECS be a better way to structure your game entities and make your
| game loop faster? Yes. Will it make your teeth whiter and your
| partner love you more? No.
|
| (There are ECS frameworks in JavaScript, which gives you
| absolutely no control over memory layout and thus completely
| defeats one of the primary purposes of the pattern.)
|
| Like any pattern, it exists to solve concrete problems. It
| shouldn't be the One True Way To Think About All Programming
| Henceforth and Forever.
|
| When the author says things like:
|
| _> For example, how would you model chat messages in your game?
| I suppose you'd have to represent that as an entity in your game.
| How would you represent a constraint that would prevent a health
| component from being added to your chat message erroneously? In
| ECS it's straightforward to create a system which operates on
| chat messages individually, but how would you query your chat
| messages so that you can display them in order?_
|
| To me, that just means "Don't use ECS for those." I have a really
| nice coffee mug that is just perfect for holding coffee. It does
| its job very well. That does not mean I feel any need to use that
| coffee mug for digging holes in my garden.
|
| Databases and the relational model are great. ECS is great.
| Object-oriented programming is great. Functional programming is
| great. But treat them all as tools that should be used for the
| right job.
| jerf wrote:
| "At the same time, the level of hype about ECS today reminds me
| an awful lot of the amount of hype surrounding OOP in the 90s."
|
| I appreciate that most of the ECS hype has been around specific
| use cases, though.
|
| OOP was claimed as not a specific useful tool, but the answer
| to all programming, a billing it has not lived up to. It
| achieved "useful tool", no question, especially as some of the
| very rough bits were sanded off the original proposals
| (particularly the idea that objects should only and exactly
| match real world objects, an idea which I believe in hindsight
| was accidentally carried over from a simulation worldview in
| Simula where maybe it worked into the general programming world
| where it didn't), but it certainly has failed to be the one
| true programming methdology.
|
| I haven't seen anyone claiming everything should be rewritten
| in ECS.
| hyperpape wrote:
| > I appreciate that most of the ECS hype has been around
| specific use cases, though.
|
| I recently watched the first 30 minutes of Mike Acton's 2014
| talk, and while that portion of the talk wasn't about ECS
| specifically, it very much presented an absolutist
| perspective.
| c048 wrote:
| This is a issue with how most potential solutions or
| languages are marketed, even to experienced developers.
| They're all sold as silver bullets and every experienced
| programmer should know that there is no such thing.
|
| A lot of it also stems from what I call 'the hello world
| problem'. Very simple cases are brought up to sell you on
| the idea, but it's only in complex scenarios that you
| really learn the worth of something.
| hot_gril wrote:
| OOP even got mixed with databases to form nasty hype around
| ORMs and NoSQL.
| meheleventyone wrote:
| This. To me a well engineered game from a data perspective is a
| set of seperate datastores optimized for the job they are doing
| referencing one another through handles. I can see the drive to
| represent this in a general purpose way but you nearly always
| lose performance and/or flexibility. Ironically the "where do I
| put chat messages in my ECS" example illustrates that nicely.
| morganherlocker wrote:
| > (There are ECS frameworks in JavaScript, which gives you
| absolutely no control over memory layout and thus completely
| defeats one of the primary purposes of the pattern.)
|
| While JS does not provide great support for bit packing complex
| structs, typed arrays give you quite a bit of control over
| memory layout for simple numeric types, which is what you
| usually want for optimal data-oriented code anyway. This is a
| common technique used in fast JS libs for data visualization,
| ie:
|
| https://github.com/mourner/flatbush
|
| There are also basic operators required for bitarrays, which
| are useful for ECS and memory-efficient code generally.
| p1necone wrote:
| > There are ECS frameworks in JavaScript, which gives you
| absolutely no control over memory layout and thus completely
| defeats one of the primary purposes of the pattern.
|
| It doesn't give you _direct_ control over the memory layout,
| but it 's still fairly safe to assume that arrays are going to
| end up in relatively contiguous memory, which the relevant part
| for the performance difference between structs of arrays and
| arrays of structs.
|
| I don't recall explicitly telling many compiled languages to
| stick all the items in an array together in memory either - it
| just happens by default, same as in JS.
| Rusky wrote:
| JS arrays don't store their elements directly, they store
| references to their elements. (Unless the elements are
| primitives and the engine is using NaN-boxing or pointer
| tagging to store those inline, or something like that, which
| doesn't apply to ECS components generally.)
|
| Depending on how the GC works, the elements themselves might
| still wind up next to each other in memory some of the time.
| But that is definitely not a pattern that you would expect to
| hold in general- components will be added and removed over
| the course of the game, with lots of other stuff happening in
| between.
|
| In C++, the layout of the array elements is actually part of
| the language semantics. In JS, the language semantics don't
| even have a way to _talk about_ that layout, and engines in
| practice don 't use the layout you want.
| meheleventyone wrote:
| To get this in JS you should do it explicitly with
| TypedArrays otherwise you're at risk of what the underlying
| VM does and in current modern implementations it most
| assuredly doesn't make sure most kinds of arrays end up
| relatively contiguous. Notably for this conversation arrays
| of objects are unlikely to.
| DrDroop wrote:
| What about TypedArrays though, they could be useful to
| implement a ECS in JS. I do agree with your point, but ECS
| could be a useful pattern in JS not just for graphics or games
| but also OLAP workloads.
| tlarkworthy wrote:
| I don't think memory layout is the sole draw, it's also about
| incrementally building up programs live.
| lijok wrote:
| SpacetimeDB looks great for game dev. I've been hearing chitter
| chatter in the FaaS world about connecting clients directly to
| the DB as a means of reducing number of components and complexity
| in simple-ish CRUD apps. Not sure how well that design holds up
| there, time will tell.
|
| For those seeing this (SpacetimeDB) and immediatelly conjuring
| images of nightmares to be, consider the following: if you had an
| extremely latency sensitive usecase and had the opportunity to
| host your database and business logic on the same machine, why
| wouldn't you?
| Nezteb wrote:
| I agree!
|
| I first learned about SpacetimeDB from a HN thread posted three
| months ago: https://news.ycombinator.com/item?id=37146952
| inopinatus wrote:
| Author feedback: the large print, and the static elements
| (screen-thieving banner + "AI assistant" button), made attempting
| to read this blog as presented a vile experience on my phone. The
| AI button can't be dismissed, if you ask it how to dismiss itself
| it doesn't know, and it won't even accept negative feedback due
| to obscured submit buttons.
|
| Fortunately there's Reader mode.
| cloutiertyler wrote:
| Thanks for the feedback, we've definitely got to fix the mobile
| scaling stuff.
| slifin wrote:
| There's a good interview with Nathan Marz on his Rama project
| interviewed by Juxt
|
| He makes the point that databases are today prescriptive on the
| structure of data you provide to them
|
| I think databases as we know them today are not yet "end game"
___________________________________________________________________
(page generated 2023-12-06 23:00 UTC)