[HN Gopher] Deno KV internals: building a database for the moder...
___________________________________________________________________
Deno KV internals: building a database for the modern web
Author : avinassh
Score : 122 points
Date : 2024-05-04 17:01 UTC (6 days ago)
(HTM) web link (deno.com)
(TXT) w3m dump (deno.com)
| jaxelr wrote:
| (2023)
| hpeter wrote:
| I think Deno KV is a great tool with a nice API, it's great to
| see how it works. Really well designed.
|
| I used it a couple of times locally with Sqlite for CLI apps, if
| you want to do some data manipulation stuff with TS from the CLI
| and need a db, don't look further.
|
| I also used it in Production with FoundationDB on Deno Deploy.
|
| It does not replace your postgres/mysql database but a different
| beast entirely for many reasons. One is pricing. You pay per read
| and write.
|
| An issue I had is that it's hard to migrate from using KV with
| Deno Deploy. You can migrate to a local SQLite backed instance
| but will need to develop your own solutions on how you migrate
| and it will cost more the larger your database gets because you
| pay for reads.
|
| I do think it's great, but I would recommend using Deno Deploy
| only if your reads and writes produce value that offset the
| costs, else you can find yourself in the issue of needing to
| migrate.
|
| For example, use it for features you offer to authenticated
| users, but don't use it for things available in the open, else
| you open up yourself to high fees from DDOS.
| skybrian wrote:
| For reads, maybe in-memory caching would help with DDOS?
| mbb70 wrote:
| Deno charges for "inbound HTTP requests", so the DDOS can
| just query uuids till your checks start to bounce
| fidotron wrote:
| I happened to use foundationdb for the leaderboards for an
| experimental game recently, for which fdb was the main
| experiment.
|
| The big difference was I used the "record layer", however,
| naively believing this meant only needing to understand the
| record layer . . . No. In foundationdb it is vital to understand
| all the layers up to and including the one you are using, partly
| so you can make sense of the documentation, but also because the
| problems you run into are so tied to the semantics of the lower
| layers.
|
| That said, the great thing about fdb is deploying it is so easy.
| It is like a secret weapon hiding in plain sight.
|
| The game https://luduxia.com/showdown/
| dzonga wrote:
| beautiful game!! all on the web :clap
| fidotron wrote:
| Thanks! I also added them to my other game that's been here
| before: https://www.luduxia.com/whichwayround/
|
| The funny thing is I modeled the service for leaderboards on
| a commercial product an old boss of mine wrote v1 of in
| PHP/MySQL over 20 years ago. https://web.archive.org/web/2004
| 0806114005/http://www.macros...
|
| Games people end up with things like massively sharded mysql
| and replication fun. One of the nice potential things with
| fdb is keeping transactions within user scope, and not having
| to deal with the sharding/replication yourself, you just have
| to arrange the keyspace correctly instead. I have worked on
| games where people sorely underestimated the complexities of
| db replication and literally burned tens of millions in
| recovering from the mistake.
| the_gipsy wrote:
| "Deno KV" looks uncomfortably shoehorned into Deno. Some people
| are of course gonna use it if it's built in, but it doesn't look
| something that a good language/platform would ship with, if it
| wasn't for the money.
| satvikpendem wrote:
| Seems like these companies want to be like Vercel or Supabase,
| a full backend as a service where you just write code and
| deploy and the backend handles everything for you.
| BoorishBears wrote:
| Doesn't just seem like it, follow Guillermo Rauch's
| investments.
|
| The same game plan every. single. time. Get into the JS
| developers' zeitgeist, wrap an existing cloud service with a
| new frontend, sell it at a markup.
|
| Bonus points if the services can sell for each other.
| satvikpendem wrote:
| True, however the DX is outstanding for Vercel, and the
| price is pretty decent enough, even though I know they all
| just wrap AWS. If I don't have to deal with AWS, I would
| (and do) pay extra to not do so.
| BoorishBears wrote:
| Vercel is not a problem. Pumping millions of dollars into
| the JS ecosystem through sponsorships and events to
| define the development landscape in terms of what helps
| your bottom line... that's bad.
|
| React has RSC because Vercel wanted to fight Shopify.
| NextAuth was practically bought out by Clerk to serve as
| sales funnel. <img> tags are marked as "not good" by a
| leading React framework because the hosting provider
| behind it wants you to pay them for image optimization.
|
| What Rauch is doing is the developer equivalent of
| private equity squeezing, and what's insane is how well
| it's working.
| satvikpendem wrote:
| Eh I'll disagree with much of that. RSCs are amazing and
| basically what I've been looking for for quite a while,
| using TypeScript and JSX as a templating language for
| HTML, as I used to use PHP, but in a much safer way.
| Similarly, the image optimization is a good thing too,
| not necessarily that it goes through Vercel's servers but
| good in general. Dealing with srcsets with many images
| can be a pain and you can always use alternative image
| optimizer too, even open source local ones like sharp.
| Finally, there are lots of alternatives to what you're
| stating, no one forces you to use NextJS over, say, Remix
| or Vite with SSR. You can even not use React at all, or
| even JS / TS, there are lots of languages that compile
| HTML together.
| BoorishBears wrote:
| You're free to disagree but you didn't disagree at all.
|
| No one is saying they're not useful at all, the problem
| is Vercel (or really Rauch's entire cartel of companies)
| strong arm different technologies and topics to fit a
| narrative that's disproportionately in favor of "use our
| thing".
|
| RSCs are not amazing if you're not Vercel and don't have
| access to their closed build format (especially after
| they killed the serverless output format)
|
| I use image optimizers, I'm not about to go around
| telling people that img tags are bad.
|
| > Finally, there are lots of alternatives to what you're
| stating, no one forces you to use NextJS over, say, Remix
| or Vite with SSR.
|
| Remix had to reject RSCs to start because as one might
| expect, having one (1) design partner doesn't make for
| the most fully baked proposition.
|
| Also the "there's other choices" refrain ignores the fact
| that no one is developing in a vacuum. Vercel is pumping
| enough money into the ecosystem that the mindshare is
| shifting towards Next regardless of technical merit. So
| it's not enough to say "who cares if Vercel distorts what
| Next is, just use something else"
| satvikpendem wrote:
| Remix is adding RSCs in the future, they've already
| stated. And Vercel isn't as big in influence as you
| think, many are also having problems with NextJS that are
| causing them to go elsewhere such as Vite or Remix as I
| mentioned. There is no monopoly in the frontend.
| reducesuffering wrote:
| > React has RSC because Vercel wanted to fight Shopify
|
| Dan Abramov (react core team) already said that the React
| team wanted to do RSC, Vercel were the followers that
| were eager to integrate and productionize what the React
| team wanted to do.
|
| NextAuth is a competitor to Clerk auth. How is it bought
| out? Because Vercel pays an OSS developer to further
| develop NextAuth, and Guillermo also invested in Clerk?
| Someone using NextAuth means they're not using Clerk.
| hpeter wrote:
| Yes, vendor lock in. That's why you need to be careful what you
| use it for. You rely on it too much, you can't get away from it
| and can end up costing you a lot of money with the pay per
| request pricing.
|
| I don't want to write negatives about it, it's a well thought
| out product. But it 's not free. It is a paid service at the
| end and that is fine as long as you know what you are getting
| into.
| galaxyLogic wrote:
| I guess the question is, is the interface to Deno-KV specified
| well enough that it could be replaced by another back-end?
|
| Is it a "replaceable part"?
| jshen wrote:
| "it doesn't look something that a good language/platform would
| ship with"
|
| Why do you think that?
| evbogue wrote:
| As many of the comments here are getting at KV is best suited
| for making your app state available at all edge servers at Deno
| Deploy.
|
| You can also use localStorage in Deno but it won't synchronize
| the data between edge servers on Deploy, so you have to figure
| out another strategy.
|
| and of course KV works without Deploy, so while you don't get
| the data distribution benefit, you can still use KV on other
| deployment platforms.
|
| The alternative, of course, is to use another database. You
| don't have to use KV in your Deno apps.
|
| Maybe what people want here is the ability to configure their
| own data synchronization mirrors? Is that already implemented?
| vvpan wrote:
| Can a language that is not running on top of a VM pull this
| off? I do not think so? Which is why it seems to me we need to
| revisit JVM for example.
| afavour wrote:
| Welcome to the world of VC-driven tech ecosystems!
| baq wrote:
| make it work, make it good, make it expensive.
| mcpar-land wrote:
| paid SaaS features appearing _in my language 's runtime_ is...
| odd.
|
| I remember on launch, Deno specifically did not add backwards
| compatibility for installing Node.js packages from npm, by
| design. It was supposed to be a "greenfield" approach to
| serverside javascript/typescript... buuut then they folded and
| added npm dependency mgmt to their tooling.
|
| Some of the decisions in Deno feel like the "grow and find ways
| to monetize" strategy of your average vc-funded tech startup, but
| instead of being a SaaS it's your whole runtime.
| pjmlp wrote:
| One of the reasons .NET contracts were never widely adopted was
| that the infrastructure to make them useful was only available
| on VS Enterprise.
|
| Since most companies go with VS Professional, zero adoption.
|
| Same applies to unit testing code coverage on VS.
| skybrian wrote:
| The built-in Sqlite-based implementation is free and fully
| functional. It should be useful for local apps.
| cryptonector wrote:
| SQLite3-based implementation of... what?
|
| Oh. Deno KV uses SQLite3 under the covers. That's... funny.
| threecheese wrote:
| Outsider perspective; wrapping SQLite to provide a simple
| k/v JavaScript object persistence that supports all the
| strangeness of their types feels valuable, no? Irrespective
| of the economics of Deno itself.
| keb_ wrote:
| It'd be more useful if Deno exposed a SQLite driver or
| included one in the standard library like Bun.
| keb_ wrote:
| Yeah, the loss of Deno's original mission statement was
| saddening. I was hoping for a scrappy, but minimal and slowly
| growing runtime. Now that's it's being pumped with VC money,
| the team is spreading themselves thin between KV, Deno Deploy,
| Fresh Framework, JSR, and NPM/Node compatibility.
| devoli wrote:
| KV is open source and can be self hosted
| cryptonector wrote:
| With what license?
|
| (Several clicks in it looks like
| https://github.com/denoland/denokv is the repo and it's an
| MIT license.)
| ianbutler wrote:
| We actually implemented FoundationDB for our Python focused cloud
| platform [0] recently too.
|
| We found it to be super easy to set up and scale and it. Setting
| up FDB is literally install a package and make sure a file gets
| to all nodes in a cluster and you're good. It also integrated
| really well with our function runtime.
|
| You get a lot of distributed goodies for free in FDB with little
| effort and they "stack" their primitives very well too. As an
| example, there's a built in multi-tenancy layer that is just
| using key prefixes under the hood, but it's built in natively and
| can be accessed by the higher level apis.
|
| It's interesting that Deno went with a full separate transaction
| layer per region on top of a global cluster instead of doing
| regional clusters and making one region the primary writer and
| then doing request coalescing.
|
| [0] https://www.bismuthos.com
| cpursley wrote:
| Half of HN posts are about poor implementations of what's already
| a first class citizen in the Erlang runtime / Elixir ecosystem.
| rgrieselhuber wrote:
| For non-Erlang people, what's the reference?
| tock wrote:
| I'm guessing: https://www.erlang.org/doc/man/mnesia.html
| cpursley wrote:
| And ets: https://www.erlang.org/docs/24/man/ets.html
| EarthLaunch wrote:
| Sort of. I was using Erlang in 2007, well before switching to
| Node/JS. Erlang/Elixir has its downsides, which is why I
| switched.
|
| The entire industry has been twisting itself in knots trying to
| solve ops problems that Erlang/OTP solved in software long ago.
|
| I'm trying Deno Deploy though, because it seems like an attempt
| to combine those benefits with the JS ecosystem. That has
| advantages in: Language usability, frontend/isomorphic,
| libraries, and serverless.
|
| So far it feels like the future. Something like this will be,
| though I'm almost expecting a new language for it.
| jauntywundrkind wrote:
| I'm used to KV systems being really simple shallow systems. WASI-
| keyvalue for example is quite simple, stopping at providing an
| increment capability. https://github.com/WebAssembly/wasi-
| keyvalue/blob/main/wit/a...
|
| I knew Deno KV was built on FoundationDB, expected this would be
| a neat but simple systems architecture rundown. But... Turns out
| Deni really went super deep in building a KV, by adding atomics!
|
| > _To maximize performance and concurrency for atomic operations,
| we built the Deno KV Transaction Layer that manages the global
| order of atomic operations. For each transaction that is received
| by this Transaction Layer:_
|
| I thought it was particularly creative how a .get returns not
| just the value, but some reference to what the get was. So when
| the atomic change comes, they can check the get. This was a neat
| way to let some sets use previous data safely, I thought.
|
| Does anyone else have any other examples of thicker kv APIs? It
| feels like we're nearing a cap'n'proto level of promise
| pipelining, as we extend kv this way; I though Denos dodge was
| extremely expertly picked to limit how complex references to
| existing operations would need to be.
| skybrian wrote:
| Deno KV has a maximum record size of 64Kib [1] which is pretty
| severe. Compare with 25Mib for Cloudflare Worker KV [2]. It can
| be worked around, but it will make your design more complex if
| you have potentially large textareas, let alone images. Hopefully
| it will be raised by the time Deno KV gets out of beta.
|
| [1] https://deno.land/api@v1.43.2?s=Deno.Kv
|
| [2] https://developers.cloudflare.com/kv/platform/limits/
| pier25 wrote:
| I big issue with Cloudflare KV (and Workers too) is that values
| do not persist at the edge. They are copied on demand much like
| a CDN and then expired after some time. This works great when
| you have a ton of traffic but sucks when you want consistent
| good performance with erratic traffic.
|
| I wish CF had a premium tier where both Workers and KV would
| persist the the edge.
| kentonv wrote:
| Durable Objects persist at the edge.
| pier25 wrote:
| so when you make a change it's replicated to all regions?
| giancarlostoro wrote:
| I have to wonder if it might have been easier to make something
| like localStorage, but call it serverStorage and add that into
| deno, then anyone could use both interchangeably. localStorage is
| well known and super simple to use.
| holoduke wrote:
| Yes of course. 40 xhr requests to load a single page.
| Developers with no clue on the impact on what they are doing.
| Sorry a bit pessimistic here. Just had to deal with some
| serious similar issues in a team.
| louismerlin wrote:
| I'd be curious to know if the SQLite version would be suitable in
| production for a small to medium-sized project. Did anybody try
| it out?
| cryptonector wrote:
| > Web apps very often require some persistent application state.
| Setting up a database involves numerous configuration steps and
| the subsequent integration of an ORM or other systems.
|
| So this is a NoSQL argument.
|
| > Atomic transactions: a database that provides atomic operations
| to ensure data integrity and enable complex application logic
|
| Ok, so let's see if they mention "ACID". Nope. They do mention
| atomicity and durability, but they don't mention consistency and
| isolation.
|
| So this is about NoSQL and NoACID, but it's not really stated
| this way.
|
| K/V stores are great for many things. And not having a query
| language is great!! right up until you need one, then you get to
| be sad (I should know since I've had to write a query engine due
| to use of a K/V store).
|
| [No]ACID is where NoSQL gets interesting, IMO. But NoACID doesn't
| mean you can't have a query language.
|
| Anyways, if you're going to implement yet another K/V store you
| really should justify it in relation to other K/V stores that
| exist, not as "using RDBMSes is ETOOHARD".
| dangoodmanUT wrote:
| This feels like an unnecessary layer on top? Increments can just
| be a function that uses an existing txn to read, incr, write. It
| feels like they implemented transactions on top of transactions,
| adding complexity?
|
| Maybe I didn't read it right though
| spirobelv2 wrote:
| just use bun and sqlite on your own server. got burned by deno
| deploy and will never use deno again. serverless is useless and
| slow.
___________________________________________________________________
(page generated 2024-05-10 23:00 UTC)